| *** olaph1 has joined #openstack-infra | 00:02 | |
| *** olaph has quit IRC | 00:03 | |
| *** katkapilatova1 has quit IRC | 00:04 | |
| ianw | ok, pass 2 now | 00:10 |
|---|---|---|
| *** rfolco has quit IRC | 00:11 | |
| *** superdan is now known as dansmith | 00:11 | |
| *** bobh has joined #openstack-infra | 00:18 | |
| *** Goneri has quit IRC | 00:18 | |
| *** markvoelker has joined #openstack-infra | 00:20 | |
| *** bobh has quit IRC | 00:22 | |
| *** edmondsw has joined #openstack-infra | 00:24 | |
| *** markvoelker has quit IRC | 00:24 | |
| *** bobh has joined #openstack-infra | 00:27 | |
| *** edmondsw has quit IRC | 00:28 | |
| *** claudiub has quit IRC | 00:31 | |
| *** bobh has quit IRC | 00:32 | |
| *** dhajare has quit IRC | 00:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: config: add statsd-server config parameter https://review.openstack.org/528969 | 00:33 |
| prometheanfire | ok, let us know when to recheck | 00:34 |
| *** ijw has joined #openstack-infra | 00:36 | |
| *** ijw has quit IRC | 00:41 | |
| *** dhajare has joined #openstack-infra | 00:45 | |
| ianw | i wonder if sharding the directories would reduce our exposure to these long fscks, or just end up more unstable as more volumes == more potential problems | 00:50 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Use olso to parse size strings https://review.openstack.org/503574 | 00:53 |
| *** auggy has quit IRC | 00:55 | |
| *** auggy has joined #openstack-infra | 00:55 | |
| *** kiennt26 has joined #openstack-infra | 00:56 | |
| *** bobh has joined #openstack-infra | 00:57 | |
| *** bobh has quit IRC | 01:02 | |
| fungi | ianw: if we sharded by volume, then yes it would very much reduce this | 01:11 |
| *** eharney has quit IRC | 01:14 | |
| *** tosky has quit IRC | 01:17 | |
| *** hongbin has joined #openstack-infra | 01:17 | |
| *** bobh has joined #openstack-infra | 01:18 | |
| *** bobh has quit IRC | 01:22 | |
| *** jascott1 has quit IRC | 01:25 | |
| fungi | at last the fsck is on to pass #2 now | 01:26 |
| *** dtruong2 has joined #openstack-infra | 01:27 | |
| *** daidv has joined #openstack-infra | 01:27 | |
| *** cuongnv has joined #openstack-infra | 01:30 | |
| *** bobh has joined #openstack-infra | 01:30 | |
| *** bobh has quit IRC | 01:34 | |
| *** bobh has joined #openstack-infra | 01:38 | |
| *** liujiong has joined #openstack-infra | 01:39 | |
| *** threestrands_ has joined #openstack-infra | 01:40 | |
| ianw | yeah it's been about an hour there | 01:42 |
| *** bobh has quit IRC | 01:43 | |
| *** threestrands has quit IRC | 01:43 | |
| *** armax has joined #openstack-infra | 01:44 | |
| *** dtruong2 has quit IRC | 01:45 | |
| *** jascott1 has joined #openstack-infra | 01:46 | |
| *** katkapilatova1 has joined #openstack-infra | 01:47 | |
| *** jascott1 has quit IRC | 01:50 | |
| *** cshastri has joined #openstack-infra | 01:54 | |
| *** namnh has joined #openstack-infra | 01:56 | |
| *** cshastri has quit IRC | 02:07 | |
| *** bobh has joined #openstack-infra | 02:08 | |
| *** bobh has quit IRC | 02:13 | |
| *** abelur_ has joined #openstack-infra | 02:13 | |
| *** sbra has quit IRC | 02:14 | |
| *** sbra has joined #openstack-infra | 02:14 | |
| *** zhurong has joined #openstack-infra | 02:28 | |
| *** bobh has joined #openstack-infra | 02:40 | |
| *** quite has quit IRC | 02:40 | |
| *** quite has joined #openstack-infra | 02:44 | |
| *** bobh has quit IRC | 02:46 | |
| *** dkliban has joined #openstack-infra | 02:50 | |
| *** sree has joined #openstack-infra | 02:51 | |
| *** markvoelker has joined #openstack-infra | 02:51 | |
| *** dkliban has left #openstack-infra | 02:53 | |
| *** annp has joined #openstack-infra | 02:54 | |
| *** dhajare has quit IRC | 02:55 | |
| *** katkapilatova1 has quit IRC | 03:06 | |
| *** liujiong has quit IRC | 03:11 | |
| *** dhajare has joined #openstack-infra | 03:11 | |
| *** jappleii__ has joined #openstack-infra | 03:12 | |
| *** jappleii__ has quit IRC | 03:13 | |
| *** cshastri has joined #openstack-infra | 03:13 | |
| *** jappleii__ has joined #openstack-infra | 03:13 | |
| *** liujiong has joined #openstack-infra | 03:15 | |
| *** threestrands_ has quit IRC | 03:15 | |
| *** bobh has joined #openstack-infra | 03:18 | |
| *** katkapilatova1 has joined #openstack-infra | 03:19 | |
| *** bobh has quit IRC | 03:23 | |
| *** markvoelker has quit IRC | 03:25 | |
| *** rosmaita has quit IRC | 03:27 | |
| *** abhishekk has joined #openstack-infra | 03:28 | |
| *** namnh has quit IRC | 03:34 | |
| *** dhill_ has quit IRC | 03:34 | |
| *** cshastri has quit IRC | 03:34 | |
| *** namnh has joined #openstack-infra | 03:35 | |
| *** liujiong has quit IRC | 03:35 | |
| *** liujiong has joined #openstack-infra | 03:36 | |
| *** bobh has joined #openstack-infra | 03:40 | |
| *** liujiong has quit IRC | 03:41 | |
| *** bobh has quit IRC | 03:44 | |
| *** sdake has quit IRC | 03:46 | |
| *** sdake has joined #openstack-infra | 03:47 | |
| *** sdake has quit IRC | 03:47 | |
| *** sdake has joined #openstack-infra | 03:47 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] GPT partitioning support https://review.openstack.org/533490 | 03:47 |
| *** psachin has joined #openstack-infra | 03:47 | |
| *** olaph1 has quit IRC | 03:48 | |
| *** olaph has joined #openstack-infra | 03:49 | |
| prometheanfire | so, still broken? | 03:49 |
| *** liujiong has joined #openstack-infra | 03:49 | |
| *** bobh has joined #openstack-infra | 03:58 | |
| *** edmondsw has joined #openstack-infra | 04:00 | |
| *** bobh has quit IRC | 04:04 | |
| *** dhajare has quit IRC | 04:04 | |
| *** yamamoto has joined #openstack-infra | 04:05 | |
| *** edmondsw has quit IRC | 04:05 | |
| ianw | prometheanfire: it's still fscking. if you don't care about the logs, you can take a chance | 04:07 |
| *** ekcs has joined #openstack-infra | 04:08 | |
| *** claudiub has joined #openstack-infra | 04:08 | |
| *** bobh has joined #openstack-infra | 04:08 | |
| ianw | just ticked over to pass 3 | 04:09 |
| *** bhavik1 has joined #openstack-infra | 04:10 | |
| *** bobh has quit IRC | 04:12 | |
| *** bhavik1 has quit IRC | 04:16 | |
| prometheanfire | k | 04:19 |
| *** ykarel|away has joined #openstack-infra | 04:24 | |
| ianw | ok, it just finished | 04:29 |
| ianw | i'm going to stop apache and remount it | 04:32 |
| *** bobh has joined #openstack-infra | 04:34 | |
| *** bobh has quit IRC | 04:37 | |
| *** olaph1 has joined #openstack-infra | 04:39 | |
| *** zhurong has quit IRC | 04:39 | |
| *** olaph has quit IRC | 04:40 | |
| ianw | i've mounted the buffered logs and rsyncing them in that screen now | 04:41 |
| *** ameliac has quit IRC | 04:42 | |
| ianw | #status alert The logs.openstack.org filesystem has been restored to full health. We are attempting to keep logs uploaded between the prior alert and this one, however if your job logs are missing please issue a recheck. | 04:45 |
| openstackstatus | ianw: sending alert | 04:45 |
| *** bobh has joined #openstack-infra | 04:48 | |
| -openstackstatus- NOTICE: The logs.openstack.org filesystem has been restored to full health. We are attempting to keep logs uploaded between the prior alert and this one, however if your job logs are missing please issue a recheck. | 04:48 | |
| *** ChanServ changes topic to "The logs.openstack.org filesystem has been restored to full health. We are attempting to keep logs uploaded between the prior alert and this one, however if your job logs are missing please issue a recheck." | 04:48 | |
| ianw | bahh, that was supposed to be an ok | 04:49 |
| openstackstatus | ianw: finished sending alert | 04:51 |
| ianw | #status ok | 04:51 |
| openstackstatus | ianw: sending ok | 04:51 |
| fungi | just checking back in. all set then? | 04:52 |
| ianw | on the level of alert f-ups this week, however, i'll consider that minor :) | 04:52 |
| *** bobh has quit IRC | 04:52 | |
| fungi | heh | 04:52 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 04:52 | |
| fungi | i've done that plenty | 04:53 |
| fungi | a'ight, back to bed for me. thanks, ianw! | 04:53 |
| ianw | fungi: goodnight! | 04:53 |
| openstackstatus | ianw: finished sending ok | 04:54 |
| *** bobh has joined #openstack-infra | 04:56 | |
| *** janki has joined #openstack-infra | 04:59 | |
| *** bobh has quit IRC | 05:01 | |
| *** dhajare has joined #openstack-infra | 05:01 | |
| *** dhajare is now known as dhajare|BNE|Mtg | 05:01 | |
| *** links has joined #openstack-infra | 05:11 | |
| *** olaph has joined #openstack-infra | 05:15 | |
| *** ramishra has joined #openstack-infra | 05:16 | |
| *** olaph1 has quit IRC | 05:17 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] GPT partitioning support https://review.openstack.org/533490 | 05:21 |
| *** markvoelker has joined #openstack-infra | 05:22 | |
| *** pcaruana has joined #openstack-infra | 05:23 | |
| *** zhurong has joined #openstack-infra | 05:24 | |
| *** ramishra has quit IRC | 05:31 | |
| *** pcaruana has quit IRC | 05:32 | |
| *** gouthamr has joined #openstack-infra | 05:35 | |
| *** dsariel has quit IRC | 05:40 | |
| *** hongbin has quit IRC | 05:43 | |
| *** dingyichen has quit IRC | 05:45 | |
| *** dingyichen has joined #openstack-infra | 05:45 | |
| *** xinliang has quit IRC | 05:46 | |
| *** edmondsw has joined #openstack-infra | 05:48 | |
| AJaeger | ianw, frickler, could you review https://review.openstack.org/#/c/533347/ and https://review.openstack.org/#/c/533348/ - neutron jobs moved in-tree | 05:50 |
| AJaeger | , please? | 05:50 |
| *** olaph1 has joined #openstack-infra | 05:52 | |
| *** olaph has quit IRC | 05:52 | |
| *** edmondsw has quit IRC | 05:53 | |
| *** gouthamr has quit IRC | 05:54 | |
| *** yamamoto_ has joined #openstack-infra | 05:55 | |
| *** yamamoto_ has quit IRC | 05:55 | |
| *** markvoelker has quit IRC | 05:55 | |
| AJaeger | ianw: do you want to recheck https://review.openstack.org/532701 - the dib 2.10 pause revert? Or wait a day? | 05:56 |
| *** yamamoto has quit IRC | 05:59 | |
| *** dingyichen has quit IRC | 06:00 | |
| *** dingyichen has joined #openstack-infra | 06:01 | |
| *** xinliang has joined #openstack-infra | 06:01 | |
| *** xinliang has quit IRC | 06:01 | |
| *** xinliang has joined #openstack-infra | 06:01 | |
| *** dhajare|BNE|Mtg has quit IRC | 06:05 | |
| *** janki is now known as janki|afk | 06:10 | |
| *** janki|afk has quit IRC | 06:10 | |
| *** janki has joined #openstack-infra | 06:11 | |
| *** janki is now known as janki|afk | 06:12 | |
| *** yamamoto has joined #openstack-infra | 06:16 | |
| *** dimak has quit IRC | 06:18 | |
| ianw | AJaeger: I was going to ping pabelanger and suggest he might like to do it, or i'll do it my time tomorrow | 06:19 |
| ianw | i didn't want to do it while we were checking the logs, just in case that turned into a whole big thing | 06:19 |
| *** dimak has joined #openstack-infra | 06:20 | |
| *** cshastri has joined #openstack-infra | 06:23 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] GPT partitioning support https://review.openstack.org/533490 | 06:25 |
| ianw | pabelanger: ^ consider that a ping :) | 06:25 |
| ianw | ^^ | 06:25 |
| *** aeng has quit IRC | 06:27 | |
| *** bobh has joined #openstack-infra | 06:33 | |
| *** felipemonteiro_ has joined #openstack-infra | 06:37 | |
| *** bobh has quit IRC | 06:37 | |
| *** abhishekk has quit IRC | 06:37 | |
| *** ekcs has quit IRC | 06:38 | |
| *** ekcs has joined #openstack-infra | 06:38 | |
| *** dingyichen has quit IRC | 06:43 | |
| *** dingyichen has joined #openstack-infra | 06:43 | |
| *** dingyichen has quit IRC | 06:44 | |
| *** dingyichen has joined #openstack-infra | 06:45 | |
| *** bobh has joined #openstack-infra | 06:46 | |
| *** dingyichen has quit IRC | 06:47 | |
| *** dingyichen has joined #openstack-infra | 06:47 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: ansible: honor command no_log module attribute https://review.openstack.org/533509 | 06:49 |
| *** abhishekk has joined #openstack-infra | 06:50 | |
| *** dingyichen has quit IRC | 06:50 | |
| *** bobh has quit IRC | 06:51 | |
| *** dingyichen has joined #openstack-infra | 06:51 | |
| *** dingyichen has joined #openstack-infra | 06:53 | |
| *** dhajare|BNE|Mtg has joined #openstack-infra | 06:53 | |
| *** dingyichen has quit IRC | 06:54 | |
| *** dingyichen has joined #openstack-infra | 06:55 | |
| *** felipemonteiro_ has quit IRC | 06:58 | |
| *** jbadiapa has joined #openstack-infra | 06:59 | |
| *** bobh has joined #openstack-infra | 07:00 | |
| *** jappleii__ has quit IRC | 07:02 | |
| *** bobh has quit IRC | 07:04 | |
| *** dingyichen has quit IRC | 07:04 | |
| *** dingyichen has joined #openstack-infra | 07:05 | |
| *** dingyichen has joined #openstack-infra | 07:05 | |
| *** cshastri has quit IRC | 07:06 | |
| *** bobh has joined #openstack-infra | 07:07 | |
| *** MasterofJOKers_ has joined #openstack-infra | 07:07 | |
| *** ankkumar has joined #openstack-infra | 07:08 | |
| *** dingyichen has quit IRC | 07:08 | |
| *** dingyichen has joined #openstack-infra | 07:09 | |
| *** slaweq has joined #openstack-infra | 07:10 | |
| *** MasterofJOKers has quit IRC | 07:10 | |
| *** MasterofJOKers_ is now known as MasterofJOKers | 07:10 | |
| *** bobh has quit IRC | 07:12 | |
| *** bobh has joined #openstack-infra | 07:13 | |
| *** dingyichen has quit IRC | 07:14 | |
| *** dingyichen has joined #openstack-infra | 07:14 | |
| *** dingyichen has quit IRC | 07:16 | |
| *** r-daneel has quit IRC | 07:17 | |
| *** dingyichen has joined #openstack-infra | 07:17 | |
| *** r-daneel has joined #openstack-infra | 07:17 | |
| *** bobh has quit IRC | 07:19 | |
| *** CrayZee has joined #openstack-infra | 07:21 | |
| *** armaan has quit IRC | 07:21 | |
| *** armaan has joined #openstack-infra | 07:21 | |
| *** dhajare|BNE|Mtg has quit IRC | 07:22 | |
| *** ramishra has joined #openstack-infra | 07:23 | |
| *** bobh has joined #openstack-infra | 07:24 | |
| *** alexchadin has joined #openstack-infra | 07:25 | |
| *** bobh_ has joined #openstack-infra | 07:28 | |
| *** bobh has quit IRC | 07:28 | |
| *** ykarel|away is now known as ykarel | 07:29 | |
| *** bobh_ has quit IRC | 07:32 | |
| *** dingyichen has quit IRC | 07:33 | |
| *** hjensas has quit IRC | 07:33 | |
| *** cshastri has joined #openstack-infra | 07:34 | |
| *** dingyichen has joined #openstack-infra | 07:34 | |
| *** hjensas has joined #openstack-infra | 07:34 | |
| *** hjensas has quit IRC | 07:34 | |
| *** hjensas has joined #openstack-infra | 07:34 | |
| *** jamesmcarthur has joined #openstack-infra | 07:35 | |
| *** dingyichen has quit IRC | 07:35 | |
| *** dingyichen has joined #openstack-infra | 07:36 | |
| *** edmondsw has joined #openstack-infra | 07:36 | |
| *** ramishra has quit IRC | 07:39 | |
| *** ramishra has joined #openstack-infra | 07:39 | |
| *** jamesmcarthur has quit IRC | 07:39 | |
| *** dingyichen has quit IRC | 07:39 | |
| *** edmondsw has quit IRC | 07:41 | |
| *** vsaienk0 has joined #openstack-infra | 07:41 | |
| *** makowals has joined #openstack-infra | 07:42 | |
| *** dingyichen has joined #openstack-infra | 07:45 | |
| *** dchen has joined #openstack-infra | 07:45 | |
| *** janki|afk is now known as janki | 07:46 | |
| *** bobh has joined #openstack-infra | 07:47 | |
| *** dchen has quit IRC | 07:49 | |
| *** dingyichen has quit IRC | 07:49 | |
| *** alexchadin has quit IRC | 07:49 | |
| *** liujiong has quit IRC | 07:50 | |
| openstackgerrit | dongwenjuan proposed openstack-infra/puppet-openstackci master: add `git_source_repo` parameter for zuul_scheduler https://review.openstack.org/528594 | 07:51 |
| *** bobh has quit IRC | 07:52 | |
| *** markvoelker has joined #openstack-infra | 07:53 | |
| ankkumar | tristanC, hi | 07:55 |
| ankkumar | I setup zuulv3 but my zuul web is configured properly.. it is giving error status.json not found.. Do have any idea about this? | 07:56 |
| *** cshastri has quit IRC | 07:56 | |
| ankkumar | *zuul web is not configured properly | 07:56 |
| *** dingyichen has joined #openstack-infra | 07:58 | |
| *** dchen has joined #openstack-infra | 07:58 | |
| *** dchen has quit IRC | 07:59 | |
| *** rcernin has quit IRC | 07:59 | |
| *** armaan has quit IRC | 08:04 | |
| *** armaan has joined #openstack-infra | 08:04 | |
| *** alexchadin has joined #openstack-infra | 08:05 | |
| tristanC | ankkumar: where is that error comming from? | 08:06 |
| ankkumar | tristanC, in browser | 08:06 |
| *** dingyichen has quit IRC | 08:07 | |
| *** eumel8 has joined #openstack-infra | 08:07 | |
| frickler | ianw: could you also take a look at https://review.openstack.org/529000 please? | 08:08 |
| *** hashar has joined #openstack-infra | 08:13 | |
| *** florianf has joined #openstack-infra | 08:15 | |
| *** dingyichen has joined #openstack-infra | 08:15 | |
| *** bobh has joined #openstack-infra | 08:18 | |
| *** zhurong has quit IRC | 08:18 | |
| *** ralonsoh has joined #openstack-infra | 08:19 | |
| *** zhurong has joined #openstack-infra | 08:21 | |
| *** amoralej|off is now known as amoralej | 08:21 | |
| *** ykarel is now known as ykarel|lunch | 08:22 | |
| *** bobh has quit IRC | 08:22 | |
| *** efoley has joined #openstack-infra | 08:24 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add buildset-artifacts-location and fetch roles https://review.openstack.org/530679 | 08:25 |
| *** markvoelker has quit IRC | 08:26 | |
| *** e0ne has joined #openstack-infra | 08:29 | |
| *** bobh has joined #openstack-infra | 08:31 | |
| *** dingyichen has quit IRC | 08:32 | |
| *** katkapilatova1 has quit IRC | 08:35 | |
| *** bobh has quit IRC | 08:36 | |
| *** olaph has joined #openstack-infra | 08:36 | |
| *** olaph1 has quit IRC | 08:39 | |
| *** sshnaidm is now known as sshnaidm|afk | 08:41 | |
| *** cshastri has joined #openstack-infra | 08:44 | |
| *** bobh has joined #openstack-infra | 08:48 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** shu-mutou-AWAY has quit IRC | 08:50 | |
| *** bobh has quit IRC | 08:52 | |
| *** e0ne has quit IRC | 08:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Removed duplicated job https://review.openstack.org/533569 | 08:55 |
| AJaeger | frickler: thanks for review, here's the followup ^ | 08:55 |
| *** adarazs has joined #openstack-infra | 08:57 | |
| *** bauwser is now known as bauzas | 08:57 | |
| *** jpich has joined #openstack-infra | 08:58 | |
| *** psachin has quit IRC | 08:59 | |
| *** shardy has joined #openstack-infra | 09:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Rename neutron grenade jobs https://review.openstack.org/533347 | 09:01 |
| *** rossella_s has joined #openstack-infra | 09:01 | |
| *** zoli is now known as zoli4 | 09:03 | |
| *** zoli4 is now known as zoli | 09:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: Rename neutron jobs https://review.openstack.org/533348 | 09:04 |
| *** alexchadin has quit IRC | 09:04 | |
| *** jtomasek has joined #openstack-infra | 09:05 | |
| *** alexchadin has joined #openstack-infra | 09:05 | |
| *** jascott1 has joined #openstack-infra | 09:08 | |
| *** efoley has quit IRC | 09:10 | |
| *** ociuhandu_ has quit IRC | 09:10 | |
| *** vsaienk0 has quit IRC | 09:13 | |
| *** lucas-afk is now known as lucasagomes | 09:15 | |
| *** alexchadin has quit IRC | 09:17 | |
| *** bobh has joined #openstack-infra | 09:18 | |
| *** bobh has quit IRC | 09:23 | |
| *** edmondsw has joined #openstack-infra | 09:25 | |
| *** dsariel has joined #openstack-infra | 09:25 | |
| *** bobh has joined #openstack-infra | 09:28 | |
| *** ykarel|lunch is now known as ykarel | 09:28 | |
| *** edmondsw has quit IRC | 09:29 | |
| *** bobh has quit IRC | 09:33 | |
| *** alexchadin has joined #openstack-infra | 09:36 | |
| *** derekh has joined #openstack-infra | 09:36 | |
| *** larsks has quit IRC | 09:40 | |
| *** larsks has joined #openstack-infra | 09:43 | |
| *** zhurong has quit IRC | 09:46 | |
| *** ganso has joined #openstack-infra | 09:49 | |
| *** bobh has joined #openstack-infra | 09:49 | |
| *** xinliang has quit IRC | 09:52 | |
| *** slaweq has quit IRC | 09:54 | |
| *** sshnaidm|afk has quit IRC | 09:55 | |
| *** bobh has quit IRC | 09:55 | |
| *** slaweq has joined #openstack-infra | 09:55 | |
| *** armaan has quit IRC | 09:55 | |
| *** armaan has joined #openstack-infra | 09:56 | |
| *** vsaienk0 has joined #openstack-infra | 09:57 | |
| *** slaweq has quit IRC | 09:59 | |
| *** jaosorior has joined #openstack-infra | 10:03 | |
| *** makowals has quit IRC | 10:04 | |
| *** xinliang has joined #openstack-infra | 10:05 | |
| *** xinliang has quit IRC | 10:05 | |
| *** xinliang has joined #openstack-infra | 10:05 | |
| *** makowals has joined #openstack-infra | 10:05 | |
| *** daidv has quit IRC | 10:06 | |
| *** slaweq has joined #openstack-infra | 10:06 | |
| *** vsaienk0 has quit IRC | 10:07 | |
| *** namnh has quit IRC | 10:11 | |
| *** vsaienk0 has joined #openstack-infra | 10:13 | |
| *** kiennt26 has quit IRC | 10:19 | |
| *** bobh has joined #openstack-infra | 10:20 | |
| *** pbourke has quit IRC | 10:21 | |
| *** vsaienk0 has quit IRC | 10:23 | |
| *** pbourke has joined #openstack-infra | 10:23 | |
| *** markvoelker has joined #openstack-infra | 10:24 | |
| *** psachin has joined #openstack-infra | 10:25 | |
| *** bobh has quit IRC | 10:25 | |
| *** sambetts|afk is now known as sambetts | 10:26 | |
| *** panda is now known as panda|ruck | 10:26 | |
| *** e0ne has joined #openstack-infra | 10:27 | |
| *** vsaienk0 has joined #openstack-infra | 10:27 | |
| *** tosky has joined #openstack-infra | 10:30 | |
| CrayZee | Hi, any chance anyone could help me with https://review.openstack.org/#/c/531720/ (in project-config)? Thanks | 10:32 |
| *** armaan has quit IRC | 10:33 | |
| *** armaan has joined #openstack-infra | 10:34 | |
| *** alexchadin has quit IRC | 10:36 | |
| *** cuongnv has quit IRC | 10:36 | |
| *** alexchadin has joined #openstack-infra | 10:36 | |
| *** sree has quit IRC | 10:39 | |
| *** sree has joined #openstack-infra | 10:39 | |
| *** ldnunes has joined #openstack-infra | 10:40 | |
| *** pgadiya has joined #openstack-infra | 10:40 | |
| *** CrayZee has quit IRC | 10:42 | |
| *** CrayZee has joined #openstack-infra | 10:43 | |
| *** pcichy has joined #openstack-infra | 10:45 | |
| *** pcichy has joined #openstack-infra | 10:46 | |
| *** sree has quit IRC | 10:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Removed duplicated jobs https://review.openstack.org/533569 | 10:46 |
| AJaeger | CrayZee: that's in system-config ;) | 10:47 |
| AJaeger | frickler: found two more duplicates - see 533569 | 10:47 |
| CrayZee | AJaeger: right. sorry, a bit tired :) meant system-config | 10:47 |
| AJaeger | CrayZee: then you need other reviewers than me... | 10:48 |
| CrayZee | AJaeger: no problem. am still looking for one... | 10:48 |
| *** dtantsur is now known as dtantsur|bbl | 10:48 | |
| AJaeger | CrayZee: not many around at this time | 10:48 |
| CrayZee | AJaeger: will ask again later today | 10:49 |
| CrayZee | Thansk | 10:49 |
| CrayZee | Thanks | 10:49 |
| *** annp has quit IRC | 10:51 | |
| *** abhishekk has quit IRC | 10:51 | |
| *** sshnaidm|afk has joined #openstack-infra | 10:52 | |
| *** vsaienk0 has quit IRC | 10:54 | |
| *** alexchadin has quit IRC | 10:54 | |
| *** alexchadin has joined #openstack-infra | 10:55 | |
| *** vsaienk0 has joined #openstack-infra | 10:55 | |
| openstackgerrit | Balazs Gibizer proposed openstack-infra/project-config master: Moving nova functional test def to in tree https://review.openstack.org/533608 | 10:55 |
| *** markvoelker has quit IRC | 10:57 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:58 | |
| *** alexchadin has quit IRC | 11:00 | |
| *** electrofelix has joined #openstack-infra | 11:06 | |
| *** rpittau has joined #openstack-infra | 11:08 | |
| *** alexchadin has joined #openstack-infra | 11:09 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** yamamoto has joined #openstack-infra | 11:11 | |
| *** rfolco has joined #openstack-infra | 11:11 | |
| *** edmondsw has joined #openstack-infra | 11:13 | |
| *** ramishra has quit IRC | 11:13 | |
| *** ramishra has joined #openstack-infra | 11:15 | |
| *** edmondsw has quit IRC | 11:17 | |
| *** bobh has joined #openstack-infra | 11:18 | |
| *** bobh has quit IRC | 11:22 | |
| *** alexchadin has quit IRC | 11:22 | |
| *** ramishra_ has joined #openstack-infra | 11:22 | |
| *** alexchadin has joined #openstack-infra | 11:23 | |
| *** ramishra has quit IRC | 11:23 | |
| *** jzerebec1i is now known as jzerebecki | 11:24 | |
| *** kjackal has joined #openstack-infra | 11:25 | |
| *** yamamoto_ has joined #openstack-infra | 11:26 | |
| *** bobh has joined #openstack-infra | 11:28 | |
| *** yamamoto has quit IRC | 11:29 | |
| *** ankkumar has quit IRC | 11:30 | |
| *** bobh has quit IRC | 11:32 | |
| *** jascott1 has quit IRC | 11:34 | |
| *** sbra has quit IRC | 11:37 | |
| *** sdague has joined #openstack-infra | 11:41 | |
| *** nicolasbock has joined #openstack-infra | 11:44 | |
| *** pcaruana has joined #openstack-infra | 11:44 | |
| *** sree has joined #openstack-infra | 11:46 | |
| *** bobh has joined #openstack-infra | 11:48 | |
| *** sree has quit IRC | 11:50 | |
| *** bobh has quit IRC | 11:53 | |
| *** alexchadin has quit IRC | 11:55 | |
| *** alexchadin has joined #openstack-infra | 11:57 | |
| openstackgerrit | Balazs Gibizer proposed openstack-infra/project-config master: Moving nova functional test def to in tree https://review.openstack.org/533608 | 12:07 |
| *** psachin has quit IRC | 12:10 | |
| AJaeger | frickler: could you do some review of the config repos, please? We have a backlog there. I listed some changes at https://etherpad.openstack.org/p/Nvt3ovbn5x | 12:10 |
| *** lucasagomes is now known as lucas-hungry | 12:13 | |
| *** jkilpatr has joined #openstack-infra | 12:15 | |
| *** tesseract has joined #openstack-infra | 12:15 | |
| * mnaser is in a plane for the next 2 hours | 12:17 | |
| mnaser | time to play reviews | 12:17 |
| *** sree has joined #openstack-infra | 12:18 | |
| *** pcichy has quit IRC | 12:18 | |
| *** bobh has joined #openstack-infra | 12:18 | |
| *** jkilpatr has quit IRC | 12:19 | |
| *** jkilpatr has joined #openstack-infra | 12:19 | |
| AJaeger | mnaser: there're plenty of to take ;) Thanks! Safe travels | 12:20 |
| mnaser | AJaeger: thanks! going to hot LA for a few days which should be nice to runaway from the cold | 12:20 |
| mnaser | ill go over your backlog | 12:21 |
| AJaeger | mnaser: work, fun - or combination? | 12:21 |
| mnaser | AJaeger: i usually fly for work only and come back but this time i extended my stay (not by much but still) | 12:21 |
| mnaser | last time i was in california, i was in san diego and I didn't even see what the beach looked like! | 12:21 |
| AJaeger | Have fun! | 12:22 |
| *** sree_ has joined #openstack-infra | 12:22 | |
| mnaser | thank you | 12:22 |
| *** sree_ is now known as Guest18249 | 12:23 | |
| *** dhajare|BNE|Mtg has joined #openstack-infra | 12:23 | |
| *** bobh has quit IRC | 12:23 | |
| *** rfolco is now known as rfolco_nho | 12:25 | |
| *** sree has quit IRC | 12:25 | |
| mnaser | that subunit change has been there a while but i dont know enough about it to review :( | 12:26 |
| AJaeger | ok, let's ping another config-core for https://review.openstack.org/#/c/529339 | 12:28 |
| *** olaph1 has joined #openstack-infra | 12:29 | |
| mnaser | AJaeger: do you think we should look into adding the version # to nodejs job names | 12:30 |
| mnaser | say if a project wants to test both nodejs 6 and nodejs 8 .. they both have the same job names | 12:30 |
| mnaser | i can approve the change for 8 but i think we should go back and add version #s to the job names to allow a project to gate two versions (unless thats overoptimizing) | 12:31 |
| *** olaph has quit IRC | 12:31 | |
| AJaeger | mnaser: a project can run both already today. Those are variants UFAIU | 12:32 |
| AJaeger | mnaser: might indeed be nice to rename so that those can be easily differentiated. | 12:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Don't run legacy-tempest-dsvm-neutron-full on non-code nova changes https://review.openstack.org/531052 | 12:35 |
| openstackgerrit | Merged openstack-infra/project-config master: add neutron-lib master periodic job to vmware-nsx https://review.openstack.org/530957 | 12:35 |
| *** jpena is now known as jpena|lunch | 12:38 | |
| *** panda|ruck is now known as panda|ruck|lunch | 12:38 | |
| *** amoralej is now known as amoralej|lunch | 12:39 | |
| *** andreas_s has joined #openstack-infra | 12:39 | |
| *** r-daneel_ has joined #openstack-infra | 12:42 | |
| mnaser | AJaeger: https://review.openstack.org/#/c/531936/ lgtm but i see you have it in the backlog but you dont have a +2.. does it look good to you to (i must admit you have a much better eye for detail :D) | 12:42 |
| *** r-daneel has quit IRC | 12:43 | |
| *** r-daneel_ is now known as r-daneel | 12:43 | |
| AJaeger | mnaser: it looked good to me - I wanted to wait for mordred to be around in case it causes problems ;) | 12:44 |
| AJaeger | mnaser: if you're happy, why not +2 - and we merge once more of the team is around? | 12:45 |
| mnaser | AJaeger: sounds good | 12:45 |
| *** smatzek has joined #openstack-infra | 12:49 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add nodejs 8 jobs https://review.openstack.org/532195 | 12:50 |
| *** andreaf has quit IRC | 12:53 | |
| *** andreaf has joined #openstack-infra | 12:53 | |
| *** markvoelker has joined #openstack-infra | 12:54 | |
| *** edmondsw has joined #openstack-infra | 12:55 | |
| *** bobh has joined #openstack-infra | 12:57 | |
| *** nhicher has joined #openstack-infra | 12:57 | |
| *** dtantsur|bbl is now known as dtantsur | 13:01 | |
| *** vsaienk0 has quit IRC | 13:01 | |
| *** beagles is now known as beagles_mtg | 13:01 | |
| *** bobh has quit IRC | 13:02 | |
| *** rhallisey has joined #openstack-infra | 13:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire openstack/anvil https://review.openstack.org/532643 | 13:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix ironic jobs run in nova experimental https://review.openstack.org/532608 | 13:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to noop jobs for evoque projects https://review.openstack.org/532519 | 13:04 |
| *** vsaienk0 has joined #openstack-infra | 13:07 | |
| openstackgerrit | Merged openstack-infra/project-config master: neutron-vpnaas: Add neutron-lib-master periodic job https://review.openstack.org/533141 | 13:07 |
| *** cshastri has quit IRC | 13:07 | |
| *** jaypipes has joined #openstack-infra | 13:08 | |
| *** dprince has joined #openstack-infra | 13:08 | |
| *** trown|outtypewww is now known as trown | 13:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Removed duplicated jobs https://review.openstack.org/533569 | 13:10 |
| *** panda|ruck|lunch is now known as panda|ruck | 13:13 | |
| *** lucas-hungry is now known as lucasagomes | 13:13 | |
| *** adarazs is now known as adarazs_lunch | 13:15 | |
| *** dhajare|BNE|Mtg has quit IRC | 13:16 | |
| *** danpawlik_ has quit IRC | 13:17 | |
| *** cshastri has joined #openstack-infra | 13:18 | |
| *** openstackgerrit has quit IRC | 13:18 | |
| *** olaph1 is now known as olaph | 13:18 | |
| *** alexchadin has quit IRC | 13:19 | |
| *** armaan has quit IRC | 13:21 | |
| *** armaan has joined #openstack-infra | 13:21 | |
| *** r-daneel has quit IRC | 13:21 | |
| *** openstackgerrit has joined #openstack-infra | 13:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove zuul-launchers from nodepool dashboard https://review.openstack.org/533287 | 13:24 |
| *** ramishra_ has quit IRC | 13:25 | |
| mnaser | AJaeger: https://review.openstack.org/#/c/514310/4/zuul.d/zuul-legacy-jobs.yaml -- are we okay with the legacy-tempest-dsvm-neutron-src-plugin getting shuffled? | 13:25 |
| mnaser | i dont mind it but i just wanted to check with you if you usually notice somoe implications by experiene | 13:27 |
| *** markvoelker has quit IRC | 13:28 | |
| *** jpena|lunch is now known as jpena | 13:29 | |
| AJaeger | mnaser: I don't think we have a policy here. In general I avoid any changes, in this case with the large comment, I'm fine. | 13:30 |
| *** janki has quit IRC | 13:31 | |
| *** janki has joined #openstack-infra | 13:32 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in neutron-vpnaas https://review.openstack.org/523368 | 13:33 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove training-labs legacy jobs https://review.openstack.org/530125 | 13:33 |
| mnaser | AJaeger: cool, i went through most of the backlog and reviewed the stuff that are within 'my capacity' :) | 13:33 |
| *** krtaylor has quit IRC | 13:33 | |
| *** lennyb_ has joined #openstack-infra | 13:33 | |
| *** lennyb_ has quit IRC | 13:34 | |
| *** jcoufal has joined #openstack-infra | 13:34 | |
| *** jascott1 has joined #openstack-infra | 13:34 | |
| AJaeger | mnaser: thanks! I added a few more, but this is great progress! | 13:34 |
| mnaser | ill keep going through them also eye out on https://review.openstack.org/#/c/513589/ as the dependency should be merging soono | 13:35 |
| *** Guest18249 has quit IRC | 13:35 | |
| AJaeger | that one needs a rebase - let me do it quickly... | 13:35 |
| *** sree has joined #openstack-infra | 13:36 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove neutron and horizion sphinx jobs https://review.openstack.org/523018 | 13:36 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Keystone https://review.openstack.org/524237 | 13:36 |
| *** armaan has quit IRC | 13:36 | |
| *** armaan has joined #openstack-infra | 13:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy job for heat-templates https://review.openstack.org/513589 | 13:37 |
| *** danpawlik has joined #openstack-infra | 13:37 | |
| *** r-daneel has joined #openstack-infra | 13:38 | |
| *** jascott1 has quit IRC | 13:39 | |
| *** amoralej|lunch is now known as amoralej | 13:40 | |
| mnaser | AJaeger: looks like project-config change needs rebase too | 13:40 |
| AJaeger | mnaser: care to +2 again, please? ^ | 13:40 |
| AJaeger | mnaser: just done ;) | 13:40 |
| *** esberglu has joined #openstack-infra | 13:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy job for heat-templates https://review.openstack.org/513587 | 13:40 |
| mnaser | AJaeger: done thank you | 13:41 |
| *** rlandy has joined #openstack-infra | 13:41 | |
| AJaeger | thanks! | 13:41 |
| *** sree has quit IRC | 13:44 | |
| *** shardy is now known as shardy_afk | 13:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-to-pypi job for heat-tempest-plugin https://review.openstack.org/530216 | 13:45 |
| *** kgiusti has joined #openstack-infra | 13:45 | |
| *** ykarel is now known as ykarel|away | 13:46 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in nova https://review.openstack.org/514310 | 13:48 |
| *** esberglu has quit IRC | 13:48 | |
| *** sree has joined #openstack-infra | 13:48 | |
| *** kgiusti has left #openstack-infra | 13:48 | |
| *** adarazs_lunch is now known as adarazs | 13:50 | |
| *** finucannot is now known as stephenfin | 13:50 | |
| *** kgiusti has joined #openstack-infra | 13:52 | |
| *** dhill_ has joined #openstack-infra | 13:53 | |
| *** beagles_mtg is now known as beagles | 13:55 | |
| *** sree has quit IRC | 13:57 | |
| *** ykarel|away has quit IRC | 13:57 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Skip tempest 35 job for doc and reno only changes https://review.openstack.org/531194 | 13:57 |
| *** krtaylor has joined #openstack-infra | 13:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy job for heat-templates https://review.openstack.org/513587 | 14:00 |
| *** pcichy has joined #openstack-infra | 14:02 | |
| *** jkilpatr has quit IRC | 14:03 | |
| *** eharney has joined #openstack-infra | 14:06 | |
| *** jkilpatr has joined #openstack-infra | 14:06 | |
| AJaeger | mnaser: https://review.openstack.org/513589 recheck - in case you want to +2A already.... | 14:07 |
| *** esberglu has joined #openstack-infra | 14:07 | |
| *** bobh has joined #openstack-infra | 14:08 | |
| *** pgadiya has quit IRC | 14:12 | |
| *** bobh has quit IRC | 14:12 | |
| *** CrayZee1 has joined #openstack-infra | 14:13 | |
| *** CrayZee1 has quit IRC | 14:14 | |
| *** CrayZee1 has joined #openstack-infra | 14:14 | |
| *** andreas_s has quit IRC | 14:15 | |
| *** CrayZee has quit IRC | 14:16 | |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add Grafana URL and API Key to GrafYaml CLI https://review.openstack.org/530949 | 14:17 |
| *** CrayZee1 is now known as CrayZee | 14:18 | |
| *** CrayZee has quit IRC | 14:18 | |
| *** CrayZee has joined #openstack-infra | 14:19 | |
| *** rossella_s has quit IRC | 14:19 | |
| *** andreas_s has joined #openstack-infra | 14:19 | |
| *** rossella_s has joined #openstack-infra | 14:21 | |
| *** r-daneel has quit IRC | 14:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix blazar tag permissions https://review.openstack.org/529709 | 14:23 |
| *** andreas_s has quit IRC | 14:25 | |
| *** andreas_s has joined #openstack-infra | 14:25 | |
| *** andreas__ has joined #openstack-infra | 14:26 | |
| *** makowals has quit IRC | 14:27 | |
| *** makowals has joined #openstack-infra | 14:29 | |
| *** andreas_s has quit IRC | 14:30 | |
| *** camunoz has joined #openstack-infra | 14:33 | |
| honza | AJaeger: thanks for the quick reviews! | 14:37 |
| AJaeger | you're welcome, honza | 14:39 |
| ttx | fungi: I'll need the blazar-release-branch (https://review.openstack.org/#/admin/groups/1857,info) and Telemetry-release-branch (https://review.openstack.org/#/admin/groups/1858,info) owner to be set to 'Release Managers' so that we can manipulate their contents properly. (For reminder, -release-branch groups are used to transition control over stable/$foo from release management to stable maint | 14:43 |
| ttx | around release time | 14:43 |
| ttx | ) | 14:43 |
| *** gcb has joined #openstack-infra | 14:44 | |
| fungi | ttx: thanks, will do that momentarily | 14:45 |
| AJaeger | fungi, are you around so that I can merge https://review.openstack.org/#/c/531936/ ("Use setup.py if we can't get the name from setup.cfg") and https://review.openstack.org/#/c/533059/ ("fedora 27 builds and nodes")? mnaser and myself wanted an infra-root around in case of surprises... | 14:47 |
| fungi | AJaeger: i'll be properly around in a few minutes (though also in a meeting) | 14:48 |
| AJaeger | fungi: ok, will approve one after the other then... Thanks | 14:49 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoint add attendee by summit https://review.openstack.org/533684 | 14:49 |
| AJaeger | Oh, missed that fedora 27 is build on top of the dib pause. Will wait for more people to be around for that | 14:50 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added new endpoint add attendee by summit https://review.openstack.org/533684 | 14:50 |
| *** eharney has quit IRC | 14:52 | |
| *** jkilpatr has quit IRC | 14:54 | |
| *** jkilpatr has joined #openstack-infra | 14:55 | |
| *** yamahata has joined #openstack-infra | 14:59 | |
| mnaser | I’ll check pending reviews shortly. Just landed | 15:02 |
| mnaser | Grabbing some food and taking a break. Our plane landed declaring an emergency heh. | 15:02 |
| AJaeger | emergency? What have you done? | 15:03 |
| mnaser | Nose wheel wouldn’t turn apparently. Tons of firemen and what not after landing | 15:03 |
| AJaeger | ;( | 15:03 |
| mnaser | https://usercontent.irccloud-cdn.com/file/e0rUpQWB/IMG_3672.JPG | 15:03 |
| mnaser | I’ll be more careful with my reviews, hah | 15:03 |
| AJaeger | mnaser: I understand, everything was fine? | 15:03 |
| mnaser | Yes! We’re good now. I have a few hours wait here and leaving again for another 5 hours from Minneapolis but it was an experience that’s for sure lol | 15:04 |
| pabelanger | morning, I'll catch up on backlog later this morning. I have some errands to run and should be back in a few hours | 15:04 |
| *** eharney has joined #openstack-infra | 15:05 | |
| mnaser | The pilot reset the system on the ground and it started working so I guess turn it off and on again is a common fix all over the world | 15:05 |
| *** yamamoto_ has quit IRC | 15:06 | |
| *** makowals has quit IRC | 15:07 | |
| *** makowals has joined #openstack-infra | 15:10 | |
| *** afred312 has quit IRC | 15:11 | |
| *** afred312 has joined #openstack-infra | 15:11 | |
| AJaeger | ;( | 15:12 |
| *** afred312 has quit IRC | 15:12 | |
| *** rosmaita has joined #openstack-infra | 15:12 | |
| *** armaan has quit IRC | 15:13 | |
| *** verdurin has joined #openstack-infra | 15:13 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Retire Evoque project https://review.openstack.org/533698 | 15:14 |
| AJaeger | fungi, could you review https://review.openstack.org/#/c/530101/ (ozj swift3 cleanup) and https://review.openstack.org/#/c/531382 (gerrit ACL change), please? | 15:14 |
| fungi | yup | 15:15 |
| *** verdurin_ has quit IRC | 15:15 | |
| AJaeger | thanks | 15:16 |
| *** armax has quit IRC | 15:17 | |
| *** r-daneel has joined #openstack-infra | 15:17 | |
| fungi | ttx: those group ownerships have been set as requested now | 15:18 |
| corvus | fungi: the slaves report serial 1515959169 but still have the cname | 15:19 |
| ttx | fungi: thx | 15:20 |
| *** links has quit IRC | 15:20 | |
| fungi | corvus: yes, i noticed that when revisiting this morning. i winder if the interleaved signatures make it hard to delete records. i was going to look that up in a bit | 15:21 |
| fungi | s/winder/wonder/ | 15:21 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Retire Evoque project https://review.openstack.org/533698 | 15:22 |
| *** verdurin has quit IRC | 15:24 | |
| *** markvoelker has joined #openstack-infra | 15:25 | |
| *** pcichy has quit IRC | 15:25 | |
| *** pcichy has joined #openstack-infra | 15:26 | |
| *** vsaienk0 has quit IRC | 15:28 | |
| *** hongbin has joined #openstack-infra | 15:29 | |
| *** vsaienk0 has joined #openstack-infra | 15:30 | |
| *** larainema has joined #openstack-infra | 15:33 | |
| *** caphrim007_ has quit IRC | 15:33 | |
| *** r-daneel has quit IRC | 15:34 | |
| *** r-daneel has joined #openstack-infra | 15:35 | |
| corvus | fungi: apparently this will show what's in the binary file: named-compilezone -D -f raw -o - zuul-ci.org /var/lib/bind/zones/zuul-ci.org/zone.db.signed | 15:35 |
| *** verdurin has joined #openstack-infra | 15:37 | |
| fungi | ooh, nice. i haven't seen that command before | 15:38 |
| *** bobh has joined #openstack-infra | 15:38 | |
| *** dsariel has quit IRC | 15:38 | |
| fungi | i was just going to do a authenticated axfr from localhost or something | 15:39 |
| *** slaweq has quit IRC | 15:40 | |
| *** slaweq has joined #openstack-infra | 15:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Pause builds for dib 2.10 release" https://review.openstack.org/532701 | 15:41 |
| openstackgerrit | Merged openstack-infra/project-config master: Add fedora 27 build and nodes https://review.openstack.org/533059 | 15:41 |
| *** andreas__ has quit IRC | 15:42 | |
| *** andreas_s has joined #openstack-infra | 15:43 | |
| *** slaweq has quit IRC | 15:44 | |
| *** andreas_s has quit IRC | 15:46 | |
| *** andreas_s has joined #openstack-infra | 15:46 | |
| fungi | corvus: i also find the "next resign node" line in rndc zonestatus output odd | 15:47 |
| *** armax has joined #openstack-infra | 15:47 | |
| *** dsariel has joined #openstack-infra | 15:48 | |
| corvus | fungi: i agree -- though it could be coincidence, it was likely the last record i added (originally), so it therefore may simply have a resign time offset from the rest | 15:48 |
| corvus | fungi: should "rndc signing -list zuul-ci.org" produce something other than "No signing records found" ? | 15:48 |
| *** felipemonteiro has joined #openstack-infra | 15:50 | |
| fungi | probably? on one of my zones i see "Done signing with key 32139/RSASHA256" and "Done signing with key 61726/RSASHA256" | 15:50 |
| fungi | though some of my other zones on the same nameserver just say "No signing records found" | 15:51 |
| *** gcb has quit IRC | 15:54 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:54 | |
| corvus | fungi: i created keys with '-3' for nsec3 support, but i don't think i issued the "rndc signing -nsec3param 1 0 10 $SALT example.net." command | 15:55 |
| fungi | h, so i've just recreated our issue on one of my nameservers by attempting to replace a cname with an address record | 15:55 |
| corvus | fungi: oh wow | 15:55 |
| fungi | though in my case zonestatus reports the new signed serial while a compilezone from the signed zone has the older serial and the cname | 15:56 |
| corvus | fungi: i wonder if there's some kind of journal flushing involved? | 15:57 |
| *** felipemonteiro has quit IRC | 15:58 | |
| *** markvoelker has quit IRC | 15:59 | |
| fungi | yeah, maybe, because if i query my nameservers (master and slave) i get the new address record not the old cname | 16:00 |
| fungi | so perhaps the records in memory don't match the records in the signed copy on disk | 16:00 |
| fungi | reading up on ixfr vs axfr and the disparity with jnl files | 16:04 |
| CrayZee | Hi, any chance anyone could help me with https://review.openstack.org/#/c/531720/ (in system-config)? | 16:05 |
| *** yamamoto has joined #openstack-infra | 16:07 | |
| fungi | looks like the bind source refers to jnl writebacks as "compacting" | 16:09 |
| *** slaweq has joined #openstack-infra | 16:11 | |
| *** sbra has joined #openstack-infra | 16:11 | |
| *** yamamoto has quit IRC | 16:14 | |
| *** slaweq has quit IRC | 16:16 | |
| *** kjackal has quit IRC | 16:24 | |
| corvus | fungi: maybe this is a bind bug, and maybe we should just clear out the signed file and start over? ttl is only 5m anyway. | 16:25 |
| fungi | i'm still suspecting something got out of sync (though the telltale errors for out of sync jnl files aren't in syslog that i can tell) related to puppet restarting bind when modifying the raw zonefile | 16:25 |
| corvus | fungi: doesn't your local reproduction reduce the likelihood that puppet is involved? | 16:26 |
| fungi | no, i'm not sure i entirely reproduced it | 16:26 |
| *** dave-mccowan has joined #openstack-infra | 16:27 | |
| fungi | what i apparently reproduced is that the signed zone on disk doesn't get updated, but the master and slave nameservers still serve the corect new records | 16:27 |
| fungi | leading me to believe that the difference lies in the signed.jnl file and won't be reflected in the signed file until the journal is compacted | 16:28 |
| fungi | but yeah, i'm wondering if we shouldn't just blow away the generated files and have it regenerate them, then we can also instrument a safer zonefile reload process in hopes that avoids similar issues in the future | 16:28 |
| *** dave-mcc_ has joined #openstack-infra | 16:29 | |
| fungi | and yeah, named-compilezone -j should in theory read the journal | 16:30 |
| corvus | oh ince | 16:30 |
| corvus | fungi: if we're blowing it away anyway, want to try recreating a cname removal real quick? | 16:30 |
| corvus | add it to emergency, remove signed file, manually revert to old copy of zone with cname, reset the slaves to use that, then manually replace the zone file with new version and try the rndc reload method of updating? | 16:31 |
| *** dave-mccowan has quit IRC | 16:32 | |
| *** gema has quit IRC | 16:32 | |
| *** andreas_s has quit IRC | 16:33 | |
| *** andreas_s has joined #openstack-infra | 16:33 | |
| fungi | yeah, can't hurt. we'll want to clear out caches on the slaves to make that happen, i expect | 16:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Temporarily pin yarl while aiohttp is broken https://review.openstack.org/533726 | 16:34 |
| corvus | fungi: okay, i'll start on that -- what's the command you think we should use when we update the zone? | 16:35 |
| fungi | rndc reload (rndc reload zuul-ci.org would be more focused, but just doing a full reload is basically hitless unless the nameserver has a massive number of zones) | 16:36 |
| *** makowals has quit IRC | 16:37 | |
| fungi | i'm still playing around with named-compilezone, and even with -j or -J and the explicit path to the .signed..jnl file i still see the old cname but in a zone with the new serial | 16:37 |
| fungi | however the nameservers themselves aren't serving that cname, so i suspect that the state on disk is lagging behind the state in memory | 16:37 |
| fungi | if that's normal, then perhaps that explains why a restart of the server left it serving old records? | 16:38 |
| *** rossella_s has quit IRC | 16:38 | |
| corvus | fungi: interesting theory, though sounds dangerous :| | 16:38 |
| fungi | yeah, i'm still reading up on whether it needs a separate rndc command to flush updates to disk or something | 16:39 |
| *** shardy_afk is now known as shardy | 16:39 | |
| *** erlon has joined #openstack-infra | 16:39 | |
| *** rossella_s has joined #openstack-infra | 16:40 | |
| *** tesseract has quit IRC | 16:40 | |
| fungi | oh, or i may just be low on caffeine | 16:41 |
| *** pcichy has quit IRC | 16:41 | |
| corvus | fungi: i've started the experiment i described above | 16:41 |
| *** gouthamr has joined #openstack-infra | 16:42 | |
| fungi | yeah, sorry, i switched to checking the wrong record name | 16:42 |
| *** andreas_s has quit IRC | 16:42 | |
| fungi | named-compilezone is showing me the correct/expected state for the record i updated in my test zone | 16:43 |
| fungi | so no, i haven't recreated any part of this failure pattern | 16:43 |
| fungi | things seem to be working like expected on my similarly-configured server | 16:43 |
| corvus | fungi: ah ok | 16:43 |
| corvus | fungi: adns1 has serials 1513902355 (unsigned) and 1513902358 (signed) and ns2 is serving 1513902358. this is the initial stage with cname in the zone. | 16:43 |
| *** gyee has joined #openstack-infra | 16:43 | |
| corvus | fungi: anything else we should inspect before i change the cname to a? | 16:43 |
| *** jpena is now known as jpena|brb | 16:44 | |
| corvus | (ns1 is stopped; we only need one slave for this test) | 16:44 |
| fungi | the unsigned serial is lower than the signed serial? | 16:44 |
| fungi | i would have expected them to be the same integer | 16:45 |
| corvus | fungi: my understanding is that inline signing causes automatic increments of the serial | 16:45 |
| corvus | (why it's 58 and not 56 i don't know) | 16:45 |
| corvus | maybe the initial signing had 3 internal events or something | 16:46 |
| *** sree has joined #openstack-infra | 16:46 | |
| *** caphrim007 has joined #openstack-infra | 16:46 | |
| fungi | strange. it's not doing that on my nameserver, and both seem to be using bind 9.10.3 | 16:47 |
| openstackgerrit | Bernard Cafarelli proposed openstack-infra/release-tools master: bugs-fixed-since: add argument to list StoryBoard entries https://review.openstack.org/533730 | 16:47 |
| corvus | i'm sure i read that somewhere... | 16:47 |
| fungi | i do notice that `rndc signing -list zuul-ci.org` is showing signing completion | 16:47 |
| corvus | fungi: https://kb.isc.org/article/AA-00626/0/Inline-Signing-in-ISC-BIND-9.9.0-Examples.html | 16:47 |
| *** markvoelker has joined #openstack-infra | 16:47 | |
| corvus | fungi: "Note that the serial number" ... | 16:47 |
| *** nicolasbock has quit IRC | 16:48 | |
| fungi | aha! | 16:48 |
| fungi | "If the unsigned zone is updated with a new serial number that's higher than the one in the signed copy, then the signed copy will be increased to match it" | 16:48 |
| fungi | so i suppose the signed will have the same serial or higher | 16:49 |
| fungi | but could be the same | 16:49 |
| corvus | fungi: yeah, i guess it depends on whether bind has changed anything? | 16:49 |
| *** gouthamr has quit IRC | 16:50 | |
| fungi | may have to do with my zone using iso8601-oriented serials | 16:50 |
| *** nicolasbock has joined #openstack-infra | 16:50 | |
| fungi | serial: 2018011500 | 16:50 |
| fungi | signed serial: 2018011500 | 16:50 |
| corvus | fungi: i'd still expect an ...01 if it changed rrsig records | 16:51 |
| fungi | hrm, though no your unsigned serial was presumably already higher than the previous signed serial | 16:51 |
| corvus | fungi: i wiped everything out, that's actually an old serial | 16:51 |
| fungi | yeah, not sure how my nameserver gets away with matching unsigned and signed | 16:51 |
| *** sree has quit IRC | 16:51 | |
| fungi | earlier when i was testing with zuul-ci.org they matched too... old paste: http://paste.openstack.org/show/644247/ | 16:52 |
| fungi | maybe if it needs to resign it increases the serial each time | 16:53 |
| smcginnis | There;s a few release job failures with POST_FAILUREs. Any known issues right now? | 16:53 |
| fungi | re-sign | 16:53 |
| smcginnis | Ah - HTTPError: 503 Server Error: Service Unavailable for url: https://upload.pypi.org/legacy/ | 16:53 |
| smcginnis | Pypi down? | 16:53 |
| fungi | oh joy | 16:53 |
| smcginnis | Hmm, no. Must just be the upload. | 16:53 |
| corvus | fungi: i'll proceed to the next step | 16:54 |
| smcginnis | Though they do not show an issue here: https://status.python.org/ | 16:54 |
| smcginnis | But upload.pypi.org is not there. | 16:54 |
| fungi | corvus: yeah, i'm fine with the notion that the signed zone may sometimes have a slightly higher serial than the unsigned original, and leave the rest of the mystery for another time | 16:54 |
| fungi | smcginnis: nothing posted to distutils-sig about it either (people sometimes mention outages in there) | 16:56 |
| CrayZee | fungi: seems like an issue with the logs copying. See https://review.openstack.org/#/c/491246/, looks like it is stuck for a long time | 16:56 |
| corvus | fungi: adns1 and ns2 both have serial 1515860138 and are serving the a record instead of the cname | 16:57 |
| *** gouthamr has joined #openstack-infra | 16:57 | |
| fungi | corvus: okay, so that matches my experience with the same bind version on debian | 16:57 |
| CrayZee | fungi: this is the task that is stuck http://zuulv3.openstack.org/stream.html?uuid=ed7bccf132b049abb0f63f0da1120eef&logfile=console.log | 16:58 |
| corvus | fungi: named-compilezone shows the cname without the journal, but with -J, it shows the A record | 16:58 |
| corvus | fungi: should i restart bind now and see if it starts serving old data? | 16:59 |
| fungi | corvus: worth a try | 16:59 |
| thingee | these patches have 2 +2's, can someone in infra-core please approve them? https://review.openstack.org/#/q/topic:statusbot-pageurl | 16:59 |
| fungi | corvus: though to recreate what we saw originally, restarting bind after editing the zone without first doing an rndc reload might be interesting to try afterward | 16:59 |
| fungi | smcginnis: have a link to the pypi upload job log with the failure? | 17:01 |
| smcginnis | fungi: http://logs.openstack.org/36/36d923df51e900c1769c8984c7c80ff0414b71b5/release/release-openstack-python/b17dea3/job-output.txt.gz#_2018-01-15_16_52_53_958871 | 17:01 |
| smcginnis | fungi: I get an "Application error" page if I try to go directly to https://upload.pypi.org/ too. | 17:02 |
| corvus | fungi: i did a clean stop then start. compilezone *without the journal* now shows the correct info, and the slave did an axfr and still got the correct info as well. | 17:02 |
| fungi | smcginnis: per https://packaging.python.org/guides/migrating-to-pypi-org/#uploading we're using the current correct url at least | 17:02 |
| corvus | fungi: i think that points pretty solidly at 'corrupted data' as a culprit. | 17:02 |
| fungi | corvus: agreed | 17:02 |
| corvus | fungi: did you make a change to switch to nsdc reload, or should i do that real quick? | 17:03 |
| fungi | rndc reload? i haven't yet if you want to work on that and i'll switch context to looking deeper into the pypi upload failures | 17:04 |
| fungi | i was going to write it last night, and when i came back online to start it i discovered logs.o.o was broken | 17:04 |
| corvus | fungi: heh, right, that one. :) [i said the nsd equivalent]. yeah, i'll do that, thanks. | 17:04 |
| fungi | so ended up getting filesystem recovery going instead | 17:04 |
| fungi | thanks! | 17:05 |
| corvus | fungi: ftr, i did issue the correct command during my test. :) | 17:05 |
| fungi | heh | 17:05 |
| *** armaan has joined #openstack-infra | 17:06 | |
| *** dave-mcc_ has quit IRC | 17:08 | |
| *** andreas_s has joined #openstack-infra | 17:09 | |
| *** trown is now known as trown|lunch | 17:09 | |
| fungi | are the issues with aiohttp trying (and failing) to import yarl.unquote known and being worked on? | 17:10 |
| *** ihrachys has joined #openstack-infra | 17:10 | |
| *** felipemonteiro_ has quit IRC | 17:10 | |
| fungi | ahh, looks like it's hitting zuul and 533726 is the fix there | 17:13 |
| fungi | smcginnis: how many releases do you believe hit that pypi upload error? | 17:14 |
| *** rossella_s has quit IRC | 17:15 | |
| *** rossella_s has joined #openstack-infra | 17:16 | |
| *** janki has quit IRC | 17:17 | |
| smcginnis | fungi: I show 5 failures. Is that what you saw dhellmann? | 17:18 |
| fungi | smcginnis: i notice that https://upload.pypi.org/legacy/ is no longer returning an error to my browser, so it was presumably intermittent | 17:18 |
| smcginnis | fungi: Could we get those requeued? | 17:19 |
| *** andreas_s has quit IRC | 17:19 | |
| *** efried is now known as efried_rollin | 17:19 | |
| fungi | i can reenqueue those tags into the release pipeline as long as they didn't get uploaded to pypi, yes | 17:19 |
| fungi | they'll end up with regenerated tarballs.o.o uploads and so different checksums/signatures than the first time around, but presumably the releases site will update to take that into account? | 17:19 |
| *** cshastri has quit IRC | 17:20 | |
| *** links has joined #openstack-infra | 17:21 | |
| *** cshastri has joined #openstack-infra | 17:21 | |
| smcginnis | fungi: I believe it should be OK. I think it will end up updating what's published there. | 17:22 |
| smcginnis | fungi: These are the failures I saw: http://paste.openstack.org/show/645288/ | 17:22 |
| smcginnis | captcha on paste.o.o is killer. :) | 17:23 |
| *** e0ne has quit IRC | 17:23 | |
| fungi | the captcha only kicks in if you post a bunch of urls | 17:23 |
| fungi | (also, if you use a tool like pastebinit you can get around to needing to do the captcha) | 17:24 |
| smcginnis | Yeah, first time I hit that. And the images were so bad it took me a few attempts to actually get it through. | 17:24 |
| fungi | yeesh | 17:24 |
| fungi | cool, i'll get started reenqueuing these in a moment | 17:24 |
| smcginnis | That or my eyes are getting worse than I thought, which very well may be. | 17:24 |
| smcginnis | fungi: Thanks! | 17:24 |
| *** vsaienk0 has quit IRC | 17:26 | |
| *** jpena|brb is now known as jpena | 17:27 | |
| *** lucasagomes is now known as lucas-afk | 17:30 | |
| CrayZee | Infra: I have a task that is stuck for a very long time now. Could someone please take a look? It seems to be stuck on some zuul cleanup code.... http://zuulv3.openstack.org/stream.html?uuid=ed7bccf132b049abb0f63f0da1120eef&logfile=console.log | 17:33 |
| *** jamesmcarthur has joined #openstack-infra | 17:35 | |
| *** andreas_s has joined #openstack-infra | 17:36 | |
| *** jpich has quit IRC | 17:46 | |
| *** jamesmcarthur has quit IRC | 17:46 | |
| *** pcichy has joined #openstack-infra | 17:47 | |
| *** andreas_s_ has joined #openstack-infra | 17:47 | |
| *** links has quit IRC | 17:50 | |
| *** andreas_s has quit IRC | 17:50 | |
| *** pcichy has quit IRC | 17:50 | |
| *** hemna_ has joined #openstack-infra | 17:51 | |
| EmilienM | we have some post-failures now: | 17:53 |
| EmilienM | SSH Error: data could not be sent to remote host "166.78.99.88". Make sure this host can be reached over ssh | 17:53 |
| EmilienM | http://logs.openstack.org/80/533380/8/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/cb17b17 | 17:53 |
| EmilienM | is that expected? | 17:53 |
| *** kjackal has joined #openstack-infra | 17:54 | |
| fungi | EmilienM: any correlation to specific providers? that one looks like it was in rax-ord | 17:55 |
| *** cshastri has quit IRC | 17:55 | |
| fungi | EmilienM: looks like there was a run failure before the post failures according to ara | 17:55 |
| *** vsaienk0 has joined #openstack-infra | 17:55 | |
| *** rossella_s has quit IRC | 17:55 | |
| fungi | but same thing, the executor seems to have decided it was unable to reach the job node for whatever the next task was after the zuul-cloner task | 17:56 |
| EmilienM | another failure on rax-iad | 17:57 |
| fungi | which suggests to me that something happened in the provider to cause the node to just go away | 17:57 |
| fungi | or else we've got network connectivity issues | 17:57 |
| EmilienM | so recheck? | 17:57 |
| *** jamesmca_ has joined #openstack-infra | 17:57 | |
| fungi | https://rackspace.service-now.com/system_status/ is all green | 17:57 |
| *** rossella_s has joined #openstack-infra | 17:58 | |
| fungi | EmilienM: well, yes probably not a lot we can do on that besides suck it up and recheck, but it is curious that you got a cluster of multiple failures like that | 17:58 |
| fungi | going to check on whether they opened any tickets for that tenant | 17:59 |
| fungi | no new tickets for the openstackjenkins tenant in rackspace either | 17:59 |
| *** camunoz has quit IRC | 18:00 | |
| EmilienM | fungi: it also happens on puppet jobs https://review.openstack.org/#/c/533528/ | 18:01 |
| EmilienM | puppet-openstack-beaker-centos-7 puppet-openstack-beaker-centos-7 : NODE_FAILURE | 18:01 |
| fungi | huh, i've never seen NODE_FAILURE results before | 18:02 |
| fungi | infra-root: ^ that's disconcerting | 18:02 |
| *** derekh has quit IRC | 18:02 | |
| fungi | these are node request failures, according to the zuul source? | 18:03 |
| *** armaan has quit IRC | 18:04 | |
| corvus | fungi: yeah, it's probably most likely to happen if we have 3 subsequent launch failure attempts in a single provider | 18:04 |
| clarkb | but also possible if a cloud has an outage. I am still working on a fix for this case (trying to get a test working but its somewhat convoluted) | 18:05 |
| *** vsaienk0 has quit IRC | 18:05 | |
| fungi | the scheduler is logging a lot of "Exception: Node <Node 0001895098 ['ubuntu-xenial']:ubuntu-xenial> is not locked" | 18:05 |
| *** jkilpatr has quit IRC | 18:05 | |
| fungi | and by "a lot" i mean pretty much continuously | 18:06 |
| *** ralonsoh has quit IRC | 18:07 | |
| *** pcichy has joined #openstack-infra | 18:07 | |
| corvus | oh we're probably out of memory then | 18:07 |
| fungi | oh, yeah, 6+gb into swap | 18:08 |
| corvus | does maybe someone want to rebuild the server? | 18:08 |
| fungi | i can switch gears and start launching a replacement, but happy to yield to other volunteers on that | 18:09 |
| corvus | infra-root: ^ any other volunteers? | 18:10 |
| *** andreas_s_ has quit IRC | 18:11 | |
| clarkb | I'm sort of not here today due to sick family so I'm not a good choice | 18:11 |
| clarkb | am trying to figure out this test while I have time though | 18:11 |
| corvus | i guess i'll restart zuul and we'll lose the queues. i don't think dmsimard's cron patch has made it out of WIP yet. | 18:14 |
| *** sambetts is now known as sambetts|afk | 18:14 | |
| fungi | should the zuulv3.o.o replacement be named zuul.openstack.org? (or i guess zuul01.openstack.org?) | 18:15 |
| *** jamesmca_ has quit IRC | 18:16 | |
| corvus | fungi: i guess zuul01.openstack.org -- but that'll need some system-config changes first i think | 18:16 |
| *** jamesmcarthur has joined #openstack-infra | 18:16 | |
| fungi | right, new host group for the scheduler server | 18:16 |
| *** dtantsur is now known as dtantsur|afk | 18:16 | |
| fungi | i'll get that worked out | 18:16 |
| corvus | and new regex host entry in site.pp | 18:16 |
| fungi | yep | 18:17 |
| corvus | which should probably just be a copy of zuulv3's entry for now, until we get the openstack stuff worked out | 18:17 |
| fungi | agreed | 18:17 |
| *** jpena is now known as jpena|off | 18:18 | |
| *** jkilpatr has joined #openstack-infra | 18:18 | |
| *** trown|lunch is now known as trown | 18:19 | |
| *** kmalloc has joined #openstack-infra | 18:19 | |
| dhellmann | smcginnis, fungi : in addition to those 5 failures, the novaclient announce job failed. I can run that by hand. | 18:20 |
| mnaser | i know folks are dealing with zuul stuff but regarding reviews to retire projects, is it ok to approve them if the "final retiring commit" has not been merged into the project? | 18:20 |
| *** Swami_ has joined #openstack-infra | 18:20 | |
| *** gouthamr has quit IRC | 18:20 | |
| fungi | dhellmann: thanks, i held off reenqueuing any of those yet anyway what with the current zuul issues (hopefully under control in a bit) | 18:20 |
| mnaser | (my gut is to merge them in case we are never able to find someone to merge those final retiring commits) | 18:21 |
| mnaser | s/merge/approve/ | 18:21 |
| dhellmann | fungi : makes sense | 18:21 |
| *** caphrim007_ has joined #openstack-infra | 18:21 | |
| *** corvus is now known as jeblair | 18:21 | |
| jeblair | #status notice Zuul has been restarted and has lost queue contents; changes in progress will need to be rechecked. | 18:21 |
| openstackstatus | jeblair: sending notice | 18:21 |
| *** jeblair is now known as corvus | 18:21 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run rndc reload instead of restarting bind https://review.openstack.org/533753 | 18:22 |
| -openstackstatus- NOTICE: Zuul has been restarted and has lost queue contents; changes in progress will need to be rechecked. | 18:22 | |
| corvus | fungi: ^ when you have a sec | 18:22 |
| *** caphrim007_ has quit IRC | 18:23 | |
| *** caphrim007_ has joined #openstack-infra | 18:24 | |
| openstackstatus | jeblair: finished sending notice | 18:24 |
| *** caphrim007 has quit IRC | 18:24 | |
| mnaser | corvus: does reloading named/bind reload rndc? | 18:25 |
| mnaser | could you not has hasreload => true to the service definition | 18:25 |
| fungi | mnaser: we could, but we don't define the service | 18:25 |
| fungi | it's provided by theforeman/puppet-dns | 18:25 |
| *** rossella_s has quit IRC | 18:25 | |
| mnaser | aaah | 18:26 |
| fungi | and is set up as a service-agnostic generalization | 18:26 |
| fungi | and uses hasrestart=>true | 18:26 |
| *** rossella_s has joined #openstack-infra | 18:26 | |
| mnaser | fungi: ah gotcha, well in that case that patch is good :D | 18:27 |
| fungi | which ends up calling rndc stop and then starting the service fresh (based on my reading of the systemd unit on xenial anyway) | 18:27 |
| mnaser | yeah thats not a ffun time | 18:27 |
| fungi | it's a hidden master, so the brief outage for the service isn't necessarily troublesome, but i suspect the weird issue we saw with inline signed records being out of sync with the master zone could be related | 18:28 |
| openstackgerrit | Merged openstack-infra/project-config master: grafana: Update networking-midonet https://review.openstack.org/530220 | 18:29 |
| fungi | especially since we couldn't recreate the problem when using a normal bind editing and reloading workflow | 18:30 |
| *** ekcs has quit IRC | 18:31 | |
| *** rossella_s has quit IRC | 18:31 | |
| *** ekcs has joined #openstack-infra | 18:31 | |
| *** Goneri has joined #openstack-infra | 18:31 | |
| *** CrayZee has quit IRC | 18:32 | |
| *** rossella_s has joined #openstack-infra | 18:32 | |
| fungi | corvus: looking through `git grep zuulv3` results, are we going to need to rebuild our certificate authority? looks like we used CN=zuulv3.openstack.org for the current ca | 18:33 |
| fungi | we could in theory continue using that one and just live with the fact that the cn refers to a (eventually) removed server | 18:34 |
| *** felipemonteiro has joined #openstack-infra | 18:34 | |
| corvus | fungi: i think it'll be safe to just ignore that | 18:35 |
| *** felipemonteiro_ has joined #openstack-infra | 18:35 | |
| fungi | i'll leave it alone for now | 18:36 |
| fungi | though i suppose we may need to issue a new server certificate if zk is picky about the cert matchign the fqdn | 18:37 |
| *** rossella_s has quit IRC | 18:37 | |
| corvus | fungi: i think we're only using that for gearman right now? | 18:38 |
| fungi | oh, that was gearman not zk? | 18:38 |
| EmilienM | why when zuul is restarted the queue content is lost? | 18:38 |
| *** myoung|pto is now known as myoung | 18:39 | |
| corvus | EmilienM: because there's no persistent storage of queues. | 18:39 |
| fungi | EmilienM: because it was so deep into swap utilization that we couldn't reliably interrogate the zuul scheduler to get a list of what changes it was testing | 18:39 |
| *** felipemonteiro has quit IRC | 18:39 | |
| fungi | and, yes, what corvus said (it's retained in-memory in the daemon) | 18:40 |
| EmilienM | and are there any plans to change that? | 18:41 |
| *** rossella_s has joined #openstack-infra | 18:41 | |
| EmilienM | because asking projects to do 'recheck' isn't scaling much when dealing with hundred of patches | 18:41 |
| corvus | EmilienM: yes, but not soon. i understand it's asking a lot. i'm not happy about it either. | 18:42 |
| EmilienM | ok np, just making sure there is a plan, sometimes :D | 18:42 |
| corvus | in the mean time, we're increasing the size of the server to try to reduce the likelihood we need to do this so we have time to find ways to reduce the memory usage. | 18:43 |
| dmsimard | Sorry, I haven't had time to circle back to the status.json backup patch -- I'll look now. | 18:46 |
| fungi | corvus: does https://docs.openstack.org/infra/system-config/github.html#openstack-zuul-app imply that users of the github web app are going to need to update their configs too? | 18:46 |
| corvus | fungi: we control that -- it's defined in the openstack-infra github org, so we can make that change when the new server is up | 18:47 |
| *** e0ne has joined #openstack-infra | 18:47 | |
| fungi | oh, i think i'm misreading and this is referring to hiera file paths | 18:47 |
| corvus | fungi: well, we will need to make a change on this page: https://github.com/organizations/openstack-infra/settings/apps/openstack-zuul | 18:48 |
| corvus | but honestly, it's not a big deal at this point -- we can just make that change once the new server is in prod | 18:49 |
| *** vsaienk0 has joined #openstack-infra | 18:49 | |
| corvus | (i'm assuming we'll have a zuulv3.o.o cname for a bit, which would probably do the trick as well) | 18:50 |
| fungi | yeah, for that matter the zuul.o.o cname can fairly easily be switched to from zuulv3 to zuul01 when the time comes, so we could set it to use that ahead of time | 18:51 |
| *** shardy has quit IRC | 18:51 | |
| corvus | fungi: oh indeed, i'll just do it now | 18:51 |
| *** dsariel has quit IRC | 18:51 | |
| *** kjackal has quit IRC | 18:52 | |
| pabelanger | and back | 18:52 |
| fungi | i'm updating a lot of references in system-config docs to use that url instead of zuulv3 | 18:52 |
| corvus | github app updated | 18:53 |
| *** rmcall has joined #openstack-infra | 18:55 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Periodically retrieve and back up Zuul's status.json https://review.openstack.org/532955 | 18:57 |
| dmsimard | I addressed ianw's comment in ^ | 18:57 |
| pabelanger | fungi: corvus: let me know how a can help with zuulv3 rebuild | 18:57 |
| fungi | pabelanger: i'll have a fairly largeish system-config change you can review in moments | 18:57 |
| pabelanger | fungi: ack | 18:58 |
| corvus | #status log updated github zuul app to use new hostname: zuul.openstack.org | 18:58 |
| *** corvus is now known as jeblair | 18:58 | |
| jeblair | #status log updated github zuul app to use new hostname: zuul.openstack.org | 18:58 |
| openstackstatus | jeblair: finished logging | 18:58 |
| *** jeblair is now known as corvus | 18:58 | |
| *** vsaienk0 has quit IRC | 18:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Allow corvus to use statusbot https://review.openstack.org/533758 | 18:59 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Temporarily pin yarl while aiohttp is broken https://review.openstack.org/533726 | 19:04 |
| AJaeger | mnaser: regarding retiring: There was a depends-on - we do not approve before the repos are retired | 19:08 |
| mnaser | AJaeger: didnt spot that, that would have answered my question, thank you | 19:09 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: create goal-tools repository https://review.openstack.org/533761 | 19:09 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add zuul settings for goal-tools repository https://review.openstack.org/533762 | 19:09 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a zuul01.openstack.org https://review.openstack.org/533763 | 19:11 |
| fungi | pabelanger: corvus: ^ | 19:11 |
| corvus | on it | 19:11 |
| *** rfolco_nho has quit IRC | 19:12 | |
| pabelanger | looking | 19:13 |
| pabelanger | fungi: left comment | 19:15 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Periodically retrieve and back up Zuul's status.json https://review.openstack.org/532955 | 19:16 |
| dmsimard | corvus: ^ slight tweak, escaped the "%" sign in the date/timestamp format | 19:16 |
| *** felipemonteiro__ has joined #openstack-infra | 19:17 | |
| *** electrofelix has quit IRC | 19:17 | |
| *** felipemonteiro_ has quit IRC | 19:20 | |
| corvus | fungi, pabelanger: why are we backing up zuulv3 at all? | 19:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Configure tripleo-ui to use nodejs8 jobs https://review.openstack.org/532197 | 19:20 |
| corvus | oh right! | 19:20 |
| corvus | i remember, keys :) | 19:20 |
| corvus | secrets | 19:20 |
| corvus | fungi: which we will want to copy over | 19:20 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Periodically retrieve and back up Zuul's status.json https://review.openstack.org/532955 | 19:21 |
| corvus | fungi, pabelanger: we could either keep the same bup user, or start a new backup for the new host. | 19:21 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Periodically retrieve and back up Zuul's status.json https://review.openstack.org/532955 | 19:22 |
| corvus | fungi, pabelanger: well want to make sure to copy /var/lib/zuul/keys and /var/lib/zuul/times to the new server | 19:22 |
| *** andreas_s has joined #openstack-infra | 19:23 | |
| pabelanger | Ah, I forgot about that | 19:25 |
| *** pcichy has quit IRC | 19:25 | |
| *** pcichy has joined #openstack-infra | 19:25 | |
| *** jamesmcarthur has quit IRC | 19:27 | |
| openstackgerrit | Merged openstack-infra/system-config master: Run rndc reload instead of restarting bind https://review.openstack.org/533753 | 19:28 |
| fungi | corvus: pabelanger: i'll also readily admit that i wasn't sure which of those parts of the proposed change are safe to do now and which ones might have to wait until cut-over (gearman stuff maybe?) though i did at least avoid changing things i knew would disrupt operation for zuulv3.o.o right now | 19:29 |
| fungi | but half expect i'll need to split that change into at least a couple | 19:29 |
| *** olaph has quit IRC | 19:29 | |
| fungi | and we'll need at least one more patch to merge when cutting over obviously | 19:30 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Revert "Revert "Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header"" https://review.openstack.org/514489 | 19:31 |
| *** olaph has joined #openstack-infra | 19:31 | |
| dmsimard | clarkb, pabelanger: ^ | 19:31 |
| corvus | fungi: afaict, that change should be okay to merge now -- we'll need to coordinate another change to update the merge/executor configs. | 19:36 |
| corvus | i can't think of anything else that shouldn't change | 19:36 |
| *** camunoz has joined #openstack-infra | 19:37 | |
| *** andreas_s has quit IRC | 19:38 | |
| *** e0ne has quit IRC | 19:38 | |
| *** jamesmcarthur has joined #openstack-infra | 19:39 | |
| fungi | i went ahead and added the corresponding hiera group file on the puppetmaster before pushing the change, so should be all set i guess? | 19:39 |
| *** e0ne has joined #openstack-infra | 19:45 | |
| *** rossella_s has quit IRC | 19:48 | |
| *** rossella_s has joined #openstack-infra | 19:51 | |
| *** ekcs has quit IRC | 19:51 | |
| *** ekcs has joined #openstack-infra | 19:52 | |
| *** jkilpatr has quit IRC | 19:55 | |
| *** efried_rollin is now known as efried | 19:55 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Add test_launcher test https://review.openstack.org/533771 | 19:56 |
| *** pcichy has quit IRC | 19:58 | |
| *** amoralej is now known as amoralej|off | 20:01 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: create goal-tools repository https://review.openstack.org/533761 | 20:02 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add zuul settings for goal-tools repository https://review.openstack.org/533762 | 20:02 |
| corvus | fungi: looks like we need to drop the parts of that change that add iptables entries for zuul01 until the server exist | 20:06 |
| fungi | ahh, yeah i thought something like that might end up being the case | 20:07 |
| fungi | also worried about the cacti addition when the server's not in existence yet | 20:07 |
| fungi | i can split those into a child commit and wip them | 20:07 |
| corvus | fungi: cool | 20:07 |
| dmsimard | mordred: did you ever figure out that include out of workspace thing ? Hitting it now, I'll take a good look. | 20:10 |
| dmsimard | kinda hard to troubleshoot without keep though | 20:10 |
| fungi | huh... just noticed our zuul vars don't include the trigger value | 20:12 |
| corvus | fungi: i'm not parsing that | 20:12 |
| fungi | i was whipping up a quick poc wrapper to digest zuul-info/inventory.yaml from a job run and spit out the parameters for zuul enqueue{,-ref} | 20:13 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Use setup.py if we can't get the name from setup.cfg https://review.openstack.org/531936 | 20:14 |
| fungi | going to hard-code --trigger=gerrit for now because i don't see the actual trigger as an ansible var | 20:14 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Add test_launcher test https://review.openstack.org/533771 | 20:14 |
| corvus | fungi: i think that's the way to go, i'm not sure we want to add trigger info to zuul vars | 20:14 |
| *** slaweq has joined #openstack-infra | 20:15 | |
| corvus | (if anything grows a dependency on that, then it impacts our ability to enqueue things via other triggers) | 20:15 |
| fungi | yeah, i can imagine people making some poor decisions based on trigger='periodic' or whatever | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| corvus | fungi: we should actually look at whether we can remove that trigger arg to enqueue-ref -- i think perhaps after i finish up with the cross-source deps, we'll be close enough to our ideal data model to be able to do that. | 20:16 |
| corvus | (with my change, zuul should be able to always figure out what connection to use for a given ref, that may be the last thing that enqueue-ref relies on the trigger name for) | 20:17 |
| fungi | that would be nifty | 20:17 |
| dhellmann | I'm trying to figure out the expected setup for a zuul job to run a script that is part of the role definition in order to get https://review.openstack.org/#/c/524862/ working. Do we have an example of something like that I can refer to? | 20:17 |
| dmsimard | fungi: the best you have is zuul.change_url (for the trigger) | 20:17 |
| dhellmann | I think the step we're missing is to either copy the script somewhere, or to refer to it by its full path within the role. I don't know where to copy it for the former and I don't know the path for the latter. | 20:18 |
| corvus | dmsimard: that's only for changes, that won't be there for ref-updated items | 20:18 |
| fungi | dmsimard: even that doesn't contain the trigger string (in this case, "gerrit") | 20:18 |
| dmsimard | yeah, like I said, it's the best option given the current circumstances :/ | 20:18 |
| tobiash | dhellmann: you mean run a script which is stored in a variable? | 20:19 |
| corvus | dhellmann: looking | 20:19 |
| dhellmann | tobiash : no, the role has a script under the files directory and I want to execute that script | 20:19 |
| dhellmann | I can obviously move it somewhere else if there is a more correct place to put it within the role | 20:19 |
| tobiash | dhellmann: then use the copy module | 20:19 |
| dhellmann | ok. where should I copy it? :-) | 20:19 |
| tobiash | src is relative to the role root | 20:20 |
| corvus | or perhaps the 'script' module? | 20:20 |
| tobiash | and you can freely choose dest on the target node | 20:20 |
| dhellmann | corvus : well, it's a rather long python script and it's useful to be able to run it by hand, too, so I didn't want to embed it in the yaml | 20:20 |
| corvus | http://docs.ansible.com/ansible/latest/script_module.html | 20:20 |
| corvus | that copies a script to the remote node into a tmp location and executes it | 20:21 |
| dhellmann | is it safe to just copy it to /tmp or something? | 20:21 |
| tobiash | dhellmann: ups, copy is relative to the files folder in the role | 20:21 |
| dhellmann | I don't know how these nodes are laid out at runtime or what sorts of variables I have access to | 20:21 |
| dhellmann | yeah, I know *how* to copy it but i don't know *where* to put the copy | 20:21 |
| dhellmann | like if I wanted to copy it into the workspace, is there a variable with that path in it? | 20:21 |
| *** e0ne has quit IRC | 20:21 | |
| tobiash | I would choose /tmp/myscript.sh or something like that | 20:22 |
| corvus | dhellmann: this is running on a remote nodepool-supplied node? | 20:22 |
| tobiash | or just ./myscript.sh | 20:22 |
| *** dansmith has quit IRC | 20:22 | |
| *** markvoelker has quit IRC | 20:22 | |
| dhellmann | corvus : I don't know what all of those words mean. It's going to run in the doc jobs after the html is built to build a sitemap file | 20:22 |
| dhellmann | tobiash : I don't like using "." with ansible after a recent upgrade made `pwd` different in different cases, so I'd rather just pick a full path. If /tmp is safe I can use that. | 20:23 |
| corvus | dhellmann: nodepool supplies nodes for zuul to use. most of our jobs run on these nodes. this is one of those jobs. | 20:23 |
| tobiash | corvus: oh cool, the script module is more elegant than copy and execute | 20:23 |
| dhellmann | corvus : ok, that's what I thought but I didn't want to assume | 20:24 |
| corvus | tobiash: yep. it's for exactly this case. | 20:24 |
| tobiash | never used it before | 20:24 |
| tobiash | so this would be the better option | 20:24 |
| corvus | zuul v2.5 used it extensively | 20:24 |
| dhellmann | how can I refer to the script that's in the role? | 20:24 |
| dhellmann | the examples all use full paths | 20:25 |
| *** jamesmcarthur has quit IRC | 20:25 | |
| fungi | smcginnis: dhellmann: the failures listed at http://paste.openstack.org/show/645288/ have all had their refs reenqueued | 20:25 |
| dhellmann | fungi : thanks! | 20:25 |
| corvus | dhellmann: the pwd might be the role... or it might be the playbook. i'm not sure. | 20:26 |
| *** e0ne has joined #openstack-infra | 20:27 | |
| tobiash | dhellmann: ansible has the role_path variable which should give you the path of the role | 20:27 |
| tobiash | http://docs.ansible.com/ansible/latest/playbooks_variables.html#magic-variables-and-how-to-access-information-about-other-hosts | 20:28 |
| dhellmann | tobiash : cool, so I should be able to say something like script: {{role_path}}/files/sitemap_gen.py ... | 20:28 |
| dhellmann | tobiash, corvus : thank you | 20:28 |
| corvus | ah cool, so building on {role_path}/files/... should work yeah? | 20:28 |
| smcginnis | Thanks fungi | 20:28 |
| tobiash | I think so | 20:28 |
| dhellmann | I'll give that a try | 20:28 |
| corvus | dhellmann, tobiash: one last thing -- this may be a good candidate to turn that script into an ansible module -- the advantage there is better structured reporting (via tools like ara, etc). | 20:29 |
| dhellmann | yeah. I want to get it working before we add that level of complexity. | 20:29 |
| AJaeger | dmsimard: see my comments on https://review.openstack.org/#/c/532304 - does that help with your change https://review.openstack.org/514489 ? | 20:30 |
| corvus | dhellmann: i hear you -- though ironically, that takes care of all the 'copy onto remote node and execute' bits for you :) -- but yeah, given that it's so close already, it probably makes the most sense to finish it up this way, then look at making it a module (assuming that the copy+execute stuff doesn't turn into a time sink) | 20:30 |
| dmsimard | AJaeger: mordred and I first noticed that issue last week I think. We don't understand what's going on yet. | 20:31 |
| dhellmann | yeah, this started out as a 1 line bash script and evolved | 20:31 |
| AJaeger | dmsimard: I left some comments | 20:32 |
| dmsimard | AJaeger: we experimented a bit in a different patch, sec | 20:32 |
| AJaeger | dmsimard: "Is role_path not set correctly? Or is the problem that the base-minimal change does not use configure-unbound and thus role_path is not set here?" | 20:32 |
| dmsimard | role_path is dynamically set by ansible when running in the context of a role | 20:32 |
| AJaeger | dmsimard: one difference between base and base-minimal is that base-minimal does *not* include configure-unbound | 20:33 |
| AJaeger | dmsimard: and the test fails in configure-unbound... | 20:33 |
| AJaeger | dmsimard: didn't check your change - just mordred's | 20:33 |
| dmsimard | AJaeger: so we have https://review.openstack.org/#/c/532314/ and https://review.openstack.org/#/c/532319/ | 20:33 |
| dmsimard | the errors don't make sense, I suspect there's something going on around how zuul provides "external" roles implicitely but I need to search through the directory tree of a zuul build to witness what's going on. | 20:34 |
| AJaeger | dmsimard: yeah, this is really strange... | 20:35 |
| *** Guest49522 has joined #openstack-infra | 20:35 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a zuul01.openstack.org https://review.openstack.org/533781 | 20:35 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Update firewalls, zookeeper and cacti for zuul01 https://review.openstack.org/533782 | 20:35 |
| fungi | corvus: pabelanger: split version ^ | 20:35 |
| pabelanger | looking | 20:36 |
| fungi | hopefully that works out a little better | 20:36 |
| clarkb | fungi: pabelanger is this server siwtch a good time to change zk servers? | 20:37 |
| AJaeger | dmsimard: could you review https://review.openstack.org/#/c/529339 , please? | 20:37 |
| fungi | clarkb: umm... maybe? i really hadn't considered it | 20:38 |
| fungi | clarkb: does doing them both around the same timeframe make things easier? | 20:38 |
| pabelanger | clarkb: fungi: I think we need to first switch nodepool-builder to new zookeepre | 20:39 |
| pabelanger | let that run for a day, so images get populated | 20:39 |
| fungi | pabelanger: before replacing the zuul scheduler server, or before changing zk servers (presumably the latter)? | 20:40 |
| clarkb | just thinking in thus case we could copy the data files from the old zk server | 20:40 |
| *** Guest49522 is now known as dansmith | 20:40 | |
| clarkb | if we did that then we dont need to prep imaves ahead of time right? | 20:40 |
| pabelanger | I mean, we could copy data from zookeeper on nodepool.o.o, is that going to be too much? | 20:41 |
| clarkb | but I'm also not really here because esick | 20:41 |
| *** jamesmcarthur has joined #openstack-infra | 20:42 | |
| pabelanger | fungi: will defer to you, if we also want to do zookeeper now, it will be more moving parts. | 20:43 |
| clarkb | we would stop zk on old server, copy files to new servers, start one at a time then update client condiga | 20:44 |
| clarkb | *configs | 20:44 |
| clarkb | thought of it because requires a zuul restart anyways | 20:44 |
| clarkb | but it is more work | 20:45 |
| *** verdurin_ has joined #openstack-infra | 20:45 | |
| pabelanger | we've also not tested clustering yet in zookeeper, so this would be first time usage in production | 20:45 |
| fungi | pabelanger: clarkb: yeah, i'm more keen to change as few things at a time as possible so if something goes sideways we have a better shot at diagnosing what caused it quickly | 20:46 |
| fungi | this is a very large, very complex distributed system after all | 20:46 |
| pabelanger | yah | 20:47 |
| *** sree has joined #openstack-infra | 20:47 | |
| *** verdurin has quit IRC | 20:47 | |
| *** sree has quit IRC | 20:52 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation https://review.openstack.org/524862 | 20:52 |
| *** jkilpatr has joined #openstack-infra | 20:53 | |
| dmsimard | infra-root: Is it generally okay to enable keep on all executors for a few minutes ? https://review.openstack.org/#/c/514489/ fails quickly and is symptomatic of a problem mordred and I have been hunting down | 20:55 |
| *** jamesmcarthur has quit IRC | 20:56 | |
| *** jamesmcarthur has joined #openstack-infra | 20:57 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Use new firewall syntax for zookeeper cluster https://review.openstack.org/533791 | 20:58 |
| fungi | dmsimard: i would keep an extremely close eye on filesystem utilization if you do | 21:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix blazar tag permissions https://review.openstack.org/529709 | 21:02 |
| *** vivsoni_ has joined #openstack-infra | 21:03 | |
| *** dprince has quit IRC | 21:06 | |
| *** vivsoni has quit IRC | 21:06 | |
| *** e0ne_ has joined #openstack-infra | 21:07 | |
| corvus | fungi: first change +3 | 21:07 |
| fungi | thanks corvus! i will try to get time to spin up the new server this evening | 21:07 |
| corvus | i'm going to remove the nameservers from emergency | 21:08 |
| fungi | sounds good | 21:08 |
| *** e0ne has quit IRC | 21:09 | |
| *** jamesmcarthur has quit IRC | 21:09 | |
| dmsimard | fungi: ack, I will enable it and disable it as soon as I get something back -- should take no longer than 5 minutes | 21:10 |
| *** flwang has quit IRC | 21:11 | |
| *** e0ne has joined #openstack-infra | 21:14 | |
| *** e0ne_ has quit IRC | 21:15 | |
| ianw | pabelanger: thanks, looks like new images are all out there. i'll work on moving the f27 devstack jobs into tree maybe ... and think about how to get rid of f26. few users for python3.6 i think | 21:16 |
| pabelanger | ianw: cool, I have noticed any issues with new images so far | 21:19 |
| *** jamesmcarthur has joined #openstack-infra | 21:22 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation https://review.openstack.org/524862 | 21:23 |
| *** sshnaidm is now known as sshnaidm|afk | 21:23 | |
| dmsimard | I just disabled keep on the executors, it was enabled for ~9 minutes | 21:23 |
| dhellmann | corvus , tobiash : do either of you have any tips on how to deal with this linter issue in that role? http://logs.openstack.org/62/524862/11/check/openstack-zuul-jobs-linters/8861016/ara/result/fb415ac6-389a-4548-bd98-3639336211e4/ | 21:24 |
| fungi | oh, oops, just noticed i forgot to preserve the gerrit change id for 533763 when i split it. will abandon the old one | 21:24 |
| *** flwang has joined #openstack-infra | 21:25 | |
| *** jamesmcarthur has quit IRC | 21:26 | |
| *** nicolasbock has quit IRC | 21:26 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation https://review.openstack.org/524862 | 21:26 |
| *** dbecker has quit IRC | 21:27 | |
| *** dbecker has joined #openstack-infra | 21:27 | |
| pabelanger | dhellmann: you don't need to quota each variable, you should be able to just single quota the complete line. | 21:27 |
| *** caphrim007 has joined #openstack-infra | 21:27 | |
| dhellmann | pabelanger : oh? that ansible module seems to want the first argument on the line to be separate because it uses that as a filename to copy somewhere | 21:28 |
| pabelanger | dhellmann: not sure, but you should be able to do script: "foo.sh {{ var_a }] {{ var_b }} | 21:29 |
| pabelanger | " | 21:29 |
| *** ganso has quit IRC | 21:29 | |
| *** kgiusti has left #openstack-infra | 21:30 | |
| *** pcaruana has quit IRC | 21:30 | |
| dhellmann | ok, I'll let the jobs run the way I have it now without quotes and then try that variation | 21:31 |
| *** dave-mccowan has joined #openstack-infra | 21:31 | |
| *** caphrim007_ has quit IRC | 21:31 | |
| *** jappleii__ has joined #openstack-infra | 21:32 | |
| *** dgonzalez has quit IRC | 21:33 | |
| dmsimard | AJaeger: that's not within the work directory... "DEBUG zuul.AnsibleJob: [build: 406ded37f2f5469fbe0ca8af3011fd5d] Adding role path /var/lib/zuul/builds/406ded37f2f5469fbe0ca8af3011fd5d/ansible/pre_playbook_0/role_0/project-config/roles" | 21:33 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add consolidated role for processing subunit https://review.openstack.org/529339 | 21:33 |
| openstackgerrit | Merged openstack-infra/system-config master: Add a zuul01.openstack.org https://review.openstack.org/533781 | 21:34 |
| corvus | fungi: ^ | 21:35 |
| dmsimard | fungi, corvus: I guess it's too late by now but I would've probably named that node zs01 (zuul-scheduler01) and then probably put zw01 (zuul-web) on a separate machine | 21:35 |
| dmsimard | going by the ze and zm nomenclature for consistency | 21:35 |
| corvus | dmsimard: why put zuul-web on a separate machine? | 21:37 |
| *** kjackal has joined #openstack-infra | 21:38 | |
| dmsimard | don't have strong opinions, it occurred to me that naming the node "zs" while keeping the zuul-web service on it probably wouldn't be ideal | 21:38 |
| dmsimard | (I mean from a "server name represents what's on it" approach) | 21:39 |
| *** erlon has quit IRC | 21:39 | |
| corvus | dmsimard: zuul-scheduler won't be a scaleout component for a while yet. zuul-web will be as soon as we fully move the github event stuff into it. | 21:39 |
| corvus | but isn't at the moment, so right now it's going to be a lot easier to colocate them. once zuul-web is done, we can scale it out if necessary. though i don't think it will be | 21:40 |
| fungi | i'm ambivalent as to whether we call our scheduler zuulNN (though there will only presumably ever be one of them at any given point in time) or zsNN | 21:42 |
| *** dgonzalez has joined #openstack-infra | 21:42 | |
| fungi | if there is any one server which can be said to be the most zuulish of all servers, it's the scheduler | 21:42 |
| fungi | though perhaps at some point if zuul-web is serving the api and is horizontally scalable, we may want zuul.openstack.org to go to a load balancer instead of the scheduler? | 21:43 |
| corvus | ya | 21:43 |
| *** lbragstad has quit IRC | 21:44 | |
| dmsimard | ¯\_(ツ)_/¯ I thought I would bring it up so we think about it while we have the opportunity now rather than realize later that we should've picked another name | 21:45 |
| dmsimard | zuul01 is fine | 21:45 |
| *** lbragstad has joined #openstack-infra | 21:47 | |
| *** erlon has joined #openstack-infra | 21:48 | |
| *** jascott1 has joined #openstack-infra | 21:50 | |
| *** Goneri has quit IRC | 21:52 | |
| *** e0ne has quit IRC | 21:54 | |
| *** armaan has joined #openstack-infra | 21:55 | |
| fungi | zuul.o.o is going to be a cname regardless, so we're not tied to pointing it at zuulNN indefinitely even if we keep the same naming scheme | 21:56 |
| *** e0ne has joined #openstack-infra | 21:59 | |
| dmsimard | AJaeger, mordred: So it's as I thought... the implicitely prepared roles by Zuul aren't in the work root and that's what is killing the include_vars from configure-unbound | 22:00 |
| dmsimard | The failure happens here http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/paths.py?h=feature/zuulv3#n25 and it triggers because the home directory of the user is the work_root http://paste.openstack.org/raw/645521/ | 22:01 |
| dmsimard | I'll discuss it further in #zuul | 22:01 |
| *** e0ne has quit IRC | 22:01 | |
| fungi | oh, also zuul meeting time in #openstack-meeting-alt | 22:04 |
| *** tpsilva has quit IRC | 22:04 | |
| *** trown is now known as trown|outtypewww | 22:05 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Add test_launcher test https://review.openstack.org/533771 | 22:07 |
| *** lbragstad has quit IRC | 22:10 | |
| *** caphrim007 has quit IRC | 22:11 | |
| *** caphrim007 has joined #openstack-infra | 22:12 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation https://review.openstack.org/524862 | 22:12 |
| *** florianf has quit IRC | 22:14 | |
| *** kjackal has quit IRC | 22:15 | |
| fungi | poop. i think i missed adding the host pattern for the ansible group | 22:16 |
| fungi | Could not find data item gerrit_zu | 22:16 |
| fungi | ul_user_ssh_key_contents in any Hiera data file and no default supplied at /opt/ | 22:16 |
| fungi | system-config/production/manifests/site.pp:1450 on node zuul01.openstack.org | 22:16 |
| *** hashar has quit IRC | 22:18 | |
| *** lbragstad has joined #openstack-infra | 22:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add zuul-scheduler ansible group for zuul01 https://review.openstack.org/533807 | 22:19 |
| fungi | corvus: pabelanger: ^ for after the meeting | 22:19 |
| *** jascott1 has quit IRC | 22:19 | |
| *** rcernin has joined #openstack-infra | 22:21 | |
| *** slaweq has quit IRC | 22:24 | |
| *** smatzek has quit IRC | 22:24 | |
| *** slaweq has joined #openstack-infra | 22:24 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation https://review.openstack.org/524862 | 22:28 |
| *** slaweq has quit IRC | 22:29 | |
| *** armaan has quit IRC | 22:30 | |
| *** jamesmcarthur has joined #openstack-infra | 22:33 | |
| *** dbecker has quit IRC | 22:33 | |
| *** felipemonteiro__ has quit IRC | 22:33 | |
| *** slaweq has joined #openstack-infra | 22:39 | |
| *** jamesmcarthur has quit IRC | 22:44 | |
| *** slaweq has quit IRC | 22:44 | |
| *** jcoufal has quit IRC | 22:48 | |
| *** jamesmcarthur has joined #openstack-infra | 22:49 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Periodically retrieve and back up Zuul's status.json https://review.openstack.org/532955 | 22:50 |
| *** markvoelker has joined #openstack-infra | 22:53 | |
| *** jamesmcarthur has quit IRC | 22:54 | |
| dmsimard | ianw: fixed ^ thanks | 22:54 |
| *** jamesmcarthur has joined #openstack-infra | 22:54 | |
| *** eharney has quit IRC | 22:57 | |
| dmsimard | actually there's another thing .. | 22:57 |
| fungi | gonna disappear for to find some dinner, but will retry to launch zuul01.o.o if 533807 has merged by the time i get back | 22:57 |
| dmsimard | puppet throws an "Unrecognized escape sequence" error because of the \% | 22:57 |
| fungi | dmsimard: oh, yeah, we have that same issue in other places. you have to do something like \\\% i think | 22:58 |
| dmsimard | yuck | 22:58 |
| *** markvoelker has quit IRC | 22:58 | |
| dmsimard | 3 of them ? | 22:58 |
| dmsimard | oops, codesearch.o.o is broken "ERROR:open /home/hound/data/idx-6ab7a27cda29da25/raw/service/files/local_settings.j2: too many open files" | 22:59 |
| fungi | checking for a working example | 22:59 |
| dmsimard | fungi: I was trying to find a working example with codesearch :D | 22:59 |
| fungi | :? | 22:59 |
| fungi | :/ | 22:59 |
| *** lbragstad has quit IRC | 23:00 | |
| *** armaan has joined #openstack-infra | 23:01 | |
| fungi | not having much luck turning one up... also i think the behavior may differ depending on whether you use "" or '' strings | 23:01 |
| dmsimard | I'll try and figure something out | 23:01 |
| fungi | oh, it may just be \\% but yeah, my recollection is that cron wants \% because % has a special meaning in a crontab, and puppet wants the literal \ escaped, so that becomes \\ followed by % | 23:02 |
| ianw | dmsimard: hmm, yeah, maybe houndd is in a loop? | 23:02 |
| dmsimard | ianw: I think it's actually more like a bug | 23:02 |
| dmsimard | ianw: searching for \% breaks it :) | 23:02 |
| dmsimard | searching for other things work | 23:03 |
| *** jamesmcarthur has quit IRC | 23:03 | |
| ianw | the logs have 2018/01/15 23:02:06 vcs pull error (oslotest - git://git.openstack.org/openstack/oslotest): open /dev/null: too many open files | 23:03 |
| ianw | although, yeah, maybe you've found a DOS | 23:03 |
| dmsimard | I've ended up finding some occurrences of \\% so I'll try that. | 23:04 |
| fungi | we do have some examples in system-config with just '...\%...' in a cron resource command definition | 23:04 |
| fungi | a couple in puppetmaster.pp and one in elasticsearch_node.pp | 23:04 |
| fungi | looking in the root crontab on puppetmaster.o.o, the \% makes it into the file correctly | 23:05 |
| *** bobh has quit IRC | 23:05 | |
| ianw | dmsimard / fungi : not helped that i don't have all the info ... could you take a quick look at https://review.openstack.org/#/c/529188/ which fixes the hound sync log capturing | 23:05 |
| fungi | anyway, really disappearing now... bbiaw | 23:05 |
| *** esberglu has quit IRC | 23:10 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Periodically retrieve and back up Zuul's status.json https://review.openstack.org/532955 | 23:10 |
| dmsimard | ok re-tested to get rid of the escape sequence warning (which is totally not fatal fwiw, it actually creates the cron properly..) | 23:10 |
| dmsimard | puppet4 doesn't complain at all | 23:11 |
| dmsimard | puppet3 does | 23:11 |
| *** esberglu has joined #openstack-infra | 23:11 | |
| *** esberglu has quit IRC | 23:16 | |
| corvus | dhellmann: you may have trouble with role_path -- if so, you may want to see if just relative paths work... | 23:20 |
| *** esberglu has joined #openstack-infra | 23:27 | |
| *** esberglu has quit IRC | 23:28 | |
| *** hongbin has quit IRC | 23:28 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on merge speakers https://review.openstack.org/533820 | 23:29 |
| *** r-daneel has quit IRC | 23:33 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on merge speakers https://review.openstack.org/533820 | 23:35 |
| *** dhill_ has quit IRC | 23:38 | |
| *** olaph1 has joined #openstack-infra | 23:38 | |
| *** olaph has quit IRC | 23:39 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation https://review.openstack.org/524862 | 23:44 |
| dhellmann | corvus : thanks, I'll see if i can figure out what relative path would work | 23:44 |
| *** nicolasbock has joined #openstack-infra | 23:47 | |
| *** lbragstad has joined #openstack-infra | 23:53 | |
| *** dingyichen has joined #openstack-infra | 23:54 | |
| *** armax has quit IRC | 23:54 | |
| *** dingyichen has quit IRC | 23:55 | |
| *** dingyichen has joined #openstack-infra | 23:56 | |
| *** dingyichen has joined #openstack-infra | 23:57 | |
| *** erlon has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
