| jeblair | sdague: what's the latest on getting rid of the rechecks page? | 00:00 |
|---|---|---|
| sdague | jeblair: the latest is I've completely run out of time to work on this this cycle, as I've got nova things I need to sort before release, so no ETA | 00:00 |
| *** jdurgin has quit IRC | 00:01 | |
| *** sarob_ has joined #openstack-infra | 00:01 | |
| jeblair | bummer; i'd really like to replace that with the e-r page. | 00:01 |
| *** sarob_ has quit IRC | 00:01 | |
| *** thuc has quit IRC | 00:01 | |
| *** thuc has joined #openstack-infra | 00:02 | |
| jeblair | jog0: i don't understand why you have 1270309 on the e-r page. | 00:03 |
| jeblair | jog0: i have read the commit message where you added it, but i still don't understand. | 00:03 |
| *** sarob has quit IRC | 00:04 | |
| jeblair | sdague: do you know ^ ? | 00:04 |
| *** vuil has joined #openstack-infra | 00:05 | |
| *** jdurgin has joined #openstack-infra | 00:05 | |
| *** mtreinish has quit IRC | 00:05 | |
| *** thuc has quit IRC | 00:07 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Revert "Add query for bug 1270309" https://review.openstack.org/72504 | 00:07 |
| uvirtbot | Launchpad bug 1270309 in openstack-ci "jenkins java.lang.InterruptedException" [Undecided,Opinion] https://launchpad.net/bugs/1270309 | 00:07 |
| *** flaper87 is now known as flaper87|afk | 00:07 | |
| *** fbo_away is now known as fbo | 00:08 | |
| *** amcrn_ has joined #openstack-infra | 00:08 | |
| *** amcrn has quit IRC | 00:09 | |
| *** amcrn_ is now known as amcrn | 00:09 | |
| *** mtreinish has joined #openstack-infra | 00:10 | |
| *** vuil has quit IRC | 00:11 | |
| *** hughsaunders has quit IRC | 00:13 | |
| *** jhesketh__ has joined #openstack-infra | 00:13 | |
| *** mwagner_lap has quit IRC | 00:15 | |
| *** mwagner_lap has joined #openstack-infra | 00:15 | |
| *** mwagner_lap has quit IRC | 00:16 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Revert "Better query for bug 1260311" https://review.openstack.org/72507 | 00:16 |
| uvirtbot | Launchpad bug 1260311 in openstack-ci "hudson.Launcher exception causing build failures" [Undecided,Invalid] https://launchpad.net/bugs/1260311 | 00:16 |
| *** pdmars has quit IRC | 00:17 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Revert "Better query for bug 1260311" https://review.openstack.org/72507 | 00:17 |
| *** apevec has quit IRC | 00:19 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove tox locale overrides https://review.openstack.org/72215 | 00:20 |
| *** hughsaunders has joined #openstack-infra | 00:20 | |
| sdague | jeblair: because we need to account for all causes of failure in ES, otherwise the the uncategorized list can't be counted | 00:20 |
| jeblair | sdague: where's the uncategorized list? | 00:20 |
| sdague | so if those can get registered as something other than fail in ES, we can remvove it | 00:20 |
| sdague | http://status.openstack.org/elastic-recheck/data/uncategorized.html | 00:20 |
| sdague | and I have to run | 00:20 |
| jeblair | sdague: thanks. probably the easiest thing to do (weighing all the competing considerations (we want data from intermediate builds, the fact that we don't want the page built by a daemon, etc...)) is | 00:23 |
| jeblair | sdague: to just suppress those queries from the e-r bug page | 00:24 |
| *** hdd_ has joined #openstack-infra | 00:24 | |
| jeblair | sdague: so they count as classified for the unclassified page but don't look like, well, basically the number two problem with openstack is that we abort jobs sometimes. :) | 00:24 |
| *** dims has quit IRC | 00:26 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Run fewer es queries with elastic_recheck. https://review.openstack.org/72509 | 00:27 |
| clarkb | sdague: jog0 ^ that should hopefully make es happier | 00:27 |
| *** stevemar has joined #openstack-infra | 00:30 | |
| *** vuil has joined #openstack-infra | 00:31 | |
| *** nati_ueno has joined #openstack-infra | 00:32 | |
| *** AlexF has joined #openstack-infra | 00:35 | |
| *** cadenzajon has quit IRC | 00:36 | |
| *** david-lyle has quit IRC | 00:37 | |
| *** CaptTofu has joined #openstack-infra | 00:39 | |
| *** dims has joined #openstack-infra | 00:41 | |
| jeblair | clarkb: i think in the long term we should have logstash subscribe to zuul instead of jenkins | 00:41 |
| *** AlexF has quit IRC | 00:41 | |
| *** masayukig has joined #openstack-infra | 00:41 | |
| jeblair | clarkb: not only to avoid the e-r problem above, but, well, so that we can continue to use it after we stop using jenkins. :) | 00:42 |
| clarkb | ++ | 00:42 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Fix alignment of tabs https://review.openstack.org/72503 | 00:42 |
| clarkb | jeblair: fwiw that was my first stab at the whole thing and you -2'd the change :) | 00:42 |
| clarkb | because zuul isn't a proxy :) | 00:43 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Suppress some bugs from the e-r graph page https://review.openstack.org/72512 | 00:43 |
| jeblair | clarkb: never! | 00:43 |
| *** dcramer_ has joined #openstack-infra | 00:44 | |
| jeblair | jog0, sdague: ^ intermediate fix for those two bugs until logstash groks aborted builds | 00:44 |
| *** masayukig has quit IRC | 00:46 | |
| *** olaph has quit IRC | 00:49 | |
| *** smurugesan has joined #openstack-infra | 00:50 | |
| *** zul has joined #openstack-infra | 00:50 | |
| *** rcleere has quit IRC | 00:51 | |
| *** oubiwann_ has quit IRC | 00:51 | |
| *** matrohon has quit IRC | 00:55 | |
| *** matrohon has joined #openstack-infra | 00:55 | |
| *** nati_ueno has quit IRC | 00:56 | |
| jog0 | jeblair: looks like a good solution | 00:56 |
| *** thuc has joined #openstack-infra | 00:56 | |
| *** AlexF has joined #openstack-infra | 00:57 | |
| *** thuc has quit IRC | 00:57 | |
| *** jooools has quit IRC | 00:58 | |
| *** thuc has joined #openstack-infra | 00:58 | |
| *** zul has quit IRC | 00:59 | |
| *** gokrokve has quit IRC | 01:00 | |
| *** gokrokve has joined #openstack-infra | 01:00 | |
| *** gokrokve has quit IRC | 01:05 | |
| *** dkliban is now known as dkliban_afk | 01:06 | |
| *** atiwari has quit IRC | 01:07 | |
| *** vuil has quit IRC | 01:08 | |
| *** andreaf has quit IRC | 01:09 | |
| *** andreaf has joined #openstack-infra | 01:09 | |
| jeblair | jog0: cool, thx | 01:13 |
| jeblair | clarkb: how about this? we add a feature to zuul so that you can specify gearman jobs to be invoked whenever a build starts, is aborted, and completes | 01:14 |
| jeblair | clarkb: we add it to the "job:" config in zuul, so in our config, we'd probably just use it on the "^.*$" job so it always runs, but you'd have the option of being more fine-grained | 01:14 |
| stevemar | jeblair, looking at your patch here: https://review.openstack.org/#/c/72492/1/setup.cfg do you have an example of how to set overrides for the errors we're seeing now? | 01:14 |
| jeblair | clarkb: and we can have nodepool handle all three events, and logstash just the last | 01:15 |
| *** e0ne has quit IRC | 01:16 | |
| clarkb | ++ | 01:16 |
| jeblair | stevemar: no, i'm not aware of a way to override those. i think you'd want to either fix them first in the tree, then land that patch, or else if they are not hard to fix, update that patch to fix all the problems at the same time it enables the flag | 01:16 |
| *** CaptTofu has quit IRC | 01:16 | |
| *** gokrokve has joined #openstack-infra | 01:17 | |
| stevemar | jeblair, alright, we fixed some of the files that were failing, but others are coming from oslo (already submitted patches to get them fixed), so we'll wait | 01:18 |
| jeblair | stevemar: does oslo have a docs build that can catch those? if so, we should set the flag there too | 01:19 |
| *** CaptTofu has joined #openstack-infra | 01:19 | |
| dims | stevemar pointer to reviews please? | 01:19 |
| stevemar | jeblair, i would think not, otherwise they wouldn't be there | 01:20 |
| stevemar | dims, sec, let me find | 01:20 |
| *** reed has quit IRC | 01:22 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: wip: see state of load on concurency 4x https://review.openstack.org/72518 | 01:25 |
| jeblair | stevemar: sorry, i meant is there a docs build that, like the keystone docs build, is currently outputting warnings that instead we could be erroring on | 01:25 |
| stevemar | dims, it was already fixed: https://review.openstack.org/#/c/71107/ we just have to sync | 01:25 |
| *** gokrokve has quit IRC | 01:25 | |
| dims | cool | 01:27 |
| stevemar | jeblair, i'm not sure, but my thinking is, if they did have it, they would have seen the errors before we sync'ed in keystone? | 01:28 |
| *** gema has quit IRC | 01:28 | |
| *** msuriar has quit IRC | 01:28 | |
| *** msuriar has joined #openstack-infra | 01:29 | |
| *** gema has joined #openstack-infra | 01:29 | |
| *** isviridov has quit IRC | 01:29 | |
| *** sileht has quit IRC | 01:29 | |
| *** masayukig has joined #openstack-infra | 01:30 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add uncategorized page into the tab navigation https://review.openstack.org/72519 | 01:30 |
| *** masayukig has quit IRC | 01:34 | |
| *** ryanpetrello has quit IRC | 01:36 | |
| *** jamesmcarthur has joined #openstack-infra | 01:37 | |
| *** sileht has joined #openstack-infra | 01:37 | |
| *** isviridov has joined #openstack-infra | 01:38 | |
| openstackgerrit | A change was merged to openstack-dev/cookiecutter: Remove tox locale overrides https://review.openstack.org/72233 | 01:42 |
| *** isviridov has quit IRC | 01:45 | |
| *** dolphm has quit IRC | 01:46 | |
| *** oubiwann_ has joined #openstack-infra | 01:49 | |
| *** weshay has quit IRC | 01:51 | |
| *** dolphm has joined #openstack-infra | 01:51 | |
| *** AlexF has quit IRC | 01:51 | |
| *** melwitt has quit IRC | 01:52 | |
| *** thuc has quit IRC | 01:53 | |
| *** thuc has joined #openstack-infra | 01:54 | |
| *** isviridov has joined #openstack-infra | 01:54 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Set concurrency to num CPUs / 2 https://review.openstack.org/69256 | 01:54 |
| *** yaguang has joined #openstack-infra | 01:54 | |
| *** mrodden has quit IRC | 01:57 | |
| jeblair | stevemar: it's sort of a truism around here that people generally don't notice errors unless they fail gate tests :/ | 01:57 |
| *** thuc has quit IRC | 01:58 | |
| *** yongli has joined #openstack-infra | 01:59 | |
| *** jamesmcarthur has quit IRC | 02:01 | |
| *** thuc has joined #openstack-infra | 02:03 | |
| *** thedodd has joined #openstack-infra | 02:06 | |
| *** thuc has quit IRC | 02:07 | |
| *** thuc has joined #openstack-infra | 02:08 | |
| *** jamesmcarthur has joined #openstack-infra | 02:11 | |
| *** pcrews has quit IRC | 02:12 | |
| *** thuc has quit IRC | 02:12 | |
| *** dstanek has quit IRC | 02:14 | |
| *** gokrokve has joined #openstack-infra | 02:14 | |
| *** yamahata has quit IRC | 02:16 | |
| *** banix has joined #openstack-infra | 02:16 | |
| dims | jeblair, stevemar - something's wrong '"Finished: FAILURE" AND build_name:"gate-oslo-incubator-docs"' shows only one failure in last 7 days... | 02:17 |
| dims | stevemar, got a pointer to the keystone docs failure? | 02:18 |
| *** AlexF has joined #openstack-infra | 02:19 | |
| stevemar | dims, in keystone, the docs gate didn't fail, since warnings weren't seen as errors (if that makes sense) | 02:20 |
| stevemar | dims, these were the failures we are seeing in keystone, http://paste.openstack.org/show/64078/ | 02:21 |
| *** sarob has joined #openstack-infra | 02:22 | |
| *** freyes has quit IRC | 02:23 | |
| dims | stevemar, thanks, looking | 02:24 |
| stevemar | dims, how do you build docs for oslo? | 02:24 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Add check for removed modules in Python 3 https://review.openstack.org/61049 | 02:26 |
| *** e0ne has joined #openstack-infra | 02:27 | |
| mordred | I agree with jeblair | 02:28 |
| *** masayukig has joined #openstack-infra | 02:31 | |
| *** e0ne has quit IRC | 02:31 | |
| *** masayuki_ has joined #openstack-infra | 02:32 | |
| *** masayukig has quit IRC | 02:32 | |
| *** VijayT has joined #openstack-infra | 02:33 | |
| *** sarob has quit IRC | 02:34 | |
| *** sarob has joined #openstack-infra | 02:34 | |
| *** thedodd has quit IRC | 02:35 | |
| *** masayuki_ has quit IRC | 02:37 | |
| *** VijayT has quit IRC | 02:37 | |
| *** sarob has quit IRC | 02:39 | |
| stevemar | dims, no worries, i sync'ed db/sqlalchemy/session.py, and now everything is good | 02:40 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: wip: attempt to collect pidstat summary https://review.openstack.org/72529 | 02:40 |
| stevemar | going to submit a keystone patch now, then 1 error left... and we can gate against doc errors again | 02:40 |
| *** sarob has joined #openstack-infra | 02:41 | |
| *** johnthetubaguy has joined #openstack-infra | 02:42 | |
| *** dcramer_ has quit IRC | 02:47 | |
| *** chandan_kumar has joined #openstack-infra | 02:48 | |
| *** jamesmcarthur has quit IRC | 02:48 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: wip: see state of load on concurency 3x https://review.openstack.org/72518 | 02:50 |
| *** morganfainberg is now known as morganfainberg_Z | 02:51 | |
| *** chandan_kumar has quit IRC | 02:54 | |
| *** esker has joined #openstack-infra | 02:55 | |
| *** chandan_kumar has joined #openstack-infra | 02:56 | |
| *** AaronGr has quit IRC | 03:00 | |
| *** mikal has quit IRC | 03:02 | |
| *** pabelanger has quit IRC | 03:02 | |
| *** mikal has joined #openstack-infra | 03:03 | |
| *** bodepd has quit IRC | 03:03 | |
| *** greghaynes has quit IRC | 03:04 | |
| *** chandan_kumar has quit IRC | 03:04 | |
| *** AaronGr has joined #openstack-infra | 03:05 | |
| *** bodepd has joined #openstack-infra | 03:05 | |
| *** pabelanger has joined #openstack-infra | 03:05 | |
| *** CaptTofu has quit IRC | 03:06 | |
| *** greghaynes has joined #openstack-infra | 03:07 | |
| *** VijayT has joined #openstack-infra | 03:09 | |
| *** bodepd has quit IRC | 03:09 | |
| *** bodepd has joined #openstack-infra | 03:11 | |
| *** AaronGr_Zzz has joined #openstack-infra | 03:13 | |
| *** sarob has quit IRC | 03:16 | |
| *** sarob has joined #openstack-infra | 03:16 | |
| *** yamahata has joined #openstack-infra | 03:16 | |
| *** AaronGr_Zzz has quit IRC | 03:19 | |
| *** AaronGr_Zzz has joined #openstack-infra | 03:20 | |
| *** sarob has quit IRC | 03:20 | |
| *** yamahata has quit IRC | 03:21 | |
| *** rcleere has joined #openstack-infra | 03:21 | |
| *** jnoller has joined #openstack-infra | 03:22 | |
| *** dims has quit IRC | 03:23 | |
| *** dims has joined #openstack-infra | 03:24 | |
| *** AaronGr_Zzz has quit IRC | 03:25 | |
| *** malini has left #openstack-infra | 03:26 | |
| *** AaronGr_Zzz has joined #openstack-infra | 03:26 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client https://review.openstack.org/68828 | 03:27 |
| *** chandan_kumar has joined #openstack-infra | 03:29 | |
| *** AaronGr_Zzz has quit IRC | 03:30 | |
| *** AaronGr_Zzz has joined #openstack-infra | 03:30 | |
| *** Ryan_Lane has joined #openstack-infra | 03:31 | |
| *** masayukig has joined #openstack-infra | 03:33 | |
| *** jnoller has quit IRC | 03:34 | |
| *** masayuki_ has joined #openstack-infra | 03:35 | |
| *** dims has quit IRC | 03:35 | |
| *** masayukig has quit IRC | 03:35 | |
| *** dims has joined #openstack-infra | 03:36 | |
| *** AlexF has quit IRC | 03:37 | |
| *** sarob has joined #openstack-infra | 03:43 | |
| *** dims has quit IRC | 03:45 | |
| *** dims has joined #openstack-infra | 03:46 | |
| jhesketh__ | jeblair: ping | 03:48 |
| *** sarob_ has joined #openstack-infra | 03:49 | |
| *** sarob has quit IRC | 03:50 | |
| *** dims has quit IRC | 03:55 | |
| *** dims has joined #openstack-infra | 03:56 | |
| *** gokrokve has quit IRC | 04:05 | |
| *** gokrokve has joined #openstack-infra | 04:05 | |
| *** masayuki_ has quit IRC | 04:07 | |
| *** gokrokve has quit IRC | 04:10 | |
| *** bhuvan has joined #openstack-infra | 04:10 | |
| *** dcramer_ has joined #openstack-infra | 04:12 | |
| *** masayukig has joined #openstack-infra | 04:14 | |
| *** yamahata has joined #openstack-infra | 04:19 | |
| *** masayukig has quit IRC | 04:20 | |
| *** harlowja is now known as harlowja_away | 04:22 | |
| *** yamahata has quit IRC | 04:23 | |
| *** bhuvan has quit IRC | 04:24 | |
| *** gokrokve has joined #openstack-infra | 04:25 | |
| *** dcramer_ has quit IRC | 04:30 | |
| *** harlowja_away is now known as harlowja | 04:30 | |
| portante | clarkb: you still around? | 04:32 |
| *** mat-lowe_ has joined #openstack-infra | 04:32 | |
| *** belliott has quit IRC | 04:33 | |
| *** igor_ has joined #openstack-infra | 04:35 | |
| *** mrmartin has joined #openstack-infra | 04:37 | |
| *** belliott has joined #openstack-infra | 04:38 | |
| *** dims has quit IRC | 04:41 | |
| *** dcramer_ has joined #openstack-infra | 04:42 | |
| *** johnthetubaguy has quit IRC | 04:42 | |
| *** dims has joined #openstack-infra | 04:43 | |
| *** yamahata has joined #openstack-infra | 04:45 | |
| *** johnthetubaguy has joined #openstack-infra | 04:49 | |
| *** talluri has joined #openstack-infra | 04:51 | |
| *** talluri has quit IRC | 04:55 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tempest-cells-full tests to check queue https://review.openstack.org/72556 | 04:56 |
| *** AaronGr has quit IRC | 04:56 | |
| *** AaronGr_Zzz is now known as AaronGr | 04:57 | |
| *** bhuvan has joined #openstack-infra | 04:59 | |
| *** eharney has quit IRC | 05:02 | |
| *** sarob_ has quit IRC | 05:06 | |
| *** AlexF has joined #openstack-infra | 05:06 | |
| *** CaptTofu has joined #openstack-infra | 05:07 | |
| *** sarob has joined #openstack-infra | 05:07 | |
| *** crank has quit IRC | 05:07 | |
| *** sarob_ has joined #openstack-infra | 05:08 | |
| *** thuc has joined #openstack-infra | 05:08 | |
| *** sarob_ has quit IRC | 05:09 | |
| *** mriedem has joined #openstack-infra | 05:09 | |
| *** sarob_ has joined #openstack-infra | 05:09 | |
| *** thuc_ has joined #openstack-infra | 05:10 | |
| *** CaptTofu has quit IRC | 05:11 | |
| *** sarob has quit IRC | 05:11 | |
| *** thuc has quit IRC | 05:13 | |
| *** sarob_ has quit IRC | 05:13 | |
| *** crank has joined #openstack-infra | 05:14 | |
| *** jhesketh__ has quit IRC | 05:15 | |
| *** chandan_kumar has quit IRC | 05:15 | |
| *** chandan_kumar has joined #openstack-infra | 05:16 | |
| *** banix has quit IRC | 05:18 | |
| *** nicedice_ has quit IRC | 05:20 | |
| *** masayukig has joined #openstack-infra | 05:25 | |
| *** VijayT has quit IRC | 05:29 | |
| *** masayukig has quit IRC | 05:29 | |
| *** sarob has joined #openstack-infra | 05:34 | |
| *** thuc_ has quit IRC | 05:35 | |
| *** thuc has joined #openstack-infra | 05:35 | |
| *** dims has quit IRC | 05:35 | |
| *** amcrn has quit IRC | 05:36 | |
| *** dims has joined #openstack-infra | 05:37 | |
| *** thuc_ has joined #openstack-infra | 05:37 | |
| *** talluri has joined #openstack-infra | 05:37 | |
| *** thuc has quit IRC | 05:40 | |
| *** mriedem has quit IRC | 05:41 | |
| *** dims has quit IRC | 05:43 | |
| *** thuc_ has quit IRC | 05:44 | |
| *** dims has joined #openstack-infra | 05:44 | |
| *** thuc has joined #openstack-infra | 05:45 | |
| *** thuc has quit IRC | 05:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: Give test mysql user create and drop perms. https://review.openstack.org/69519 | 05:49 |
| *** gokrokve has quit IRC | 05:54 | |
| *** johnthetubaguy has quit IRC | 05:54 | |
| openstackgerrit | A change was merged to openstack-infra/config: tempest-dsvm-neutron-heat-slow graduates from experimental https://review.openstack.org/70037 | 05:56 |
| *** gokrokve has joined #openstack-infra | 05:57 | |
| *** mrmartin has quit IRC | 05:58 | |
| *** AlexF has quit IRC | 06:04 | |
| *** sarob_ has joined #openstack-infra | 06:05 | |
| *** rwsu has quit IRC | 06:07 | |
| *** rwsu has joined #openstack-infra | 06:07 | |
| *** sarob has quit IRC | 06:08 | |
| *** bhuvan has quit IRC | 06:09 | |
| *** sarob_ has quit IRC | 06:12 | |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Fix incompatibility with gerrit 2.8 https://review.openstack.org/69791 | 06:12 |
| *** amcrn has joined #openstack-infra | 06:15 | |
| *** amcrn_ has joined #openstack-infra | 06:16 | |
| *** amcrn has quit IRC | 06:20 | |
| nibalizer | does anyone know how the hiera function is populated into the openstack-config/infra puppet codebase? | 06:21 |
| nibalizer | usually in a puppet 2.7 setup like this there would be a hiera module that has a lib dir with the hiera function in it | 06:21 |
| clarkb | nibalizer: it comes from puppet-hiera iirc | 06:22 |
| clarkb | not sure why you would need magic for that | 06:22 |
| nibalizer | the gem | 06:23 |
| nibalizer | ? | 06:23 |
| clarkb | the deb package | 06:23 |
| *** bhuvan has joined #openstack-infra | 06:23 | |
| *** oubiwann_ has quit IRC | 06:24 | |
| nibalizer | oh okay | 06:24 |
| *** rwsu has quit IRC | 06:25 | |
| nibalizer | it just throws hiera.rb into the puppet autoloading path i guesss? | 06:25 |
| clarkb | I guess? | 06:26 |
| clarkb | haus should know | 06:26 |
| *** amcrn_ is now known as amcrn | 06:28 | |
| *** DinaBelova_ is now known as DinaBelova | 06:31 | |
| *** cadenzajon has joined #openstack-infra | 06:33 | |
| *** harlowja is now known as harlowja_away | 06:33 | |
| *** denis_makogon has joined #openstack-infra | 06:36 | |
| *** cadenzajon has quit IRC | 06:37 | |
| *** rwsu has joined #openstack-infra | 06:38 | |
| *** saju_m has joined #openstack-infra | 06:38 | |
| *** coolsvap has joined #openstack-infra | 06:40 | |
| *** saju_m has quit IRC | 06:43 | |
| *** saju_m has joined #openstack-infra | 06:50 | |
| *** saju_m has quit IRC | 06:51 | |
| *** hdd_ has quit IRC | 06:51 | |
| *** mat-lowe_ has quit IRC | 06:54 | |
| *** yolanda has quit IRC | 06:57 | |
| *** sarob has joined #openstack-infra | 07:05 | |
| *** jhesketh has quit IRC | 07:06 | |
| *** CaptTofu has joined #openstack-infra | 07:07 | |
| *** yolanda has joined #openstack-infra | 07:09 | |
| *** lttrl has quit IRC | 07:10 | |
| *** lttrl has joined #openstack-infra | 07:11 | |
| *** CaptTofu has quit IRC | 07:12 | |
| *** yamahata has quit IRC | 07:15 | |
| *** gyee has quit IRC | 07:18 | |
| *** jhesketh has joined #openstack-infra | 07:19 | |
| *** yamahata has joined #openstack-infra | 07:20 | |
| *** yongli has quit IRC | 07:22 | |
| *** reed has joined #openstack-infra | 07:22 | |
| *** dims has quit IRC | 07:24 | |
| *** dims has joined #openstack-infra | 07:25 | |
| *** DinaBelova is now known as DinaBelova_ | 07:30 | |
| *** dims has quit IRC | 07:32 | |
| *** dims has joined #openstack-infra | 07:32 | |
| *** sarob has quit IRC | 07:33 | |
| *** e0ne has joined #openstack-infra | 07:34 | |
| *** e0ne has quit IRC | 07:35 | |
| *** saju_m has joined #openstack-infra | 07:37 | |
| wenlock | clarkb, nibalizer im not sure it's really bootstrapped fully in install_module.sh or install_puppet.sh.... i know we had to add automation to fully get it setup on our end | 07:39 |
| wenlock | gem install hiera | 07:40 |
| wenlock | puppet resource package hiera-puppet ensure=installed | 07:40 |
| wenlock | you also still need to setup /etc/puppet/hieradata | 07:41 |
| *** dims has quit IRC | 07:42 | |
| *** dims has joined #openstack-infra | 07:43 | |
| *** jhesketh has quit IRC | 07:45 | |
| *** salv-orlando has quit IRC | 07:46 | |
| *** zigo has joined #openstack-infra | 07:47 | |
| *** rlandy has joined #openstack-infra | 07:49 | |
| *** saju_m has quit IRC | 07:52 | |
| *** denis_makogon has quit IRC | 08:04 | |
| *** flaper87|afk is now known as flaper87 | 08:04 | |
| *** sarob has joined #openstack-infra | 08:05 | |
| nibalizer | wenlock: okay thanks for the confirmation | 08:08 |
| nibalizer | install_puppet.sh (as i understand it) is more about bootstrapping nodes with puppet apply, and doesn't need hiera | 08:08 |
| nibalizer | im just trying to get my own dev environment up so I can start testing my changes before I propose them in gerrit | 08:08 |
| wenlock | nibalizer, yep... you only need hiera function on the puppet master | 08:08 |
| *** sarob has quit IRC | 08:10 | |
| *** bhuvan has quit IRC | 08:22 | |
| *** bhuvan has joined #openstack-infra | 08:24 | |
| *** d0ugal has joined #openstack-infra | 08:24 | |
| d0ugal | Hey, how can I move this patch along? jenkins finished with a "LOST" result - despite each gate passing individually at one point https://review.openstack.org/#/c/72295/ | 08:25 |
| *** thomasbiege has joined #openstack-infra | 08:26 | |
| *** e0ne has joined #openstack-infra | 08:27 | |
| *** bhuvan has quit IRC | 08:30 | |
| *** bhuvan has joined #openstack-infra | 08:31 | |
| *** DinaBelova_ is now known as DinaBelova | 08:32 | |
| *** hashar has joined #openstack-infra | 08:33 | |
| d0ugal | 'recheck no bug' seems to have started something. | 08:34 |
| *** jhesketh has joined #openstack-infra | 08:35 | |
| *** AJaeger has left #openstack-infra | 08:35 | |
| ttx | jeblair: re: https://review.openstack.org/#/c/65103/ - since I don't plan to use that for icehouse, maybe the smartest move is to WIP it until juno opens and make final call at design summit | 08:35 |
| ttx | jeblair: if you confirm I'll WIP it. | 08:35 |
| *** hashar has quit IRC | 08:38 | |
| *** hashar has joined #openstack-infra | 08:40 | |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin https://review.openstack.org/71658 | 08:41 |
| *** hashar_ has joined #openstack-infra | 08:41 | |
| SergeyLukjanov | jeblair, I think that it's enough to prevent merges and not give up the change | 08:42 |
| SergeyLukjanov | ttx, jeblair, ^^ | 08:42 |
| *** hashar has quit IRC | 08:45 | |
| *** wenlock has quit IRC | 08:47 | |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin https://review.openstack.org/71658 | 08:48 |
| *** shardy_afk is now known as shardy | 08:48 | |
| *** e0ne_ has joined #openstack-infra | 08:49 | |
| *** pblaho has joined #openstack-infra | 08:50 | |
| *** saju_m has joined #openstack-infra | 08:51 | |
| *** e0ne has quit IRC | 08:53 | |
| *** psedlak has quit IRC | 08:56 | |
| *** gokrokve has quit IRC | 08:57 | |
| *** psedlak has joined #openstack-infra | 08:58 | |
| *** markmc has joined #openstack-infra | 09:00 | |
| *** jpich has joined #openstack-infra | 09:01 | |
| *** bhuvan has quit IRC | 09:03 | |
| *** yassine has joined #openstack-infra | 09:03 | |
| *** bhuvan has joined #openstack-infra | 09:04 | |
| *** sarob has joined #openstack-infra | 09:05 | |
| *** yamahata has quit IRC | 09:06 | |
| *** hashar_ is now known as hashar | 09:06 | |
| *** yamahata has joined #openstack-infra | 09:07 | |
| *** CaptTofu has joined #openstack-infra | 09:08 | |
| *** bhuvan has quit IRC | 09:09 | |
| *** sarob has quit IRC | 09:10 | |
| *** bhuvan has joined #openstack-infra | 09:11 | |
| *** jcoufal has joined #openstack-infra | 09:12 | |
| *** DinaBelova is now known as DinaBelova_ | 09:13 | |
| *** CaptTofu has quit IRC | 09:13 | |
| openstackgerrit | Nadya Privalova proposed a change to openstack/requirements: Fix happybase version https://review.openstack.org/68435 | 09:16 |
| *** mrmartin has joined #openstack-infra | 09:18 | |
| *** afazekas has joined #openstack-infra | 09:21 | |
| *** fbo is now known as fbo_away | 09:22 | |
| *** DinaBelova_ is now known as DinaBelova | 09:24 | |
| *** rossella_s has joined #openstack-infra | 09:24 | |
| *** afazekas has quit IRC | 09:25 | |
| openstackgerrit | sahid proposed a change to openstack-infra/elastic-recheck: Adds query for devstack bug #1260723 https://review.openstack.org/72605 | 09:31 |
| uvirtbot | Launchpad bug 1260723 in devstack "Invalid OpenStack Nova credentials." [Undecided,New] https://launchpad.net/bugs/1260723 | 09:31 |
| *** fbo_away is now known as fbo | 09:31 | |
| openstackgerrit | Alexei Kornienko proposed a change to openstack/requirements: Added argcomplete requirement https://review.openstack.org/65863 | 09:32 |
| *** gokrokve has joined #openstack-infra | 09:33 | |
| *** stevemar has quit IRC | 09:33 | |
| *** jp_at_hp has joined #openstack-infra | 09:35 | |
| *** jp_at_hp has quit IRC | 09:35 | |
| *** jp_at_hp has joined #openstack-infra | 09:35 | |
| *** gokrokve_ has joined #openstack-infra | 09:35 | |
| *** coolsvap has quit IRC | 09:36 | |
| *** boris-42_ has quit IRC | 09:36 | |
| *** gokrokve has quit IRC | 09:38 | |
| *** dizquierdo has joined #openstack-infra | 09:38 | |
| *** gokrokve_ has quit IRC | 09:39 | |
| *** jooools has joined #openstack-infra | 09:41 | |
| *** bhuvan has quit IRC | 09:52 | |
| *** bhuvan_ has joined #openstack-infra | 09:52 | |
| *** andre__ has joined #openstack-infra | 09:55 | |
| *** reed has quit IRC | 09:58 | |
| *** DinaBelova is now known as DinaBelova_ | 10:03 | |
| *** sarob has joined #openstack-infra | 10:05 | |
| *** sarob has quit IRC | 10:10 | |
| *** DinaBelova_ is now known as DinaBelova | 10:19 | |
| *** saju_m has quit IRC | 10:25 | |
| *** saju_m has joined #openstack-infra | 10:26 | |
| *** ok_delta has joined #openstack-infra | 10:26 | |
| *** masayukig has joined #openstack-infra | 10:30 | |
| *** ociuhandu has quit IRC | 10:33 | |
| *** gokrokve has joined #openstack-infra | 10:34 | |
| *** tru_tru has quit IRC | 10:36 | |
| *** gokrokve has quit IRC | 10:39 | |
| *** bhuvan_ has quit IRC | 10:39 | |
| *** saper has quit IRC | 10:41 | |
| *** bhuvan has joined #openstack-infra | 10:44 | |
| *** hashar is now known as hasharAway | 10:45 | |
| *** max_lobur_afk is now known as max_lobur | 10:46 | |
| *** thomasbiege has quit IRC | 10:46 | |
| *** rfolco has joined #openstack-infra | 10:48 | |
| *** vbala has joined #openstack-infra | 10:53 | |
| *** yaguang has quit IRC | 10:54 | |
| vbala | Please refer https://review.openstack.org/#/c/70761/ I would like to know what are the next steps once this is approved. | 10:55 |
| *** yassine has quit IRC | 10:56 | |
| *** hasharAway has quit IRC | 10:57 | |
| *** CaptTofu has joined #openstack-infra | 10:57 | |
| *** sneelakantan has joined #openstack-infra | 11:00 | |
| *** ArxCruz has joined #openstack-infra | 11:00 | |
| *** sneelakantan is now known as Guest10292 | 11:00 | |
| *** Guest10292 has left #openstack-infra | 11:00 | |
| *** Guest10292 has joined #openstack-infra | 11:00 | |
| SergeyLukjanov | vbala, hey, after approval, new projects/jobs will created and you can start using them | 11:01 |
| *** masayukig has quit IRC | 11:01 | |
| *** masayukig has joined #openstack-infra | 11:02 | |
| *** CaptTofu has quit IRC | 11:02 | |
| *** tru_tru has joined #openstack-infra | 11:03 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Create PuppetDB server https://review.openstack.org/72618 | 11:04 |
| *** sarob has joined #openstack-infra | 11:05 | |
| *** masayukig has quit IRC | 11:06 | |
| *** reed has joined #openstack-infra | 11:06 | |
| nibalizer | anteaya: AaronGr fungi ^ | 11:08 |
| *** bhuvan has quit IRC | 11:08 | |
| *** bhuvan has joined #openstack-infra | 11:08 | |
| *** sarob has quit IRC | 11:10 | |
| *** reed has quit IRC | 11:11 | |
| *** masayukig has joined #openstack-infra | 11:12 | |
| *** ArxCruz has quit IRC | 11:14 | |
| *** ociuhandu has joined #openstack-infra | 11:15 | |
| *** boris-42_ has joined #openstack-infra | 11:16 | |
| *** lcestari has joined #openstack-infra | 11:18 | |
| *** e0ne_ has quit IRC | 11:19 | |
| *** tristanC has quit IRC | 11:21 | |
| *** tristanC has joined #openstack-infra | 11:21 | |
| *** tristanC has quit IRC | 11:22 | |
| *** tristanC has joined #openstack-infra | 11:22 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add uncategorized page into the tab navigation https://review.openstack.org/72519 | 11:23 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 11:23 |
| *** alexpilotti has joined #openstack-infra | 11:26 | |
| *** ok_delta has quit IRC | 11:31 | |
| *** gokrokve has joined #openstack-infra | 11:34 | |
| *** DinaBelova has quit IRC | 11:39 | |
| *** gokrokve has quit IRC | 11:39 | |
| *** DinaBelova has joined #openstack-infra | 11:39 | |
| *** vladan has quit IRC | 11:40 | |
| *** vladan has joined #openstack-infra | 11:42 | |
| *** vladan has quit IRC | 11:47 | |
| *** ArxCruz has joined #openstack-infra | 11:48 | |
| *** vladan has joined #openstack-infra | 11:49 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 11:50 |
| *** NikitaKonovalov_ has quit IRC | 11:50 | |
| *** e0ne has joined #openstack-infra | 11:50 | |
| *** SergeyLukjanov has quit IRC | 11:51 | |
| *** oanufriev has quit IRC | 11:52 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add storyboard-webclient to ATC lists generator https://review.openstack.org/72626 | 11:53 |
| *** NikitaKonovalov_ has joined #openstack-infra | 11:54 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 11:54 | |
| *** SergeyLukjanov has joined #openstack-infra | 11:55 | |
| *** e0ne_ has joined #openstack-infra | 11:59 | |
| *** yamahata has quit IRC | 12:02 | |
| *** masayukig has quit IRC | 12:02 | |
| *** d0ugal has quit IRC | 12:03 | |
| *** e0ne has quit IRC | 12:03 | |
| *** sarob has joined #openstack-infra | 12:05 | |
| *** sarob has quit IRC | 12:10 | |
| *** yassine has joined #openstack-infra | 12:11 | |
| *** Guest10292 is now known as sneelakantan | 12:13 | |
| *** bhuvan has quit IRC | 12:19 | |
| *** masayukig has joined #openstack-infra | 12:19 | |
| *** reed has joined #openstack-infra | 12:19 | |
| openstackgerrit | Eric Guo proposed a change to openstack/requirements: Sort global-requirements https://review.openstack.org/64943 | 12:21 |
| *** reed has quit IRC | 12:24 | |
| *** bhuvan has joined #openstack-infra | 12:25 | |
| *** lcostantino has joined #openstack-infra | 12:25 | |
| dhellmann | SergeyLukjanov, jeblair, fungi, clarkb : I would appreciate your feedback on this WIP document, when you have a moment: https://wiki.openstack.org/wiki/Oslo/CreatingANewLibrary | 12:27 |
| SergeyLukjanov | dhellmann, hi, sure, I'll take a look on it | 12:27 |
| dhellmann | SergeyLukjanov: thanks! | 12:28 |
| SergeyLukjanov | dhellmann, does oslo projects use separated launchpad projects? | 12:28 |
| dhellmann | SergeyLukjanov: good question, and we're still on the fence about that | 12:29 |
| *** dizquierdo has quit IRC | 12:29 | |
| dhellmann | some projects do (the new adoptees and oslo.messaging) but others don't (incubator, oslo.config) | 12:29 |
| dhellmann | at this point, I think having them separate makes my PTL work more difficult, but when we move to storyboard we will probably split them up | 12:29 |
| dhellmann | I should mention launchpad in there somewhere, though | 12:30 |
| SergeyLukjanov | dhellmann, "group" could be specified in modules/openstack_project/files/review.projects.yaml | 12:31 |
| SergeyLukjanov | it'll be used as launchpad project link now and as a project group in storyboard in future | 12:32 |
| dhellmann | what form does that take? | 12:32 |
| * dhellmann looks for example | 12:32 | |
| SergeyLukjanov | dhellmann, https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/review.projects.yaml#L286 | 12:33 |
| SergeyLukjanov | it means that savanna-dashboard managed on launchpad.net/savanna | 12:33 |
| SergeyLukjanov | project name by default | 12:34 |
| dhellmann | SergeyLukjanov: ok, so to use https://launchpad.net/oslo I would set the group to "oslo" | 12:34 |
| SergeyLukjanov | dhellmann, it's oslo by default, so, no need to set it explicitly | 12:34 |
| dhellmann | what is using that? | 12:34 |
| *** gokrokve has joined #openstack-infra | 12:34 | |
| dhellmann | so if the library is "oslo.db" the default is "oslo"? | 12:34 |
| SergeyLukjanov | dhellmann, nope, default is the project name, it'll "oslo.db" for "oslo.db" :) | 12:35 |
| dhellmann | ok, I'll add the explicit value we want | 12:35 |
| *** ttx has quit IRC | 12:35 | |
| dhellmann | SergeyLukjanov: is this correct? "Set the "group" to "oslo" to use https://launchpad.net/oslo for tracking bugs and milestones. For libraries with existing launchpad projects, set the "group" to the name of the launchpad project." | 12:37 |
| *** reed has joined #openstack-infra | 12:38 | |
| SergeyLukjanov | dhellmann, ... if it differs from the project name | 12:38 |
| *** gokrokve has quit IRC | 12:39 | |
| dhellmann | SergeyLukjanov: ok, if those match then it's redundant but that's ok | 12:39 |
| SergeyLukjanov | dhellmann, sure | 12:39 |
| dhellmann | I'd rather have clear, easy to follow, instructions, than optimize out a little typing :-) | 12:39 |
| *** vbala has quit IRC | 12:39 | |
| SergeyLukjanov | dhellmann, hm, I think you'd like to have docs and docs jobs in all oslo projects? | 12:39 |
| dhellmann | yes! | 12:40 |
| SergeyLukjanov | dhellmann, I'm now editing your jobs layout | 12:40 |
| dhellmann | cool, thanks | 12:40 |
| SergeyLukjanov | dhellmann, https://wiki.openstack.org/wiki/Oslo/CreatingANewLibrary#Configure_Zuul_to_Run_Jobs | 12:42 |
| *** smarcet has joined #openstack-infra | 12:42 | |
| *** bhuvan has quit IRC | 12:43 | |
| * dhellmann looks at diff | 12:43 | |
| *** reed_ has joined #openstack-infra | 12:44 | |
| dhellmann | SergeyLukjanov: what is the difference between "project-name-branch-tarball" and "project-name-tarball"? | 12:44 |
| *** zul has joined #openstack-infra | 12:45 | |
| SergeyLukjanov | dhellmann, the first one creates project-name-master.tar.gz and the second one will create tarballs for each ref id like release tags with this ref in end | 12:45 |
| SergeyLukjanov | like project-name-0.1.tar.gz | 12:46 |
| SergeyLukjanov | so, we need first one in post pipeline to have actual creates project-name-master.tar.gz | 12:46 |
| *** reed has quit IRC | 12:47 | |
| SergeyLukjanov | dhellmann, additionally, I've added gate-project-name-requirements | 12:47 |
| dhellmann | that enforces the global requirements? | 12:48 |
| *** yamahata has joined #openstack-infra | 12:48 | |
| SergeyLukjanov | dhellmann, yup | 12:49 |
| SergeyLukjanov | dhellmann, IMO all projects under openstack org should have this job :) | 12:50 |
| dhellmann | SergeyLukjanov: +1 | 12:50 |
| SergeyLukjanov | dhellmann, everything else looks ok | 12:50 |
| dhellmann | good, thanks for helping out with that | 12:51 |
| SergeyLukjanov | dhellmann, np, sure | 12:51 |
| dhellmann | fwiw, the examples I took straight from the stackforge instructions on ci.openstack.org, so any edits you made to this may want to go back there | 12:51 |
| openstackgerrit | A change was merged to openstack/requirements: Taskflow 0.1.3 was released https://review.openstack.org/71699 | 12:52 |
| dhellmann | I promised jeblair that once this doc is complete, it could be used as the basis for a new "create a new openstack project" page on that document, but I need to add the bits about using cookiecutter to create a new project layout first | 12:52 |
| *** vladan has quit IRC | 12:52 | |
| *** banix has joined #openstack-infra | 12:53 | |
| *** yamahata has quit IRC | 12:54 | |
| *** banix has quit IRC | 12:54 | |
| *** vladan has joined #openstack-infra | 12:54 | |
| *** jhesketh has quit IRC | 12:54 | |
| *** reed__ has joined #openstack-infra | 12:56 | |
| sdague | dhellmann: where does the oslo messaging barfing errors over everything currently stand? | 12:57 |
| SergeyLukjanov | dhellmann, great, I'll take a look on project creation instructions, but templates usage is probably too complex for new stackforge projects, most of then don't want to have requirements and docs jobs for example | 12:57 |
| sdague | that's basically preventing us from enabling error enforcement on logs | 12:57 |
| dhellmann | sdague: https://review.openstack.org/#/c/71035/ | 12:57 |
| dhellmann | waiting for review | 12:57 |
| dhellmann | I'll nudge the oslo-core team | 12:57 |
| sdague | please do, this is kind of critical to get resolved | 12:58 |
| sdague | if we ship like this, our logs will be 100% garbage to deployers | 12:58 |
| *** ttx_ has joined #openstack-infra | 12:58 | |
| *** reed_ has quit IRC | 13:00 | |
| dhellmann | sdague: email sent | 13:00 |
| *** reed__ has quit IRC | 13:01 | |
| *** sneelakantan has quit IRC | 13:02 | |
| *** markmcclain has joined #openstack-infra | 13:02 | |
| dhellmann | sdague: I could use your attention on https://review.openstack.org/#/c/72487/ (I think you have +2 there?) | 13:03 |
| *** DeeJay1 has joined #openstack-infra | 13:03 | |
| sdague | sure | 13:04 |
| *** heyongli has joined #openstack-infra | 13:04 | |
| sdague | dhellmann: so https://review.openstack.org/#/c/71035/ doesn't fix the whole problem | 13:05 |
| sdague | http://logs.openstack.org/35/71035/3/check/check-tempest-dsvm-full/8e1daa2/console.html.gz#_2014-02-05_18_28_31_859 | 13:05 |
| *** sarob has joined #openstack-infra | 13:05 | |
| *** dcramer_ has quit IRC | 13:05 | |
| dhellmann | sdague: it should make it so that only exceptions which are not declared as expected are being logged, and then changes in the app can fix those | 13:07 |
| dhellmann | sdague: I'll take another look today to see if some of those remaining messages should be "expected" | 13:08 |
| *** weshay has joined #openstack-infra | 13:09 | |
| flaper87 | dhellmann: sdague FWIW, that patch looks good to me | 13:10 |
| *** pdmars has joined #openstack-infra | 13:10 | |
| *** sarob has quit IRC | 13:10 | |
| flaper87 | dhellmann: feel free to ping me when you double check it, I'm ready to +2 it | 13:10 |
| sdague | dhellmann: so I actually question oslo.messaging using ERROR level here at all | 13:10 |
| dhellmann | sdague: from what I can tell, the intent there is to indicate that something unexpected has gone wrong | 13:11 |
| dhellmann | is that not an error? | 13:11 |
| sdague | ERROR in openstack log means page the operator, they have to fix the cloud now | 13:11 |
| dhellmann | is this a warning? info? | 13:11 |
| sdague | warning might be more fair | 13:11 |
| flaper87 | dhellmann: maybe warning then | 13:11 |
| dhellmann | ok, I can make that as a follow-up change | 13:12 |
| sdague | though, realistically, I think we'll have to revisit that regardless | 13:12 |
| sdague | in juno | 13:12 |
| dhellmann | I suspect that's just going to hide all of these undeclared exceptions and they're going to still be logged | 13:12 |
| dhellmann | "hide" from the gate job, that is | 13:13 |
| sdague | dhellmann: well, I think they are mostly getting exposed on the nova side | 13:13 |
| sdague | every single one of those that's left seems to have a nova error being logged first | 13:13 |
| sdague | not all shown, because we've whitelisted some of them | 13:14 |
| dhellmann | sdague: for example, snapshot_instance() does not declare that it expects to raise any errors at all, which is why InstanceNotFound is being logged | 13:14 |
| sdague | so here is the dillema... no one is working on cleaning up these issues in nova | 13:15 |
| openstackgerrit | Flavio Percoco proposed a change to openstack/requirements: Use OS's pypi mirror for Marconi https://review.openstack.org/72646 | 13:15 |
| sdague | so the merge of oslo.messaging just increased the stack traces in nova logs by 20 - 50% | 13:16 |
| dhellmann | sdague: we can turn this feature off, I guess, but it's meant to help with exactly the project you're working on | 13:17 |
| sdague | dhellmann: well, right now, it's preventing us locking things down again | 13:17 |
| dhellmann | maybe it's less helpful than expected? I'd have to ask markmc about its origins. | 13:17 |
| sdague | because we can't enforce no errors in logs at the moment | 13:17 |
| sdague | because we have things that exceed our whitelist | 13:18 |
| markmc | dhellmann, what's unclear to me is whether this was really introduced by switching to oslo.messaging | 13:18 |
| markmc | dhellmann, because, if so, something's messed up - this should have the same behaviour as @client_exceptions from oslo-incubator | 13:19 |
| sdague | markmc: maybe it just moved all the whitelisting around? | 13:19 |
| markmc | sdague, don't know what you mean | 13:20 |
| *** yamahata has joined #openstack-infra | 13:20 | |
| markmc | sdague, behaviour should be the same | 13:20 |
| sdague | markmc: but the messages would have changed | 13:20 |
| markmc | sdague, the whitelisting is based on exception class name | 13:20 |
| sdague | currently, in a successful tempest run, we've got hundreds of ERROR messages | 13:20 |
| markmc | not name, type | 13:20 |
| dhellmann | the only thing that has changed is the names of the decorator | 13:20 |
| markmc | yes, I see them | 13:20 |
| *** hashar has joined #openstack-infra | 13:21 | |
| markmc | e.g. | 13:21 |
| sdague | markmc: the attempt at clean log enforcement includes a whitelist | 13:21 |
| markmc | @messaging.expected_exceptions(exception.InstanceNotFound) | 13:21 |
| sdague | which is regex in the log | 13:21 |
| dhellmann | the name of the logger changed too, was something filtering on that? | 13:21 |
| markmc | sdague, ah, we're talking about different whitelisting | 13:21 |
| sdague | what leaks through to console.html | 13:21 |
| markmc | sdague, where is that whitelist? | 13:21 |
| sdague | is a small subset | 13:21 |
| sdague | in tempest | 13:21 |
| markmc | where? | 13:22 |
| sdague | https://github.com/openstack/tempest/blob/master/etc/whitelist.yaml | 13:22 |
| markmc | thanks | 13:22 |
| markmc | well, e.g. | 13:23 |
| markmc | - module: "nova.openstack.common.rpc.amqp" | 13:23 |
| markmc | message: "Exception during message handling" | 13:23 |
| markmc | that's obviously changed | 13:23 |
| sdague | yeh | 13:23 |
| *** dstanek has joined #openstack-infra | 13:24 | |
| *** oubiwann_ has joined #openstack-infra | 13:25 | |
| *** oubiwann_ has quit IRC | 13:25 | |
| *** rfolco has quit IRC | 13:25 | |
| *** oubiwann_ has joined #openstack-infra | 13:25 | |
| sdague | that would only handle the ERROR oslo.messaging._executors.base messages though | 13:26 |
| * dhellmann is meeting with ttx but will read the backlog | 13:27 | |
| *** oanufriev has joined #openstack-infra | 13:29 | |
| *** gokrokve has joined #openstack-infra | 13:34 | |
| *** hdd_ has joined #openstack-infra | 13:36 | |
| *** gokrokve_ has joined #openstack-infra | 13:36 | |
| dhellmann | sdague: no, it looks like the logging was just in the amqp module before | 13:38 |
| *** thomasem has joined #openstack-infra | 13:38 | |
| dhellmann | sdague: the logic that is now in the base executor used to be in the amqp ProxyCallback | 13:38 |
| *** gokrokve has quit IRC | 13:39 | |
| dhellmann | well, and in the base driver | 13:39 |
| dhellmann | sdague: so to achieve the same effect as before, one a new filter for is needed | 13:40 |
| dhellmann | but that's the stuff we discussed changing to warning | 13:40 |
| *** gokrokve_ has quit IRC | 13:41 | |
| dhellmann | sdague: I can still change that log level, but that just cleans up the logs it doesn't fix the fact that we have undeclared exceptions | 13:41 |
| sdague | dhellmann: I agree | 13:41 |
| sdague | but we're in this bind where there are too many issues to address, and more getting added every day | 13:41 |
| dhellmann | shall I make it warning, and we can revisit this during juno? | 13:41 |
| *** dcramer_ has joined #openstack-infra | 13:42 | |
| sdague | so if we can set a line somewhere that we can enforce on, we can at least not let it get worse | 13:42 |
| dhellmann | I'm with you, I'm just trying to figure out how to put it off in a way that means it is addressed at some point, rather than never | 13:42 |
| sdague | dhellmann: yeh, I think that would be good | 13:42 |
| dhellmann | ok | 13:42 |
| sdague | man, if you figure that one out, let me know :) | 13:42 |
| dhellmann | I'll do that as a follow-up patch to the one I've already submitted that is close to merging | 13:42 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Clean up puppet (deploy LAMP / setup app config) https://review.openstack.org/69636 | 13:44 |
| *** CaptTofu has joined #openstack-infra | 13:44 | |
| *** eharney has joined #openstack-infra | 13:46 | |
| *** saper has joined #openstack-infra | 13:48 | |
| dhellmann | sdague, markmc : https://review.openstack.org/72658 | 13:48 |
| *** smarcet has left #openstack-infra | 13:51 | |
| *** zul has quit IRC | 13:53 | |
| *** mriedem has joined #openstack-infra | 13:57 | |
| *** julim has joined #openstack-infra | 13:57 | |
| *** heyongli has quit IRC | 14:00 | |
| *** exz has quit IRC | 14:01 | |
| *** exz has joined #openstack-infra | 14:01 | |
| *** dprince has joined #openstack-infra | 14:02 | |
| *** dims has quit IRC | 14:05 | |
| *** sarob has joined #openstack-infra | 14:05 | |
| *** dkliban_afk is now known as dkliban | 14:06 | |
| *** rfolco has joined #openstack-infra | 14:06 | |
| *** thuc has joined #openstack-infra | 14:07 | |
| *** dims has joined #openstack-infra | 14:10 | |
| *** changbl has quit IRC | 14:10 | |
| *** sarob has quit IRC | 14:10 | |
| *** stevemar has joined #openstack-infra | 14:11 | |
| *** yaguang has joined #openstack-infra | 14:17 | |
| *** stevemar2 has joined #openstack-infra | 14:17 | |
| *** stevemar has quit IRC | 14:18 | |
| *** markmcclain has quit IRC | 14:18 | |
| *** pblaho has quit IRC | 14:19 | |
| *** dstanek has quit IRC | 14:20 | |
| *** talluri has quit IRC | 14:21 | |
| *** talluri has joined #openstack-infra | 14:21 | |
| sdague | fungi: you up yet? | 14:23 |
| sdague | SergeyLukjanov: you have +2 on d-g now as well, right? | 14:25 |
| SergeyLukjanov | sdague, yup | 14:25 |
| *** talluri has quit IRC | 14:25 | |
| sdague | https://review.openstack.org/#/c/71010/ - finally kills off using exercises for grenade | 14:26 |
| SergeyLukjanov | sdague, ok | 14:26 |
| SergeyLukjanov | sdague, approving | 14:27 |
| *** mattymo has joined #openstack-infra | 14:27 | |
| *** homeless has joined #openstack-infra | 14:27 | |
| *** saju_m has quit IRC | 14:27 | |
| mattymo | Where is the best place to sign up for notifications about recently fixed security issues in OpenStack? | 14:27 |
| *** esker has quit IRC | 14:28 | |
| *** saju_m has joined #openstack-infra | 14:29 | |
| *** thuc has quit IRC | 14:29 | |
| *** esker has joined #openstack-infra | 14:29 | |
| *** dkranz has quit IRC | 14:29 | |
| *** thuc has joined #openstack-infra | 14:29 | |
| markmc | mattymo, http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-announce is probably best | 14:30 |
| mattymo | no other routes currently? | 14:31 |
| *** igor___ has joined #openstack-infra | 14:33 | |
| *** thuc has quit IRC | 14:34 | |
| *** esker has quit IRC | 14:34 | |
| *** igor_ has quit IRC | 14:34 | |
| *** banix has joined #openstack-infra | 14:34 | |
| *** gokrokve has joined #openstack-infra | 14:34 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Extracted DB API layer https://review.openstack.org/72356 | 14:36 |
| *** dkranz has joined #openstack-infra | 14:36 | |
| *** stevemar2 has quit IRC | 14:38 | |
| *** gokrokve has quit IRC | 14:39 | |
| *** mfer has joined #openstack-infra | 14:39 | |
| *** thuc has joined #openstack-infra | 14:40 | |
| *** mrmartin has quit IRC | 14:41 | |
| *** dstanek has joined #openstack-infra | 14:43 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Extracted DB API layer https://review.openstack.org/72356 | 14:45 |
| *** thomasbiege has joined #openstack-infra | 14:45 | |
| *** dkranz has quit IRC | 14:47 | |
| *** thuc has quit IRC | 14:48 | |
| *** thuc has joined #openstack-infra | 14:49 | |
| *** prad_ has joined #openstack-infra | 14:53 | |
| *** wenlock has joined #openstack-infra | 14:53 | |
| *** thuc has quit IRC | 14:53 | |
| *** wchrisj|away is now known as chris_johnson | 14:53 | |
| *** ilyashakhat has quit IRC | 14:54 | |
| *** mriedem has quit IRC | 14:56 | |
| fungi | sdague: yeah, just being quiet and trying to get work done ;) | 14:56 |
| *** dizquierdo has joined #openstack-infra | 14:57 | |
| *** mriedem has joined #openstack-infra | 14:57 | |
| *** thomasbiege1 has joined #openstack-infra | 14:57 | |
| *** thomasbiege2 has joined #openstack-infra | 14:59 | |
| cody-somerville | What version of logstach are we using? | 14:59 |
| fungi | is that like mustache? | 14:59 |
| * fungi checks | 14:59 | |
| sdague | fungi: https://review.openstack.org/#/c/72556/ | 15:00 |
| sdague | turns out ... we aren't running cells tempest jobs anywhere, even in experimental | 15:00 |
| cody-somerville | *logstash | 15:00 |
| *** thomasbiege has quit IRC | 15:00 | |
| *** CaptTofu has quit IRC | 15:00 | |
| *** ProfFalken has joined #openstack-infra | 15:01 | |
| sdague | as grenade will stop using exercises in 1 hr, that means the only test, anywhere in OpenSTack that uses exercises is the cells job | 15:01 |
| sdague | and I really want to turn off anyone using exercises, as it's a terrible gating strategy | 15:01 |
| *** thomasbiege1 has quit IRC | 15:01 | |
| *** mriedem has quit IRC | 15:02 | |
| cody-somerville | ProfFalken: I think we use logstash 1.2.1. | 15:02 |
| fungi | cody-somerville: logstash-1.2.1-flatjar.jar | 15:02 |
| cody-somerville | ProfFalken: https://github.com/openstack-infra/config/commit/1f07114346d4bb9bb1a016c8eede38480ab1a60e | 15:02 |
| fungi | mattymo: we (the vmt members) publish openstack security advisories to the oss-security, openstack and openstack-announce mailing lists currently | 15:03 |
| mattymo | markmc, what about membership on the security team to learn about bugs as they come up and contribute? | 15:03 |
| mattymo | or fungi ^ | 15:03 |
| ProfFalken | cody-somerville: thanks | 15:04 |
| *** hdd_ has quit IRC | 15:04 | |
| fungi | mattymo: the openstack security group is generally only aware of publicly announced vulnerabilities, not embargoed vulnerabilities which are in the process of being fixed | 15:04 |
| *** dcramer_ has quit IRC | 15:04 | |
| fungi | mattymo: the vulnerability management team does have a list of downstream distributions and service providers who request advance notice (usually roughly two business days warning) of upcoming announcements for embargoed fixes | 15:05 |
| *** sarob has joined #openstack-infra | 15:05 | |
| *** mgagne has joined #openstack-infra | 15:05 | |
| *** wenlock has quit IRC | 15:05 | |
| fungi | mattymo: the vmt's processes are described at https://wiki.openstack.org/wiki/Vulnerability_Management | 15:05 |
| mattymo | fungi, I represent one of these downstream distros :) | 15:06 |
| fungi | mattymo: i know ;) that's why i mentioned it | 15:06 |
| fungi | mattymo: you're with mirantis, right? | 15:06 |
| mattymo | aye | 15:06 |
| mattymo | or I should say da | 15:07 |
| * fungi checks to see if we already notify someone there | 15:07 | |
| *** thomasbiege2 has quit IRC | 15:07 | |
| mattymo | the doc says I should write ttx_ or you, fungi, to request notification | 15:08 |
| fungi | mattymo: yep, you just wrote me. i'm finding out from ttx_ if we need anything more official-record-like | 15:08 |
| ttx_ | fungi: not really | 15:09 |
| mattymo | ttx_, great presentation at FOSDEM on our testing infra | 15:09 |
| fungi | ttx_: okay. do we need to follow up with anyone else at mirantis to confirm mattymo is who they want receiving these, or if they're rather it go to another address there first? | 15:10 |
| ttx_ | fungi: we need a short description of the distribution / public deployment | 15:10 |
| mattymo | fungi, I'm setting up an alias we can manage to direct that to the necessary person/people | 15:10 |
| *** sarob has quit IRC | 15:10 | |
| ttx_ | fungi: some confirmation from mirantis would not hurt yes | 15:10 |
| fungi | ttx_: should we be maintaining a more detailed/public list of notified downstream entities? (or do we already have one i've forgotten about?) | 15:11 |
| mattymo | fungi, you can talk to anyone at Mirantis you want, but you have me here and now. You can talk to SergeyLukjanov if you like :) | 15:11 |
| ttx_ | fungi: we have a private list (on the google doc) | 15:11 |
| ttx_ | mattymo: can you prove you are who you claim to be though ? :) | 15:11 |
| fungi | ttx_: right. didn't know if that was out of convention or necessity | 15:11 |
| *** zul has joined #openstack-infra | 15:12 | |
| mattymo | ttx_, I'm registered to nickserv and I am a manager of #fuel and #fuel-dev channels | 15:12 |
| mattymo | but you can write me an email to mmosesohn@mirantis.com and I can reply | 15:12 |
| ttx_ | mattymo: sounds good enough. I suspect the address will be some@mirantis.com | 15:12 |
| ttx_ | ack | 15:12 |
| fungi | ttx_: by public list, i mean something more like what the linux-distros list maintains... http://oss-security.openwall.org/wiki/mailing-lists/distros | 15:14 |
| mattymo | the alias will take a couple hours to get pushed through by our IT, but it will be osci-security@mirantis.com | 15:14 |
| ttx_ | fungi: I guess we /could/ | 15:14 |
| fungi | though that would probably encourage lots of parties to start requesting it | 15:14 |
| *** BobBallAway is now known as BobBall | 15:17 | |
| BobBall | Could I ask a devstack core to approve https://review.openstack.org/#/c/70571/ pls - it's currently blocking XenAPI tempest tests | 15:17 |
| *** ruhe_ has joined #openstack-infra | 15:17 | |
| mattymo | ttx_, fungi is getting added to this list an enormous burden? What further steps are needed to discuss? | 15:18 |
| *** thomasbiege has joined #openstack-infra | 15:18 | |
| fungi | BobBall: the devstack developers are generally more active in #openstack-qa, for future reference | 15:18 |
| BobBall | ah, thanks fungi | 15:18 |
| *** esker has joined #openstack-infra | 15:18 | |
| fungi | mattymo: ttx_: i suppose i can e-mail the osci-security@mirantis.com address once mattymo confirms it's live, and that ought to be good enough? | 15:18 |
| fungi | as long as he can reply from it | 15:18 |
| *** esker has quit IRC | 15:18 | |
| *** jgrimm has joined #openstack-infra | 15:19 | |
| *** pafuent has joined #openstack-infra | 15:19 | |
| mattymo | fungi, it won't be immediate. Let's plan to do this tomorrow because I can't expect our IT guys to stick around late in the office | 15:19 |
| *** esker has joined #openstack-infra | 15:19 | |
| mattymo | it's already 7:19pm here | 15:19 |
| fungi | mattymo: yep, no problem | 15:19 |
| ttx_ | fungi: ack | 15:20 |
| *** DeeJay1 has quit IRC | 15:20 | |
| *** mrmartin has joined #openstack-infra | 15:22 | |
| anteaya | nibalizer: a great beginning, thank you | 15:22 |
| anteaya | nibalizer: my first comment is that at 25000 lines of code, there is no way I am going to be able to understand this colossus | 15:24 |
| anteaya | nibalizer: what can be done to create bit sized patches (we aim for 200 lines of code max per patch) that are dependent on each other | 15:25 |
| anteaya | nibalizer: but we also try for a dependency tree no more than 3 deep, so we seem to be looking at smaller seperate changes | 15:26 |
| *** AJaeger has joined #openstack-infra | 15:26 | |
| anteaya | nibalizer: I will be away for most of the day running errands so do respond in channel and I will read the logs/backscroll | 15:27 |
| anteaya | fungi: I won't be here for meeting so for the manage-projects item, the only update is what you and I have already discussed | 15:27 |
| anteaya | jeblair mattoliverau I won't be here for the meeting time change item either, so sorry about that | 15:28 |
| fungi | sdague: approved 72556. if memory serves we ripped out a bunch of the cells jobs to cut down on the number of nodes per change we were using , since they were non-voting and broken for a very long time. is the expectation that they'll be working soonish? | 15:28 |
| fungi | anteaya: thanks for the heads up | 15:28 |
| anteaya | mattoliverau said he would make the meeting so he can speak for himself | 15:28 |
| anteaya | and whatever is decided is what I am fine with, just wanted to spark a conversation | 15:29 |
| fungi | anteaya: just for confirmation, you're still in eastern standard timezone (utc-0500 or so)? | 15:29 |
| sdague | fungi: honestly, I have no idea, however we currently have no way of even knowing their state | 15:29 |
| anteaya | and since SergeyLukjanov is core now, we have to keep his timing in mind, which I hadn't considered before, so hopefully he will be there to speak up too | 15:29 |
| fungi | sdague: shrödinger's tests | 15:30 |
| *** wenlock has joined #openstack-infra | 15:30 | |
| anteaya | fungi: I am in utc -0500 Eastern standare timezone, correct | 15:30 |
| anteaya | fungi: I have to do some updown errands, banking stuff and such, daytime only appointments | 15:30 |
| fungi | er, schrödinger's | 15:31 |
| * fungi always forgets the c | 15:31 | |
| SergeyLukjanov | anteaya, I'm here, UTC+0400 | 15:31 |
| *** ttx has joined #openstack-infra | 15:31 | |
| *** ttx has quit IRC | 15:31 | |
| *** ttx has joined #openstack-infra | 15:31 | |
| anteaya | SergeyLukjanov: awesome | 15:31 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tempest-cells-full tests to check queue https://review.openstack.org/72556 | 15:31 |
| sdague | fungi: hit a rebase issue | 15:31 |
| *** ttx_ has quit IRC | 15:31 | |
| sdague | resubmitted | 15:31 |
| fungi | sdague: yep, saw that--thanks. i'll cram it back through once i take another quick look | 15:32 |
| *** david-lyle has joined #openstack-infra | 15:32 | |
| anteaya | SergeyLukjanov: not knowing you were going to be core (yay) I had put an item on today's agenda proposing a change in the meeting time, with the goal of helping to include mattoliverau a bit | 15:32 |
| anteaya | SergeyLukjanov: but that would make it harder for you to attend | 15:32 |
| anteaya | so that item will come up, do share your thoughts when it does | 15:32 |
| SergeyLukjanov | anteaya, oh, which time are you proposing? | 15:32 |
| anteaya | I had proposed 2200 utc | 15:33 |
| anteaya | which makes it 2am for you, correct? | 15:33 |
| SergeyLukjanov | yup | 15:33 |
| SergeyLukjanov | after the project meeting | 15:33 |
| anteaya | ouch | 15:33 |
| anteaya | a bad proposal then | 15:33 |
| *** jcoufal_ has joined #openstack-infra | 15:33 | |
| sdague | anteaya: sounds like you guys might want to go to alternating times | 15:33 |
| *** gokrokve has joined #openstack-infra | 15:34 | |
| anteaya | SergeyLukjanov: can you do this for me, when the rest of us are asleep, can you look out for mattoliverau and give him a hand? | 15:34 |
| fungi | anteaya: for the foundation staff weekly meeting where we have attendees in the americas, europe and apac, we alternate 1600 and 2300 utc and people in timezones where a meeting is in the middle of the night for them sometimes just submit updates to the agenda in advance instead of being on | 15:34 |
| fungi | no idea whether or not that would work out for something like the infra team meeting though | 15:34 |
| SergeyLukjanov | anteaya, ok, I'll contact him | 15:35 |
| anteaya | he is on infra and is just getting started, so could do with some infra-y type conversation | 15:35 |
| anteaya | SergeyLukjanov: thanks, that was my main goal, just to include him in discussions more | 15:35 |
| anteaya | cool | 15:35 |
| anteaya | good suggestions | 15:35 |
| *** boris-42_ has quit IRC | 15:36 | |
| *** jcoufal has quit IRC | 15:36 | |
| anteaya | looks like we have some good ideas for that item when it arises | 15:36 |
| anteaya | I'm fine with whatever is decided | 15:36 |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/reviewday: Make reviewday.json world readable. https://review.openstack.org/72682 | 15:36 |
| SergeyLukjanov | anteaya, looks like mattoliverau is connected to sendak.freenode.net (Vilnius, Lithuania, EU) | 15:36 |
| *** pafuent has left #openstack-infra | 15:36 | |
| anteaya | my motivation was to be inclusive, so however that manifests is fine with me | 15:36 |
| SergeyLukjanov | anteaya, so, it shouldn't be too late for him too | 15:36 |
| anteaya | SergeyLukjanov: he is in australia | 15:37 |
| anteaya | Melbourne? | 15:37 |
| SergeyLukjanov | anteaya, heh, nice ;) | 15:37 |
| anteaya | :D | 15:37 |
| *** reed__ has joined #openstack-infra | 15:38 | |
| SergeyLukjanov | UTC+10/11 | 15:38 |
| SergeyLukjanov | AFAIK | 15:38 |
| anteaya | that is about it yeah | 15:38 |
| fungi | SergeyLukjanov: yeah, the freenode server round-robin rarely gets you a server geographically close by | 15:38 |
| fungi | you can't really necessarily go by that | 15:38 |
| dstanek | one of the check jobs for my patchset failed in a really odd way. it looks like it timed out starting devstack? http://logs.openstack.org/32/71932/3/check/gate-tempest-dsvm-neutron-large-ops/282ce17/console.html | 15:38 |
| SergeyLukjanov | fungi, surprising ;) | 15:38 |
| *** gokrokve has quit IRC | 15:39 | |
| fungi | SergeyLukjanov: though i'm apparently connected to a server in texas, which is only a couple timezones away from me | 15:39 |
| SergeyLukjanov | SergeyLukjanov is connected on barjavel.freenode.net (Paris, FR) | 15:39 |
| SergeyLukjanov | :) | 15:39 |
| dstanek | can someone spotcheck and let me know what you think? i want to make a bug, but i want to make sure i get it right | 15:39 |
| *** yaguang has quit IRC | 15:40 | |
| *** coolsvap has joined #openstack-infra | 15:42 | |
| fungi | dstanek: it's hard to tell exactly where it spent most of its time since we don't have timestamps in the setup-workspace log, but it definitely died during or after trying to rsync the first project git repo from the template root filesystem to the new /opt filesystem on the ephemeral block device | 15:42 |
| *** saju_m has quit IRC | 15:45 | |
| fungi | oh, nevermind, i was thrown off by the for loop getting repeated in teh xtrace output. it's not at all the first one in the list. it got through a bunch and then got killed in the middle of trying to rsync swift's git repo | 15:45 |
| *** saju_m has joined #openstack-infra | 15:45 | |
| *** sarob has joined #openstack-infra | 15:45 | |
| fungi | but hard to know whether that rsync hung for an hour, or some other step before it took way too long and it ended up an innocent bystander, or if everything on that slave was just running really, really slowly | 15:46 |
| jeblair | dstanek: the load on zuul was very high at that point; it probably was having trouble fetching git refs | 15:47 |
| *** zul has quit IRC | 15:48 | |
| fungi | ohhh... this was yesterday morning | 15:48 |
| dstanek | fungi, jeblair: should i just 'recheck no bug' then? | 15:48 |
| jeblair | dstanek: yeah. we're working on fixing that in zuul | 15:48 |
| jeblair | sdague: can we discuss https://review.openstack.org/#/c/72512/ ? | 15:48 |
| dstanek | jeblair: great thanks | 15:48 |
| jeblair | sdague: i'd really not have two infra non-bugs displayed as the top offenders on the logstash page | 15:49 |
| jeblair | sdague: i understand it seems like pepering it over, but i don't think it is; you're using the bug queries for two different things, and it's only appropriate to use that bug for one of them. | 15:49 |
| jeblair | s/that bug/those bugs/ | 15:50 |
| *** sarob has quit IRC | 15:50 | |
| fungi | worth noting that it also means 50% of the top 4 bugs are marked with either an opinion or invalid status, which probably causes some confusion as well | 15:51 |
| *** wenlock has quit IRC | 15:51 | |
| *** DinaBelova is now known as DinaBelova_ | 15:51 | |
| jeblair | sdague: changing the logstash worker is not going to be fixed in a short amount of time; these are going to sit there for a while. | 15:53 |
| sdague | jeblair: it has deeper impacts, just not displaying them means our categorization rates and any other aggregate stats are wrong | 15:54 |
| jeblair | sdague: well, that's what i tried to address by not displaying them; it shouldn't affect the uncategorized page | 15:54 |
| jeblair | sdague: what did i miss? | 15:54 |
| sdague | it affects the rate | 15:54 |
| fungi | unless they're special-cased in the categorization rate and aggregate stat calculations | 15:54 |
| jeblair | sdague: where's the rate displayed? | 15:55 |
| sdague | at the top of the page | 15:55 |
| sdague | and it impacts our ability to do other statistics | 15:55 |
| *** gokrokve has joined #openstack-infra | 15:55 | |
| jeblair | sdague: top of which page? | 15:55 |
| sdague | http://status.openstack.org/elastic-recheck/data/uncategorized.html | 15:55 |
| jeblair | sdague: it shouldn't affect that -- it's still included on that page | 15:55 |
| sdague | it does affect it | 15:55 |
| sdague | because you have just pulled a bunch of hits out of display | 15:56 |
| *** gokrokve_ has joined #openstack-infra | 15:56 | |
| sdague | because they aren't really bugs | 15:56 |
| *** reed__ is now known as reed | 15:56 | |
| sdague | but you didn't pull them out of the total pool | 15:56 |
| *** reed has quit IRC | 15:56 | |
| *** reed has joined #openstack-infra | 15:56 | |
| sdague | so they make everything looks much better than it is | 15:56 |
| sdague | so if you want to address this, it should be much deeper | 15:56 |
| sdague | classifying certain queries as being ignored, then working through the whole pipeline so they are ignored everywhere | 15:57 |
| sdague | not just on the glass | 15:57 |
| jeblair | sdague: how is that possible? the only thing i changed was in the main() function of the graph program | 15:57 |
| sdague | you are saying, interupt isn't an actual bug | 15:58 |
| *** AlexF has joined #openstack-infra | 15:58 | |
| sdague | which I actually agree with | 15:58 |
| jeblair | i'm very glad about that | 15:58 |
| sdague | however, by only removing it from the graphs page, you are saying that our classification rate for real bugs includes 730 not actual real bugs | 15:58 |
| jeblair | sdague: oh, this is not a critique of my patch | 15:59 |
| jeblair | you built a page with incorrect data. please don't blame my patch for that | 15:59 |
| sdague | this is a critique that it's the wrong place to go instrument this | 15:59 |
| jeblair | i'm trying to do what i can to make the e-r bug page which is heavely used by developers more useful and correct for them | 15:59 |
| *** gokrokve has quit IRC | 15:59 | |
| jeblair | i'm sorry i don't have a solution to the other problem now, but yeah, that's a much deeper problem | 16:00 |
| jeblair | however, my patch doesn't make that page any more or less accurate than it is now | 16:00 |
| jeblair | so i think we should merge the graph fix because it's at least a marginal improvement to a very important page | 16:00 |
| *** sarob has joined #openstack-infra | 16:02 | |
| sdague | I'm honestly kind of frustrated, because I feel like when this was going the other way, to make a gate status page, or change the zuul page, you argued against it for the same consistency reasons. | 16:02 |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/reviewday: Make reviewday.json world readable. https://review.openstack.org/72682 | 16:03 |
| jeblair | sdague: regarding the gate status page, i was _agreeing_ with you. your original plan was to extend the zuul status page -- you sent an email about that. | 16:03 |
| *** mriedem has joined #openstack-infra | 16:03 | |
| jeblair | sdague: i feel _very_ strongly that we should not put incorrect information in front of users. I'm doing my best to try to make this situation better | 16:04 |
| jeblair | sdague: i think we should make the e-r bug page as accurate as possible, and that means removing fake bugs from it | 16:04 |
| *** sarob_ has joined #openstack-infra | 16:05 | |
| jeblair | sdague: if the uncategorized page is incorrect, perhaps we should annotate those parts | 16:05 |
| jeblair | sdague: it's still useful for finding unclassified failures, but hey, for the moment, the overall rate is incorrect due to an accounting error; let's note that and try to fix it when we can | 16:06 |
| *** ociuhandu has quit IRC | 16:06 | |
| *** atiwari has joined #openstack-infra | 16:06 | |
| *** sarob has quit IRC | 16:07 | |
| *** nicedice has joined #openstack-infra | 16:09 | |
| *** lcheng_ has joined #openstack-infra | 16:09 | |
| *** sandywalsh has joined #openstack-infra | 16:09 | |
| sdague | sure | 16:09 |
| *** jnoller has joined #openstack-infra | 16:10 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Suppress some bugs from the e-r graph page https://review.openstack.org/72512 | 16:10 |
| *** zul has joined #openstack-infra | 16:10 | |
| *** jnoller_ has joined #openstack-infra | 16:11 | |
| openstackgerrit | Eric Windisch proposed a change to openstack/requirements: Add Docker requirement https://review.openstack.org/72691 | 16:14 |
| *** jnoller has quit IRC | 16:14 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Extracted DB API layer https://review.openstack.org/72356 | 16:14 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Refactor REST API to work on top of DB API https://review.openstack.org/72357 | 16:15 |
| *** markmcclain has joined #openstack-infra | 16:15 | |
| *** jcoufal_ is now known as jcoufal | 16:15 | |
| *** CaptTofu has joined #openstack-infra | 16:16 | |
| *** markmcclain1 has joined #openstack-infra | 16:16 | |
| *** mrda_away is now known as mrda | 16:18 | |
| *** markmcclain has quit IRC | 16:19 | |
| *** gokrokve_ has quit IRC | 16:20 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Refactor REST API to work on top of DB API https://review.openstack.org/72357 | 16:21 |
| *** gokrokve has joined #openstack-infra | 16:23 | |
| *** thomasbiege has quit IRC | 16:28 | |
| *** thuc has joined #openstack-infra | 16:29 | |
| *** mrmartin has quit IRC | 16:29 | |
| *** bnemec has quit IRC | 16:31 | |
| *** dizquierdo has quit IRC | 16:31 | |
| *** dkranz has joined #openstack-infra | 16:32 | |
| *** gyee has joined #openstack-infra | 16:33 | |
| mriedem | fungi: are you guys running with pip >= 1.5 now because of this? https://review.openstack.org/72225 | 16:33 |
| *** bnemec has joined #openstack-infra | 16:34 | |
| *** markwash has joined #openstack-infra | 16:35 | |
| fungi | mriedem: no, those patches work around pip < 1.5 problems in py3k | 16:36 |
| fungi | specifically older pip has trouble dealing with metadata files in some packages under py3k if it needs to handle characters which can't be represented in the locale of the calling environment | 16:37 |
| mriedem | fungi: ok, thought maybe related b/c i'm failing on py27 with this now: http://paste.openstack.org/show/64305/ | 16:37 |
| mriedem | and i'm guessing i'm backlevel on something | 16:37 |
| fungi | mriedem: that's clarkb's favorite problem in latest tox | 16:38 |
| fungi | unrelated | 16:38 |
| mriedem | ah, ok | 16:38 |
| fungi | mriedem: https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror | 16:38 |
| fungi | mriedem: use older tox and you should be okay | 16:38 |
| mriedem | fungi: cool, thanks | 16:38 |
| fungi | at least until clarkb's patch is (hopefully, fingers crossed) accepted and a new tox released with it | 16:39 |
| fungi | mriedem: basically they refactored the {posargs} macro handling in tox.ini to cope better with windows command-line arguments, but in doing so broke quoting behavior | 16:41 |
| mriedem | fungi: yup, quickly scanned through the issue a bit | 16:41 |
| fungi | i'm also amused at the psychology of how quickly a developer's opinion can shift from notabug/wontfix->confirmed when you dangle a patch in front of them | 16:42 |
| *** ruhe_ has quit IRC | 16:43 | |
| jeblair | fungi: ha | 16:43 |
| *** sarob_ has quit IRC | 16:43 | |
| *** DinaBelova_ is now known as DinaBelova | 16:43 | |
| *** sarob has joined #openstack-infra | 16:43 | |
| mattymo | fungi, I'm shocked even more when a patch comes out 4 years ago and you talk to the person at a conference who can literally commit the patch any time still hasn't even 3 months after you pushed him | 16:44 |
| mattymo | :) | 16:44 |
| *** gokrokve has quit IRC | 16:47 | |
| *** gokrokve has joined #openstack-infra | 16:47 | |
| *** sarob has quit IRC | 16:48 | |
| fungi | mattymo: you're talking about me, right? ;) | 16:49 |
| * fungi hopes he's never sat on anyone's patch for 4 years, but it's possible | 16:50 | |
| *** beagles has quit IRC | 16:51 | |
| *** gokrokve has quit IRC | 16:51 | |
| *** boris-42_ has joined #openstack-infra | 16:52 | |
| *** max_lobur has quit IRC | 16:52 | |
| mattymo | haha no | 16:52 |
| *** gokrokve has joined #openstack-infra | 16:52 | |
| mattymo | but I do thank you for some urgent hacks during the summit | 16:53 |
| *** alexpilotti has quit IRC | 16:54 | |
| dkehn | fungi: has there been issue with running tox -epy27 and greenlet errors, that you know of? | 16:57 |
| *** yassine has quit IRC | 16:57 | |
| *** dcramer_ has joined #openstack-infra | 16:58 | |
| *** e0ne_ has quit IRC | 16:59 | |
| jog0 | sdague: http://logs.openstack.org/20/56420/7/check/check-tempest-dsvm-neutron/24b6f66/logs/screen-key.txt.gz?level=DEBUG keystone debug logs are a mess | 17:00 |
| fungi | dkehn: nothing that's come to my attention. do you have an error you can link me to (maybe on paste.openstack.org)? | 17:00 |
| jog0 | dogpile logs everywhere | 17:00 |
| sdague | jog0: everything's a mess :) | 17:00 |
| *** gokrokve has quit IRC | 17:01 | |
| sdague | you will find nothing in the logs that surprises me at this point | 17:01 |
| *** changbl has joined #openstack-infra | 17:02 | |
| sdague | my suggestion, propose a patch to push dogpile to INFO level | 17:02 |
| *** lcheng_ has quit IRC | 17:03 | |
| dkehn | fungi: http://paste.openstack.org/show/64311/, fresh pull from neutron | 17:04 |
| *** thedodd has joined #openstack-infra | 17:04 | |
| *** b3nt_pin has joined #openstack-infra | 17:04 | |
| *** b3nt_pin is now known as beagles | 17:04 | |
| dkehn | fungi: going to rebuild the VM, to rule out anything weird | 17:04 |
| mikal | Is there a timestamp for a comment-added in the gerrit event stream? It seems like there is only a timestamp for patchset creation? | 17:04 |
| *** hashar has quit IRC | 17:07 | |
| openstackgerrit | A change was merged to openstack-infra/config: add tempest-cells-full tests to check queue https://review.openstack.org/72556 | 17:07 |
| *** markmc has quit IRC | 17:08 | |
| *** talluri has joined #openstack-infra | 17:08 | |
| jog0 | sdague: keystone was one step ahead of me https://review.openstack.org/#/c/62068/ | 17:10 |
| fungi | dkehn: no hits in logstash for greenlet.GreenletExit in the past week | 17:11 |
| fungi | dkehn: and doesn't look familiar to me | 17:11 |
| *** wenlock has joined #openstack-infra | 17:11 | |
| dkehn | fungi: ok, will keep you posted. | 17:11 |
| *** sarob has joined #openstack-infra | 17:14 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Cancel obsolete builds on reconfiguration https://review.openstack.org/72042 | 17:17 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Remove double accounting of builds https://review.openstack.org/72043 | 17:17 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Split the merger into a separate process https://review.openstack.org/71628 | 17:17 |
| *** jcoufal has quit IRC | 17:17 | |
| *** CaptTofu has quit IRC | 17:19 | |
| openstackgerrit | Dean Troyer proposed a change to openstack-infra/devstack-gate: Run stack.sh with forced line buffering https://review.openstack.org/72705 | 17:20 |
| *** max_lobur has joined #openstack-infra | 17:21 | |
| *** ianw has quit IRC | 17:21 | |
| sdague | so is ES behind, or dead? - http://logstash.openstack.org/#eyJzZWFyY2giOiIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MjEzOTM0ODk1Nn0= | 17:23 |
| sdague | it looks like 7 hrs delay in test result upload | 17:23 |
| fungi | sdague: yesterday it was still catching up from the surge in job completions | 17:23 |
| fungi | i think clarkb added a few more workers to help keep up | 17:24 |
| zaro | morning | 17:24 |
| sdague | fungi: interesting, any idea why we weren't keeping up? It didn't seem like an exceptionally busy quue | 17:27 |
| openstackgerrit | Edward Raigosa proposed a change to openstack-infra/config: fix some small syntax issues in puppet manifest. https://review.openstack.org/72496 | 17:27 |
| fungi | sdague: zuul was keeping up rather well. but the job volume is much higher than it used to be, i think in part thanks to the freshness rechecking | 17:29 |
| *** Ajaeger1 has joined #openstack-infra | 17:33 | |
| sdague | ah, gotcha | 17:33 |
| *** pcrews has joined #openstack-infra | 17:35 | |
| jog0 | ohhh that that makes sense | 17:36 |
| jog0 | sdague: btw status.openstack.org/elastic-recheck/ is looking nice with the tabs | 17:36 |
| *** max_lobur is now known as max_lobur_afk | 17:36 | |
| *** bhuvan has joined #openstack-infra | 17:36 | |
| sdague | jog0: thanks | 17:37 |
| fungi | it's still hard to tell a clear trend in graphite, but after a couple weeks like this it will probably be a little more apparent i think... | 17:38 |
| fungi | http://graphite.openstack.org/compose/?from=-24hours&height=180&until=now&width=334&bgcolor=ffffff&fgcolor=000000&target=alias(summarize(sumSeries(stats_counts.zuul.pipeline.*.all_jobs),%271h%27),%27All%20Jobs%27)&title=Zuul%20Jobs%20Launched%20(per%20Hour)&_t=0.5136637508581183#1392140133043 | 17:38 |
| fungi | since it didn't get turned on until late wednesday, we have less than a week worth of development patterns to analyze | 17:39 |
| fungi | less than a week's worth of changed development pattern i mean | 17:39 |
| nibalizer | anteaya: its 100% puppetlabs modules being brought in right now | 17:40 |
| fungi | and also, yesterday's issues with nodepool throw the envelope out of its natural shape, i expect | 17:40 |
| *** sarob has quit IRC | 17:40 | |
| *** sarob has joined #openstack-infra | 17:40 | |
| *** pafuent has joined #openstack-infra | 17:40 | |
| fungi | nibalizer: i haven't had a moment to look yet. is it basically just a rename of the newer apache module so that it doesn't conflict with the version we're currently stuck on? | 17:40 |
| openstackgerrit | sahid proposed a change to openstack-infra/elastic-recheck: Adds query for bug #1278988 https://review.openstack.org/72713 | 17:41 |
| uvirtbot | Launchpad bug 1278988 in openstack-ci "wrong fs type, bad option, bad superblock on /dev/loop2" [Undecided,New] https://launchpad.net/bugs/1278988 | 17:41 |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: OpenstackId Documentation https://review.openstack.org/69620 | 17:41 |
| fungi | nibalizer: note that anything we're not conflicting on but which is a dependency, we can probably simply add to the array in install_modules.sh instead | 17:41 |
| *** CaptTofu has joined #openstack-infra | 17:43 | |
| fungi | nibalizer: also, puppet module licenses matter. we need to make sure that any we either import or copy are licensed compatibly with apache license v2 | 17:43 |
| fungi | nibalizer: i'll try to do a license audit when i look it over | 17:43 |
| dkehn | fungi: interesting, only occurrs on the hpcloud built 13.04 images, in my home rack, same os, works fine. I guess that answers that | 17:44 |
| *** smarcet has joined #openstack-infra | 17:44 | |
| openstackgerrit | sahid proposed a change to openstack-infra/elastic-recheck: Adds query for bug #1278988 https://review.openstack.org/72713 | 17:44 |
| uvirtbot | Launchpad bug 1278988 in openstack-ci "wrong fs type, bad option, bad superblock on /dev/loop2" [Undecided,New] https://launchpad.net/bugs/1278988 | 17:44 |
| fungi | dkehn: compare a dpkg -l on both systems and see if the one in hpcloud has packages you don't which could be involved or differing versions of culprit packages | 17:44 |
| *** sarob has quit IRC | 17:45 | |
| *** gokrokve has joined #openstack-infra | 17:45 | |
| *** markmcclain1 has quit IRC | 17:47 | |
| *** amcrn has quit IRC | 17:48 | |
| *** mwagner_lap has joined #openstack-infra | 17:48 | |
| *** gokrokve has quit IRC | 17:49 | |
| *** gokrokve has joined #openstack-infra | 17:49 | |
| pleia2 | fungi: can you take a look at /srv/static/reviewday/reviewday.json and see what the permissions are? trying to figure out if it's a permissiosn issue that https://review.openstack.org/#/c/72682/ will fix or still fighting with apache | 17:49 |
| pleia2 | on static | 17:50 |
| *** gokrokve has quit IRC | 17:54 | |
| *** alexpilotti has joined #openstack-infra | 17:54 | |
| *** gokrokve has joined #openstack-infra | 17:54 | |
| *** prad_ has quit IRC | 17:54 | |
| fungi | pleia2: sure | 17:56 |
| nibalizer | fungi: i think i was mistaken yesterday, i didnt find a dependency on apache in the puppetdb module | 17:57 |
| nibalizer | now, it depended on several other modules, which i pulled in and thats why the commit is such a beast | 17:57 |
| fungi | nibalizer: if they're modules we can use directly from the 'forge, they can just be listed in install_modules.sh without needing to duplicate their actual source in our repository | 17:58 |
| fungi | pleia2: drwxr-xr-x 2 reviewday reviewday 4096 Feb 11 17:37 /srv/static/reviewday/ | 17:58 |
| fungi | pleia2: cannot access /srv/static/reviewday/reviewday.json: No such file or directory | 17:58 |
| nibalizer | oh hrm! yes install_Modules.sh is where most if not all of this should go | 17:58 |
| nibalizer | that will fix the commit significantly | 17:58 |
| pleia2 | fungi: hmm, let's see where I set up to output | 18:00 |
| fungi | pleia2: if the www-data user is going to need write access, then maybe we should write it out to /var/lib/something-it-can-write-to and if necessary carry a symlink in srv to that file | 18:00 |
| jeblair | fungi, clarkb: i think some logstash workers may have stopped around 12:13 | 18:00 |
| fungi | pleia2: not sure who writes the json file | 18:01 |
| *** gokrokve has quit IRC | 18:01 | |
| pleia2 | fungi: it just needs to read it, reviewday user makes it | 18:01 |
| *** gokrokve has joined #openstack-infra | 18:01 | |
| jeblair | fungi, clarkb: rather, i'm suspecting logstash or es may be stuck again | 18:01 |
| fungi | jeblair: sounds like a probable recurrence of whatever happened over the weekend? | 18:01 |
| *** morganfainberg_Z is now known as morganfainberg | 18:02 | |
| jeblair | fungi: oh, it happened over the weekend too? | 18:02 |
| fungi | jeblair: or was that yesterday? or late last week? when you and clarkb were looking into it before | 18:02 |
| jeblair | fungi: something happened last week and clarkb said if it happened again he was going to upgrade logstash because their relase notes said they "fixed lots of bugs" or something | 18:02 |
| *** dizquierdo has joined #openstack-infra | 18:02 | |
| fungi | i love release notes with that extreme level of detail | 18:03 |
| pleia2 | fungi: it should end up in the same spot as the index.html and other output files, which should be /srv/static/reviewday according to the cron job, I'll dig around a bit on my end and see what's up | 18:03 |
| fungi | "1.8. new features! we also, um, fixed some stuff" | 18:03 |
| *** morganfainberg is now known as mrgnfeignbern | 18:03 | |
| fungi | pleia2: so, index.html is in there, but the json file is not | 18:04 |
| *** markmcclain has joined #openstack-infra | 18:04 | |
| fungi | pleia2: this runs from cron? where does it normally log? | 18:04 |
| *** ociuhandu has joined #openstack-infra | 18:04 | |
| *** AlexF has quit IRC | 18:04 | |
| pleia2 | fungi: log file should be in /srv/static/reviewday too | 18:04 |
| *** mrmartin has joined #openstack-infra | 18:05 | |
| *** lcheng has joined #openstack-infra | 18:05 | |
| fungi | pleia2: its 17:37:37 leaked some warnings which got sent to its mailbox, but nothing which looks like a critical failuerw | 18:05 |
| *** gokrokve has quit IRC | 18:06 | |
| pleia2 | fungi: thanks, I'll see if I can figure out why it's not producing the json in production (maybe it's not obeying output directory flag or something) | 18:06 |
| fungi | pleia2: doesn't seem to generate a logfile in /srv/static/reviewday | 18:06 |
| *** AlexF has joined #openstack-infra | 18:07 | |
| fungi | i'm checking the locate db on the off chance it wrote it somewhere else which is indexed | 18:07 |
| fungi | no dice | 18:09 |
| *** rossella_s has quit IRC | 18:10 | |
| *** mrgnfeignbern is now known as morganfainberg | 18:11 | |
| clarkb | morning | 18:14 |
| clarkb | jeblair: its logstash | 18:14 |
| clarkb | jeblair: I restarted all of them again last night | 18:14 |
| jeblair | clarkb: i'm restarting them now. :( | 18:14 |
| clarkb | jeblair: and yes, I think working on upgrading is the next step | 18:15 |
| clarkb | whcih is :( | 18:15 |
| mrmartin | clarkb hi | 18:15 |
| clarkb | I do think there is a bug in the libvirt filters though, I will fix that first maybe it is tickling a logstash bug | 18:15 |
| clarkb | mrmartin: hello | 18:16 |
| clarkb | sdague: jog0 and running those giant queries every 15 minutes seemed to halt ES | 18:16 |
| mrmartin | clarkb: what is the proper way to build a LAMP node for gating tasks, for openstackid and community-portal projects? | 18:16 |
| clarkb | which is why I proposed a change to shift them to running once an hour | 18:16 |
| *** amcrn has joined #openstack-infra | 18:16 | |
| clarkb | mrmartin: so, for unittests I would expect that you just need a php runtime right? | 18:17 |
| clarkb | mrmartin: you don't actually need a running lamp stack | 18:17 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Publish draft docs on draft-docs https://review.openstack.org/70594 | 18:17 |
| mrmartin | clarkb: basically for distribution tarball build, php is enough with drush in case of community portal, anyway we also need some temporary mysql db. | 18:17 |
| mrmartin | but my short-term goal that I just want to build distro tarballs first | 18:18 |
| clarkb | mrmartin: we do have mysql dbs on those slaves | 18:18 |
| mrmartin | clarkb: but openstackid requires php5.4 and community-portal requires php version included precise :) | 18:18 |
| *** gokrokve has joined #openstack-infra | 18:18 | |
| mrmartin | to make the life easy :) | 18:19 |
| mrmartin | ... | 18:22 |
| *** mrmartin has left #openstack-infra | 18:22 | |
| Ajaeger1 | infra team, could I get some reviews on https://review.openstack.org/#/c/70594/ and https://review.openstack.org/#/c/71056/, please? | 18:22 |
| *** mrmartin has joined #openstack-infra | 18:22 | |
| *** mrmartin has quit IRC | 18:23 | |
| *** gokrokve has quit IRC | 18:23 | |
| *** mrodden has joined #openstack-infra | 18:23 | |
| *** mrmartin has joined #openstack-infra | 18:23 | |
| jeblair | mrmartin: the version in precise shouldn't be a problem, but we can't yet install something else on the unit test slaves | 18:25 |
| mrmartin | jeblair, ok I got it, but openstackid depends on php 5.4 | 18:25 |
| jeblair | mrmartin: yep. we told them that was a problem. | 18:25 |
| jeblair | mrmartin: we can install it on servers (which we're only willing to do since trusty will be out by the time it's in production) | 18:26 |
| jeblair | mrmartin: but we can't install it on test nodes without a bit more infrastructure work. | 18:26 |
| mrmartin | I see that also, but do you think they will rewrite the entire app? Or the next lts will contain 5.4? | 18:26 |
| *** rcleere has quit IRC | 18:26 | |
| fungi | jeblair: mrmartin: the assertion at the time was that the production server would go on an ubuntu 14.04 lts vm (once that gets released and we have cloud provider images for it) | 18:26 |
| *** dizquierdo has quit IRC | 18:26 | |
| *** jpich has quit IRC | 18:26 | |
| jeblair | mrmartin: what fungi said | 18:27 |
| mrmartin | ok, but I want to add it somehow to gating process :) | 18:27 |
| *** dane_ has joined #openstack-infra | 18:27 | |
| jeblair | mrmartin: i would love that. there are two choices: | 18:27 |
| mrmartin | can't we go a beta release of 14.04? | 18:27 |
| *** esker has quit IRC | 18:27 | |
| *** AlexF has quit IRC | 18:27 | |
| jog0 | clarkb: sorry :/ | 18:27 |
| fungi | mrmartin: our stable cloud providers don't allow us to upload arbitrary server images (that's coming eventually) | 18:28 |
| jog0 | we need a bigger ES | 18:28 |
| jog0 | or less bugs | 18:28 |
| jeblair | mrmartin: (1) wait until trusty is out (2) configure single-use bare slaves with sudo ability and configure all current jobs so that they drop sudo before starting test runs. | 18:28 |
| clarkb | jog0: better queries :) | 18:28 |
| *** esker has joined #openstack-infra | 18:28 | |
| mrmartin | jeblair: we need to choose (2), I think (1) is unacceptable for reed | 18:28 |
| jeblair | mrmartin: (2) is something that we would like to do, but there are a few steps in order to get there | 18:28 |
| *** resker has joined #openstack-infra | 18:28 | |
| mrmartin | ok, what can we do with the community portal, it requires standard php environment included in precise | 18:29 |
| jeblair | mrmartin: as far as using a beta goes: the cloud providers don't have beta trusty images, and they don't let us upload our own images, so in order to do that, we'd have to have nodepool do a distro-upgrade when preparing the node | 18:29 |
| mrmartin | ? | 18:29 |
| jeblair | mrmartin: that's not easy either | 18:29 |
| jeblair | mrmartin: add php to the jenkins slave manifest (if it isn't already) | 18:30 |
| mrmartin | https://github.com/openstack-infra/groups/blob/master/scripts/bootstrap.sh | 18:30 |
| *** gokrokve has joined #openstack-infra | 18:31 | |
| jog0 | clarkb: we have 65 70 open bugs which odesn't help | 18:31 |
| mrmartin | I need drush also, but I guess, i can move it to local dir, so I can hack it somehow | 18:31 |
| clarkb | jog0: ya, but I also think we can write smarter queries. I see a lot of unrestricted message:"foo" queries floating around | 18:31 |
| *** esker has quit IRC | 18:33 | |
| *** resker has quit IRC | 18:33 | |
| mrmartin | jeblair: ok, thanks, I'll extend the jenkins slave if required | 18:33 |
| *** chandan_kumar has quit IRC | 18:34 | |
| jog0 | clarkb: what would make them better? | 18:34 |
| jog0 | adding a file? | 18:34 |
| *** dane_ has quit IRC | 18:35 | |
| jeblair | clarkb: all logstash workers restarted 1-16 A-D | 18:37 |
| clarkb | jog0: pretty sure that being as specific as possible helps | 18:39 |
| clarkb | jog0: otherwise you have to loda up more data into memory | 18:39 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Suppress bug 1269940 from the graphs page https://review.openstack.org/72731 | 18:40 |
| uvirtbot | Launchpad bug 1269940 in openstack-ci "[EnvInject] - [ERROR] - SEVERE ERROR occurs:" [Undecided,Invalid] https://launchpad.net/bugs/1269940 | 18:40 |
| *** dane_ has joined #openstack-infra | 18:40 | |
| *** melwitt has joined #openstack-infra | 18:41 | |
| clarkb | jeblair: thanks, did you also restart the logstash indexers on the hosts that had not been in use? | 18:41 |
| clarkb | jeblair: they may need restarting to deal with the y2k problem | 18:41 |
| jeblair | clarkb: yes -- i believe it was actually the indexers that were stuck | 18:42 |
| clarkb | I am truncating logstash indexer logs on the hosts, they have gotten quite large (after grabbing the last 10k lines and putting them in a different file), will work on libvirt timestamp parsing after this, then look at upgrading | 18:43 |
| jog0 | clarkb: if you can give me an example I can go through and fix the rest | 18:43 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revert "Reduce rax-iad usage by 50%" https://review.openstack.org/72732 | 18:43 |
| jeblair | clarkb: cool, thx | 18:43 |
| *** prad has joined #openstack-infra | 18:45 | |
| *** alexpilotti has quit IRC | 18:47 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Also copy tempest.config for old with grenade https://review.openstack.org/72736 | 18:47 |
| *** pafuent has left #openstack-infra | 18:48 | |
| *** rcleere has joined #openstack-infra | 18:49 | |
| *** e0ne has joined #openstack-infra | 18:49 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: [WIP] Extracted DB API layer https://review.openstack.org/72356 | 18:51 |
| *** ociuhandu has quit IRC | 18:52 | |
| *** moted has quit IRC | 18:53 | |
| *** markmcclain has quit IRC | 18:53 | |
| *** markmcclain has joined #openstack-infra | 18:54 | |
| *** tykeal has joined #openstack-infra | 18:55 | |
| *** pblaho has joined #openstack-infra | 18:58 | |
| *** jcoufal has joined #openstack-infra | 18:59 | |
| *** gokrokve has quit IRC | 18:59 | |
| *** jgrimm has quit IRC | 19:00 | |
| *** gokrokve has joined #openstack-infra | 19:00 | |
| mattoliverau | Good morning all, a very early morning for me. | 19:00 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: [WIP] Extracted DB API layer https://review.openstack.org/72356 | 19:00 |
| fungi | meetin' time | 19:00 |
| *** jgrimm has joined #openstack-infra | 19:00 | |
| *** olaph has joined #openstack-infra | 19:01 | |
| *** cadenzajon has joined #openstack-infra | 19:04 | |
| *** markmcclain1 has joined #openstack-infra | 19:05 | |
| *** markmcclain has quit IRC | 19:06 | |
| *** markmcclain has joined #openstack-infra | 19:07 | |
| *** apevec has joined #openstack-infra | 19:10 | |
| *** markmcclain1 has quit IRC | 19:10 | |
| *** sarob has joined #openstack-infra | 19:11 | |
| apevec | Hi, change-id links in Gerrit are rendered like https://review.openstack.org/#q,$1,n,z - known issue I guess? | 19:12 |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: OpenstackId Documentation https://review.openstack.org/69620 | 19:13 |
| fungi | apevec: yes, since the last gerrit restart, seems one of the changes to the hyperlink generation config wasn't as well-tested as we thought | 19:14 |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: OpenstackId Documentation https://review.openstack.org/69620 | 19:16 |
| *** coolsvap has quit IRC | 19:17 | |
| *** whoops has joined #openstack-infra | 19:17 | |
| *** dstanek has quit IRC | 19:18 | |
| *** esker has joined #openstack-infra | 19:19 | |
| *** moted has joined #openstack-infra | 19:19 | |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: OpenstackId Documentation https://review.openstack.org/69620 | 19:20 |
| *** jp_at_hp has quit IRC | 19:21 | |
| *** dstanek has joined #openstack-infra | 19:24 | |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: OpenstackId Documentation https://review.openstack.org/69620 | 19:25 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Copy tempest.conf file with log files https://review.openstack.org/72131 | 19:26 |
| *** igor___ has quit IRC | 19:29 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Remove exercises tests from grenade runs https://review.openstack.org/71010 | 19:30 |
| *** mrmartin has quit IRC | 19:30 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Extracted DB API layer https://review.openstack.org/72356 | 19:31 |
| *** MarkAtwood has joined #openstack-infra | 19:32 | |
| *** sarob has quit IRC | 19:33 | |
| jaypipes | fungi: hi Jeremy. I'm struggling to figure out how the Zuul puppet manifest (or anything really) sets up the Gearman workers in the openstack-infra/config project. Was hoping you might have a moment to chat? | 19:33 |
| fungi | jaypipes: we're in our weekly meeting right now, but i can discuss in about half an hour | 19:34 |
| jaypipes | fungi: sure, that would be great. much appreciated! | 19:34 |
| fungi | also, jeblair/clarkb/zaro may have a more detailed understanding of it than i do, but i can give it a shot | 19:34 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Do not force log verbosity level to info https://review.openstack.org/71374 | 19:36 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Parse timestamps in libvirt logs properly. https://review.openstack.org/72747 | 19:38 |
| mriedem | sdague: can you get here? http://dal05.objectstorage.softlayer.net/v1/AUTH_58396f85-2c60-47b9-aaf8-e03bc24a1a6f/comlog/logs/97/67297/13/check/gate-ci-devstack-test/072f97a/logs/ | 19:40 |
| openstackgerrit | A change was merged to openstack-infra/config: Parse timestamps in libvirt logs properly. https://review.openstack.org/72747 | 19:41 |
| *** ruhe_ has joined #openstack-infra | 19:41 | |
| dkehn | fungi: http://paste.openstack.org/show/64355/ here's the diff, alot of things that are different | 19:42 |
| *** jamielennox is now known as jamielennox|away | 19:43 | |
| *** jamielennox|away is now known as jamielennox | 19:43 | |
| *** dstanek has quit IRC | 19:44 | |
| fungi | dkehn: yeah, narrowing that down wouldn't be easy unless you happen to already have some inkling as to which locally-installed libraries could contribute to the problem | 19:44 |
| dkehn | fungi: I'm just happy that it works here | 19:45 |
| *** saju_m has quit IRC | 19:49 | |
| *** dstanek has joined #openstack-infra | 19:49 | |
| openstackgerrit | A change was merged to openstack/requirements: Sort global-requirements https://review.openstack.org/64943 | 19:49 |
| *** ruhe_ has quit IRC | 19:50 | |
| *** saju_m has joined #openstack-infra | 19:51 | |
| *** markmc has joined #openstack-infra | 19:53 | |
| *** bhuvan has quit IRC | 19:53 | |
| *** gokrokve has quit IRC | 19:53 | |
| *** gokrokve has joined #openstack-infra | 19:54 | |
| *** gokrokve_ has joined #openstack-infra | 19:55 | |
| *** ruhe- has joined #openstack-infra | 19:55 | |
| *** jgrimm has quit IRC | 19:56 | |
| *** ruhe- has quit IRC | 19:56 | |
| *** ianw has joined #openstack-infra | 19:57 | |
| *** ruhe has quit IRC | 19:57 | |
| *** gyee has quit IRC | 19:58 | |
| *** gokrokve has quit IRC | 19:58 | |
| *** ruhe has joined #openstack-infra | 19:58 | |
| openstackgerrit | Alexander Jones proposed a change to openstack-infra/git-review: Fix parsing of SCP-style URLs, as these are valid in Git itself https://review.openstack.org/72751 | 19:58 |
| openstackgerrit | Alexander Jones proposed a change to openstack-infra/git-review: Properly work around the fact that ssh: isn't a real URI scheme https://review.openstack.org/72752 | 19:58 |
| *** pblaho has quit IRC | 19:59 | |
| *** dcramer_ has quit IRC | 19:59 | |
| *** gokrokve_ has quit IRC | 19:59 | |
| *** gokrokve has joined #openstack-infra | 20:00 | |
| *** ArxCruz has quit IRC | 20:00 | |
| *** gokrokve_ has joined #openstack-infra | 20:01 | |
| *** rfolco has quit IRC | 20:01 | |
| clarkb | logstash update, I think I have fixed the libvirt timestamp parsing. we seem to be indexing at a record rate after restarting all of the indexers to pick up that config change. I hope this helps things dramatically but don't have a ton of hope | 20:01 |
| clarkb | I also found that some indexers were running out of heap space according to their logs when doing grok parses | 20:01 |
| clarkb | which may explain their weird deaths | 20:01 |
| *** thedodd has quit IRC | 20:02 | |
| fungi | jaypipes: so specifically you're wondering where gearman workers come into the zuul picture. in our case we're using jenkins masters which run a jenkins-gearman plugin and are configured to connect to our zuul server's internal gearman service | 20:02 |
| fungi | i believe, though could be wrong, that we currently manually install and configure that plugin on each jenkins master | 20:03 |
| sdague | mriedem: yep | 20:03 |
| *** hashar has joined #openstack-infra | 20:03 | |
| SergeyLukjanov | fungi, clarkb, jeblair, mordred, could I help somehow with manage-projects? | 20:03 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add bugdaystats to openstack-infra https://review.openstack.org/69489 | 20:03 |
| mriedem | sdague: cool | 20:03 |
| *** saju_m has quit IRC | 20:04 | |
| jaypipes | fungi: yup, after grepping through the config tree for "gearman" I finally found it... I'd removed it from my puppet manifest for my ci.pp before since I thought I'd be able to use the Gerrit Jenkins trigger plugin... | 20:04 |
| jaypipes | fungi: thx! | 20:04 |
| *** gokrokve has quit IRC | 20:04 | |
| *** saju_m has joined #openstack-infra | 20:05 | |
| fungi | SergeyLukjanov: potentially. at this point we're trying to figure out whether it's still broken, and whether the logoutput parameter we added to the puppet exec for manage-projects is going to log errors/tracebacks into syslog with the rest of the puppet loglines | 20:05 |
| SergeyLukjanov | fungi, yup, I've reviewed this changes | 20:05 |
| fungi | SergeyLukjanov: i did confirm that it includes them when manually running puppet agent --test but the ways puppet logs things varies between invocation in an interactive shell and when running as an unattended daemon | 20:06 |
| mattoliverau | Time to go wake a with a coffee or 5 :P brb | 20:06 |
| mattoliverau | *wake up | 20:06 |
| SergeyLukjanov | fungi, got it | 20:07 |
| SergeyLukjanov | fungi, AFAIR it was working ok if run manually, isn't it? | 20:07 |
| fungi | SergeyLukjanov: the last time we ran it automated for a new project change, it seemed to create the project in gerrit just fine and manage-projects was silent (which could mean puppet logged the empty output, or that puppet is filtering that output and not adding it to the syslog) | 20:07 |
| fungi | SergeyLukjanov: and yes, every time we ran manage-projects directly at the command line or via puppet agent --test it completed successfully without any errors | 20:09 |
| SergeyLukjanov | fungi, heh | 20:09 |
| fungi | though we did have one hiccup on the last run, which is that it told gerrit to start replicating but the repository creation on the git server farm is asynchronous and hadn't happened yet, so once it ran to create those repositories there they sat empty since the replication had already happened and failed | 20:10 |
| *** dguitarbite_ has joined #openstack-infra | 20:10 | |
| fungi | but i think we can determine other ways to address that | 20:10 |
| fungi | possibly salt reactors, or some sort of callback from the git farm to the gerrit servers to retrigger replication when the repos get created | 20:11 |
| SergeyLukjanov | fungi, probably we need to make this script easier and separate it to several scripts | 20:11 |
| dguitarbite_ | hey fungi: here is the gdoc link, https://docs.google.com/a/aptira.com/document/d/1w7XhcJzdZi_DMT4eenA9rBY7n3uPieCzIfllaQZkXag/edit?usp=drive_web | 20:11 |
| fungi | SergeyLukjanov: that too. mordred had suggested splitting out, for example, the project creation on gerrit and the git remote/replication bits to happen in a disconnected/async manner | 20:11 |
| dguitarbite_ | you mat have to request access, its owned by sarob: | 20:11 |
| SergeyLukjanov | fungi, yup, it could help to achieve repositories create | 20:12 |
| fungi | dguitarbite_: checking. i hardly ever touch google docs | 20:12 |
| SergeyLukjanov | fungi, I'll try to take a look on manage-projects script tomorrow, probably we'll be able to generate some not very crazy ideas | 20:12 |
| *** zehicle_at_dell has joined #openstack-infra | 20:13 | |
| fungi | SergeyLukjanov: manage-projects is already a very crazy idea, so i think any fixes we come up with will be at least a s crazy ;) | 20:13 |
| *** talluri has quit IRC | 20:13 | |
| SergeyLukjanov | fungi, oh, I'd like crazy fixes | 20:13 |
| clarkb | after the recent logstash indexer restart to pick up the config change the gearman queue has fallen fro 90k items to 75k items | 20:13 |
| clarkb | in about half an hour | 20:13 |
| clarkb | this is looking up, I will keep an eye on it | 20:14 |
| SergeyLukjanov | fungi, w/o 'd, I like crazy fixes always :) | 20:14 |
| dguitarbite_ | fungi: ok | 20:14 |
| fungi | dguitarbite_: access requested for fungi-google@yuggoth.org | 20:15 |
| fungi | sarob will presumably approve it when he sees the notification | 20:15 |
| dguitarbite_ | yep | 20:16 |
| *** gokrokve_ has quit IRC | 20:16 | |
| *** cadenzajon has quit IRC | 20:16 | |
| *** gokrokve has joined #openstack-infra | 20:16 | |
| *** cadenzajon has joined #openstack-infra | 20:17 | |
| *** moted has quit IRC | 20:18 | |
| fungi | " | 20:19 |
| *** jcoufal has quit IRC | 20:19 | |
| fungi | grr, window focus bites again | 20:19 |
| *** gokrokve has quit IRC | 20:21 | |
| openstackgerrit | A change was merged to openstack/requirements: Update openstack-doc-tools to 0.6 https://review.openstack.org/69685 | 20:21 |
| *** oanufriev has quit IRC | 20:24 | |
| fungi | clarkb: mordred(if you're around): this is an odd error in requirements integration for a stable/havana change... http://logs.openstack.org/09/62209/1/gate/gate-requirements-integration-dsvm/bae65cd/console.html.gz | 20:24 |
| *** harlowja_away is now known as harlowja | 20:24 | |
| fungi | i can't help but feel like there's something we need to fix in havana for latest virtualenv/pip to work in that job | 20:25 |
| *** jcoufal has joined #openstack-infra | 20:25 | |
| *** denis_makogon_ has joined #openstack-infra | 20:26 | |
| fungi | i'm watching zuul as it gets rechecked, but suspect it's going to fail again | 20:28 |
| *** dguitarbite_ has quit IRC | 20:29 | |
| jaypipes | LOL, I think this is the 10th time today I've spawned a new LXC container to install Jenkins/Zuul/JJB :) | 20:29 |
| * jaypipes twiddles fingers watching puppet apply... | 20:29 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Add cliff, pycadf, stevefore & taskflow https://review.openstack.org/72487 | 20:30 |
| *** jcoufal has quit IRC | 20:31 | |
| *** saju_m has quit IRC | 20:32 | |
| *** nkinder has joined #openstack-infra | 20:32 | |
| *** chris_johnson is now known as wchrisj|away | 20:33 | |
| *** sarob has joined #openstack-infra | 20:35 | |
| *** wchrisj|away is now known as chris_johnson | 20:36 | |
| *** dcramer_ has joined #openstack-infra | 20:37 | |
| *** moted has joined #openstack-infra | 20:39 | |
| *** sarob has quit IRC | 20:40 | |
| jaypipes | w00t. zuul, gearman, and company are rocking. | 20:42 |
| *** pblaho has joined #openstack-infra | 20:44 | |
| *** denis_makogon_ is now known as denis_makogon | 20:45 | |
| openstackgerrit | A change was merged to openstack-infra/reviewday: Ensure WORKINPROGRESS reviews are at the bottom of the list. https://review.openstack.org/71358 | 20:46 |
| *** DinaBelova is now known as DinaBelova_ | 20:47 | |
| jaypipes | clarkb, fungi: Seeing this in the zuul debug.log: "WARNING zuul.Scheduler: Project openstack-dev/sandbox not found". What exactly does that mean again? Does it have to do with the zuul layout.yaml config, or Jenkins Job Builder, or something to do with Gerrit/Git? Sorry for bother. | 20:48 |
| fungi | jaypipes: is openstack-dev/sandbox in the projects list in your layout.yaml? | 20:51 |
| *** rlandy has quit IRC | 20:52 | |
| *** jraim has joined #openstack-infra | 20:52 | |
| *** dane_ has quit IRC | 20:52 | |
| jaypipes | fungi: yes. https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/zuul/layout.yaml#L51 | 20:52 |
| *** ryanpetrello has joined #openstack-infra | 20:55 | |
| sdague | anyone else want to +2 this - https://review.openstack.org/#/c/72705/ ? it should dramatically help bringing -o errexit to devstack, which should help us pin point errors better | 20:56 |
| SergeyLukjanov | jaypipes, have you already send any events to it? | 20:56 |
| SergeyLukjanov | sdague, looking on it atm | 20:56 |
| jaypipes | SergeyLukjanov: nope, not sent any yet, but receiving the stream just fine. | 20:56 |
| jaypipes | SergeyLukjanov: debug.log is humming along nicely, but I think my layout.yaml may not be correct... | 20:57 |
| jaypipes | SergeyLukjanov: since for every comment event, I see a corresponding message like so: WARNING zuul.Scheduler: Project stackforge/solum not found | 20:57 |
| *** oanufriev has joined #openstack-infra | 20:57 | |
| jaypipes | SergeyLukjanov: which I would expect, but I expect to see my trigger for the openstack-dev/sandbox project fire... | 20:58 |
| ruhe | jaypipes: can you confirm that jenkins registered jobs in zuul? | 20:58 |
| ruhe | jaypipes: telnet zuul_host zuul_port | 20:58 |
| ruhe | and run status | 20:58 |
| ruhe | *zuul_gearman_port | 20:58 |
| *** dane_ has joined #openstack-infra | 21:00 | |
| *** amcrn has quit IRC | 21:00 | |
| SergeyLukjanov | sdague, +2 from me for https://review.openstack.org/#/c/72705/ | 21:00 |
| SergeyLukjanov | sdague, and +2/+A from fungi :) | 21:00 |
| jaypipes | ruhe: that might be the issue.... | 21:01 |
| jaypipes | :) | 21:01 |
| *** amcrn has joined #openstack-infra | 21:01 | |
| SergeyLukjanov | ruhe, so, it's a zuul-jenkins connection problem? | 21:01 |
| jaypipes | ruhe: for some reason I thought my puppet module ran jenkins-jobs update, but apparently not... | 21:01 |
| jaypipes | also, is this expected? | 21:02 |
| jaypipes | ubuntu@os_ext_testing:~$ sudo service zuul stop | 21:02 |
| jaypipes | No process in pidfile '/var/run/zuul/zuul.pid' found running; none killed. | 21:02 |
| jaypipes | I have to go manually kill processes in order to shut down Zuul :( | 21:02 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/gear: Solve hacking/pbr requirements conflict https://review.openstack.org/72767 | 21:03 |
| vishy | anyone in here know how to make pip pass environment variables when using -e | 21:03 |
| vishy | ? | 21:03 |
| ruhe | SergeyLukjanov: yes. most of the problems i had in this area were because jenkins didn't register it's jobs in gearman server | 21:04 |
| jaypipes | ruhe: last time I set up zuul, there was no gearman server, so I'm still figuring that piece out :) | 21:04 |
| ruhe | zuul serves gearman server | 21:05 |
| jaypipes | ruhe: I know it does *now*... but that is relatively recent :) | 21:05 |
| vishy | I"m trying very hard to get pip to respect my SSH_AUTH_SOCK when it uses git+ssh but it won't do it | 21:05 |
| *** tykeal has left #openstack-infra | 21:05 | |
| fungi | dstufft: ^ ? | 21:05 |
| dstufft | I have to go pick up my daughter, I can take a look in like 30-60 minutes | 21:06 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Allow welcome-message to configure logging https://review.openstack.org/72435 | 21:07 |
| ruhe | jaypipes: about puppet and jenkins-jobs. it will not happen on the first run, because there is a manual step expected | 21:07 |
| fungi | dstufft: no sweat, just thought you might know the answer to vishy's question off the top of your head | 21:07 |
| *** e0ne has quit IRC | 21:07 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Reduce rax-iad usage by 50%" https://review.openstack.org/72732 | 21:07 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/gear: Remove tox locale overrides https://review.openstack.org/72216 | 21:08 |
| dprince | pleia2: So 0o644 is the python way os making that file 644. Did I misread your comment? | 21:09 |
| *** sarob has joined #openstack-infra | 21:10 | |
| *** lcostantino has quit IRC | 21:10 | |
| pleia2 | dprince: you have 664, not 664 | 21:12 |
| pleia2 | err 644 | 21:12 |
| pleia2 | need to change one your 6 to a 4 :) | 21:12 |
| fungi | pleia2: did you manage to figure out why the file wasn't being created? | 21:13 |
| pleia2 | fungi: so it's created with permissions 600 in testing, which is wrong for one (so dprince's patch will help) | 21:13 |
| fungi | ahh, but still wouldn't have kept the file from being created at all, right? | 21:14 |
| pleia2 | fungi: and if you navigate to http://status.openstack.org/reviews/blah it gives you a 404, not a permission denied (which you get for the json file) | 21:14 |
| pleia2 | so something is going on here | 21:14 |
| pleia2 | was hoping to get the permission change in and then go from there | 21:15 |
| *** jgrimm has joined #openstack-infra | 21:15 | |
| mtreinish | clarkb: on https://review.openstack.org/#/c/70378/ what do we need to do to communicate this so we can get it merged? | 21:16 |
| mtreinish | I don't think it'll break the gate because swiftclient is used in tempest tests and by glance so it shouldn't be broken | 21:16 |
| *** sarob_ has joined #openstack-infra | 21:22 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/gear: Solve hacking/pbr requirements conflict https://review.openstack.org/72767 | 21:22 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/gear: Remove tox locale overrides https://review.openstack.org/72216 | 21:22 |
| *** sarob has quit IRC | 21:24 | |
| *** markmc has quit IRC | 21:24 | |
| *** pcrews has quit IRC | 21:25 | |
| *** gyee has joined #openstack-infra | 21:26 | |
| *** jamielennox is now known as jamielennox|away | 21:26 | |
| *** jamielennox|away is now known as jamielennox | 21:29 | |
| *** thedodd has joined #openstack-infra | 21:30 | |
| *** yolanda has quit IRC | 21:30 | |
| fungi | Alex_Gaynor: i know shutil.rmtree can sometimes be racy... is that the case in pypy too? http://logs.openstack.org/19/72219/1/check/gate-release-tools-pypy/26c3210/console.html | 21:30 |
| Alex_Gaynor | fungi: Probably, our implementation of shutil is identical to upstream CPythons | 21:33 |
| dstufft | vishy: looks like pip should be passing the current environment into the command | 21:33 |
| dstufft | into the sub process I mean | 21:33 |
| fungi | Alex_Gaynor: okay, good to know. thanks! | 21:34 |
| *** amcrn has quit IRC | 21:34 | |
| *** gokrokve has joined #openstack-infra | 21:36 | |
| fungi | dstufft: in that case, pip uses shutil.rmtree which is known to be racy and seems to not catch OSError when thrown... http://logs.openstack.org/19/72219/1/check/gate-release-tools-pypy/26c3210/console.html (worth me opening a bug in the tracker for pip?) | 21:37 |
| dstufft | hrm | 21:38 |
| dstufft | ignore_errors doesn't infact, ignore_errors? | 21:38 |
| fungi | it's been a while since i looked at the python bug for that, but if memory serves sometimes rmtree can remove things in the wrong order and then try to reference them, so if you're going to use it you have to look out for that | 21:38 |
| dstufft | fungi: yea open a bug | 21:38 |
| * fungi could be completely off base with that though | 21:38 | |
| fungi | will do. thanks dstufft! | 21:38 |
| fungi | i'm breaking all the things today | 21:39 |
| *** sarob_ has quit IRC | 21:39 | |
| *** amcrn has joined #openstack-infra | 21:41 | |
| *** beagles has quit IRC | 21:42 | |
| *** jnoller_ has quit IRC | 21:42 | |
| *** jhesketh_ has joined #openstack-infra | 21:43 | |
| *** markmcclain has quit IRC | 21:44 | |
| jaypipes | ruhe: http://paste.openstack.org/show/64400/ | 21:45 |
| *** thomasem has quit IRC | 21:45 | |
| jaypipes | ruhe: so gearman is working, jenkins is working (and has a JJB-created job in it called sandbox-dsvm-tempest-full that has this configuration: https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/jenkins_jobs/config/sandbox.yaml | 21:46 |
| jaypipes | ruhe: but zuul (or gearman?) does not seem to be picking up the jenkins job... | 21:46 |
| jaypipes | SergeyLukjanov: ^^ you too :) if you have any ideas... | 21:47 |
| *** apevec has left #openstack-infra | 21:49 | |
| SergeyLukjanov | clarkb, https://review.openstack.org/#/c/70378/ approve? | 21:50 |
| openstackgerrit | Walter A. Boring IV (hemna) proposed a change to openstack/requirements: Update to new hp3parclient https://review.openstack.org/72776 | 21:50 |
| SergeyLukjanov | jaypipes, looking | 21:50 |
| SergeyLukjanov | jaypipes, heh, no ideas so far | 21:51 |
| SergeyLukjanov | jaypipes, checking my zuul | 21:51 |
| *** smarcet has left #openstack-infra | 21:51 | |
| SergeyLukjanov | jaypipes, hm, looks like that means no jobs running now | 21:52 |
| jaypipes | SergeyLukjanov: DOH. | 21:53 |
| fungi | mtreinish: did you give notmyname a heads up on https://review.openstack.org/70378 ? | 21:53 |
| jaypipes | err: /Stage[main]/Os_ext_testing::Ci/File[/etc/zuul/layout.yaml]: Could not evaluate: Could not retrieve information from environment production source(s) file:/etc/zuul/layout.yaml at /home/ubuntu/os-ext-testing/puppet/modules/os_ext_testing/manifests/ci.pp:193 | 21:53 |
| jaypipes | SergeyLukjanov: silly me... | 21:53 |
| *** pcrews has joined #openstack-infra | 21:53 | |
| sdague | now that russellb's numproc patch landed, the rax nodes are our fast ones. | 21:53 |
| jaypipes | SergeyLukjanov: I had not copied the layout.yaml into /etc/zuul :( | 21:53 |
| mtreinish | fungi: I can't remember | 21:53 |
| sdague | but they have the advantage of having 2x the cores :) | 21:53 |
| SergeyLukjanov | jaypipes, IMO such fails are most difficult ever to debug :) | 21:53 |
| fungi | mtreinish: i took that as the gist of clarkb's comment when he +2'd it | 21:54 |
| jaypipes | SergeyLukjanov: indeed :) though when there is an error yelling at me like above, and I ignore it.. it's all my fault :) | 21:54 |
| SergeyLukjanov | jaypipes, fyi how it should looks like - http://paste.openstack.org/show/64405/ | 21:55 |
| jaypipes | yup.. | 21:56 |
| clarkb | I am back from lunch | 21:57 |
| clarkb | I have been pinged | 21:57 |
| mtreinish | clarkb: that was me (or at least me) | 21:59 |
| *** sarob has joined #openstack-infra | 21:59 | |
| clarkb | ah yeah, I figured we should at least make people aware of it | 22:00 |
| *** sarob has quit IRC | 22:00 | |
| *** sarob has joined #openstack-infra | 22:01 | |
| *** prad has quit IRC | 22:02 | |
| fungi | i added the ptl as a requested reviewer for a heads-up on it | 22:02 |
| *** harlowja has quit IRC | 22:03 | |
| jaypipes | SergeyLukjanov, jeblair: is there a recommended way to start up zuul-server? I've tried sudo service zuul start, sudo -uzuul service zuul start, etc, and each time that leaves me unable to stop zuul-server (something about the pid file not being found). And if I try to do a sudo killall zuul-server, instead of killing the processes, it sends one into a defunct status and the other hangs around. I end up having to ki | 22:04 |
| jaypipes | ll -9 the processes to force a zuul-server shutdown. | 22:04 |
| *** Ajaeger1 has quit IRC | 22:05 | |
| SergeyLukjanov | jaypipes, is pid file exists? | 22:05 |
| fungi | jaypipes: does /var/run/zuul/zuul.pid ever get created when you start it using the initscript? | 22:05 |
| jeblair | jaypipes: i use the init script; /etc/init.d/zuul start/stop. there's a caveat: | 22:05 |
| jaypipes | SergeyLukjanov: no, it does not exist. | 22:05 |
| jaypipes | SergeyLukjanov: ever... at least as far as I can tell. | 22:05 |
| SergeyLukjanov | so, looks like misconfiguration? | 22:06 |
| SergeyLukjanov | I'm using init.d too | 22:06 |
| jeblair | jaypipes: stop will not remove the pid file | 22:06 |
| *** sarob has quit IRC | 22:06 | |
| *** dstanek has quit IRC | 22:06 | |
| jeblair | jaypipes: however, gracefully shutting down zuul will | 22:06 |
| jaypipes | jeblair: what was the caveat? :) | 22:06 |
| *** prad has joined #openstack-infra | 22:07 | |
| jeblair | jaypipes: but the init script doesnn't have a command for that. it's SIGUSR1. | 22:07 |
| jeblair | jaypipes: basically, the init script needs two patches: | 22:07 |
| jeblair | jaypipes: a) add a "graceful-stop" command which calls the do_graceful_stop() method which already exists in the init script | 22:07 |
| fungi | jaypipes: one possibility... zuul is running as an unprivileged account when it creates its pidfile. is the /var/run/zuul directory allowing it to write there, permissions-wise? | 22:07 |
| jeblair | jaypipes: b) have the stop command rm the pid file | 22:08 |
| jaypipes | fungi: I believe so (it's using the openstack-infra/config zuul manifest to install. | 22:08 |
| jeblair | jaypipes: then we should be good. | 22:08 |
| jaypipes | jeblair: k. I'll play around with it and if I get things going, I will submit a patch. | 22:08 |
| jeblair | jaypipes: keep in mind that the graceful method is the better way to stop zuul; "stop" is more of a hard stop. | 22:09 |
| fungi | jaypipes: so permissions are 0755 zuul:zuul? | 22:09 |
| jaypipes | will do. | 22:09 |
| jaypipes | but to be clear, does it matter whether I run the commands as the zuul or the root user/ | 22:09 |
| jaypipes | ? | 22:09 |
| *** dstanek has joined #openstack-infra | 22:09 | |
| fungi | the initscript may need to be run as root (or via sudo) | 22:10 |
| jeblair | fungi: ++ | 22:10 |
| SergeyLukjanov | have a good day/evening, I'm falling asleep | 22:10 |
| jaypipes | kk | 22:10 |
| jaypipes | thx guys | 22:10 |
| fungi | it will in turn launch the zuul process with the appropriate owner | 22:10 |
| jeblair | SergeyLukjanov: good night! | 22:10 |
| fungi | thanks for all the help SergeyLukjanov! | 22:10 |
| anteaya | night SergeyLukjanov | 22:11 |
| *** masayukig has quit IRC | 22:11 | |
| jaypipes | jeblair, fungi: is there supposed to be a pidfile in /var/run/zuul/? http://paste.openstack.org/show/64406/ | 22:12 |
| jeblair | jaypipes: yes | 22:12 |
| jaypipes | jeblair: so this is a bug I'm seeing? | 22:13 |
| jeblair | jaypipes: http://paste.openstack.org/show/64408/ | 22:13 |
| *** boris-42_ has quit IRC | 22:13 | |
| *** ryanpetrello has quit IRC | 22:13 | |
| jaypipes | jeblair: right, that's what I was expecting to see in there :) | 22:14 |
| ruhe | jaypipes: maybe there was a problem during zuul start? what do the logs say? | 22:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use a sudoers file for jenkins sudo rights https://review.openstack.org/72777 | 22:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revoke sudo permissions before infra jobs https://review.openstack.org/72778 | 22:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revoke sudo from most jobs https://review.openstack.org/72779 | 22:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Allow bare-precise nodes to sudo https://review.openstack.org/72780 | 22:16 |
| *** dcramer_ has quit IRC | 22:17 | |
| jeblair | clarkb, fungi: ^ i think those are all the steps needed for "sudo on bare slaves". | 22:17 |
| jeblair | that's a one-at-a-time kind of series. | 22:17 |
| clarkb | jeblair: we want to allow that? | 22:17 |
| clarkb | oh I see you are revoking it for most jobs | 22:18 |
| jeblair | clarkb: not for openstack, but it seems that it is potentially useful for other things (like crazy php tests, etc) | 22:18 |
| jeblair | clarkb: indeed | 22:18 |
| *** e0ne has joined #openstack-infra | 22:18 | |
| *** pdmars has quit IRC | 22:20 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Run stack.sh with forced line buffering https://review.openstack.org/72705 | 22:21 |
| *** e0ne has quit IRC | 22:22 | |
| *** crank has quit IRC | 22:22 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Spin up a bare-centos6 slave https://review.openstack.org/72784 | 22:27 |
| clarkb | caught one. logstash-worker02 ran out of memory and disconnected from the cluster | 22:27 |
| jeblair | clarkb: oh, did it log that error? | 22:27 |
| clarkb | yup | 22:28 |
| jeblair | clarkb: i don't remember seeing anything like that in the indexer.log | 22:28 |
| clarkb | jeblair: tail /var/log/logstash/indexer.log, be careful it is one giant line | 22:28 |
| fungi | when it runs out of memory, is it getting zapped by the kernel oom killer? | 22:28 |
| *** mfer has quit IRC | 22:28 | |
| jeblair | clarkb: oh, i did see a line with like 12 million 2 digit numbers | 22:28 |
| clarkb | jeblair: they are there just harder to see because of the timestamp parse failure noise we had before | 22:28 |
| *** dprince has quit IRC | 22:28 | |
| clarkb | jeblair: thats it | 22:28 |
| jeblair | clarkb: no wonder it's out of memory. | 22:28 |
| clarkb | ha | 22:28 |
| jaypipes | jeblair: aha! http://paste.openstack.org/show/64413/ (looks like I need to inject known hosts into my LXC instance) | 22:28 |
| jeblair | jaypipes: yes, or i think it would be okay for zuul to accept the host key on first connection (but never accept a changed key) | 22:29 |
| *** dcramer_ has joined #openstack-infra | 22:30 | |
| jaypipes | jeblair: does it need to be in /root/.ssh or /var/lib/zuul/.ssh/? the latter, yes? | 22:31 |
| jeblair | jaypipes: (that last thing is a patch idea, just to be clear) | 22:31 |
| *** MarkAtwood has quit IRC | 22:31 | |
| jaypipes | jeblair: yes, understood that :) | 22:31 |
| *** dims has quit IRC | 22:31 | |
| jeblair | jaypipes: i think ~zuul/.ssh | 22:32 |
| *** senthil has joined #openstack-infra | 22:32 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: add dstat to our list of statistics collectors https://review.openstack.org/72786 | 22:32 |
| jeblair | fungi: centos6 bare slave config here: https://review.openstack.org/72784 | 22:33 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Allow bare-precise nodes to sudo https://review.openstack.org/72780 | 22:35 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revoke sudo permissions before infra jobs https://review.openstack.org/72778 | 22:35 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revoke sudo from most jobs https://review.openstack.org/72779 | 22:35 |
| clarkb | I bumped the heap size to 1G on 02 | 22:35 |
| jeblair | clarkb: most of the stuck workers didn't have that error though did they? | 22:36 |
| clarkb | jeblair: not sure yet | 22:36 |
| clarkb | jeblair: I will keep an eye on them and as they die check | 22:36 |
| jog0 | jeblair: https://review.openstack.org/#/c/68304/ I don't think it merged because it was a WIP :/ | 22:40 |
| jog0 | clarkb: ^ | 22:41 |
| jeblair | jog0: yep. can't merge a wip. | 22:41 |
| *** ryanpetrello has joined #openstack-infra | 22:42 | |
| jog0 | jeblair: my bad | 22:42 |
| jog0 | can you rekick it? | 22:42 |
| jeblair | jog0: done | 22:42 |
| *** dkranz has quit IRC | 22:43 | |
| clarkb | jeblair: fwiw I think fixing the parsing errors sped up processing by a large factor (as did adding all the possible workers) | 22:43 |
| clarkb | this has us mostly caught up as best I can tell | 22:43 |
| jog0 | jeblair: thanks | 22:44 |
| clarkb | which leaves us with the mysterious dying problem which seems to be related to the heap OOMing | 22:44 |
| jaypipes | jeblair: getting tantalizingly close now :) After adding known hosts, this is my output after Zuul successfully finds a new patch to sandbox repo, but Gearman is complaining about the job not being registered with it: http://paste.openstack.org/show/64418/ | 22:46 |
| jaypipes | jeblair: question: other than the telnet thing, is there any way to debug the Gearman Jenkins Job registration stuff? | 22:46 |
| clarkb | jhesketh has a zuul cli change that I thought was approved | 22:47 |
| *** hashar has quit IRC | 22:47 | |
| *** dims has joined #openstack-infra | 22:47 | |
| clarkb | jaypipes: if it isn't merged yet it should be soonish and it uses prettytable to list stuff a bit nicely | 22:47 |
| jeblair | clarkb: that's not going to help here | 22:47 |
| zaro | clarkb, fungi, jeblair : want to talk about gerrit upgrade now? | 22:47 |
| jeblair | jaypipes: telnet will help you confirm whether the job is registered | 22:47 |
| jeblair | jaypipes: if it is registered, then there is a zuul bug | 22:48 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add best and worst status details for conditional-step plugin. https://review.openstack.org/69608 | 22:48 |
| jeblair | jaypipes: if it is not registered, then you'll need to debug your workers (eg, jenkins) | 22:48 |
| *** sandywalsh has quit IRC | 22:48 | |
| jaypipes | jeblair: nope, nothing registered... (I think?) | 22:48 |
| jaypipes | status | 22:48 |
| jaypipes | zuul:promote001 | 22:48 |
| jaypipes | zuul:enqueue001 | 22:48 |
| *** jhesketh__ has joined #openstack-infra | 22:48 | |
| jeblair | jaypipes: yep. so that's up to jenkins. | 22:49 |
| jeblair | i'm assuming you're using jenkins | 22:49 |
| jaypipes | jeblair: yup, and the job is created in jenkins properly (or at least, it seems to be) | 22:49 |
| jaypipes | this job: https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/jenkins_jobs/config/sandbox.yaml | 22:49 |
| jeblair | jaypipes: check that the gearman plugin is enabled; try restarting jenkins; check the jenkins debug log | 22:49 |
| jaypipes | kk | 22:49 |
| clarkb | jeblair: 08 has disconnected from the cluster without that heap error in the logs | 22:49 |
| jeblair | zaro: ok | 22:50 |
| clarkb | I am going to bump its heap to 1.5G just in case it died before it could write stuff and see if that helps | 22:50 |
| clarkb | but am also going to start looking at an upgrade now | 22:50 |
| jeblair | clarkb: should we run fewer pushers? | 22:50 |
| jeblair | clarkb: per host | 22:50 |
| jeblair | (instead of 4) | 22:50 |
| clarkb | jeblair: oh good idea, that would cut the size of queues down which may help memory consumption | 22:51 |
| clarkb | jeblair: yeah, why don't I try setting it to 2 per | 22:51 |
| jeblair | clarkb: k | 22:51 |
| clarkb | and I will leave the heap values as is for the indexer processes | 22:51 |
| clarkb | jeblair: I guess just stop C and D by hand? | 22:52 |
| jeblair | yep | 22:52 |
| zaro | jeblair: so i think everything is ready to go. i believe all changes needs are in gerrit, etherpad has a list of the need changes. No sure what else there is to do. would like to proceed if possible. | 22:52 |
| *** mwagner_lap has quit IRC | 22:52 | |
| jeblair | zaro: have the etherpad link handy? | 22:52 |
| *** dizquierdo has joined #openstack-infra | 22:53 | |
| *** CaptTofu has quit IRC | 22:53 | |
| zaro | https://etherpad.openstack.org/p/gerrit-2.8-upgrade and | 22:54 |
| zaro | https://etherpad.openstack.org/p/gerrit_upgrade_script | 22:54 |
| *** jgrimm has quit IRC | 22:54 | |
| jeblair | zaro: okay so the first review i see linked from there is abandoned | 22:56 |
| jeblair | https://review.openstack.org/#/c/64978/1 | 22:56 |
| jhesketh__ | Hey | 22:57 |
| jeblair | jhesketh__: hi there | 22:57 |
| *** gokrokve has quit IRC | 22:58 | |
| *** dstanek has quit IRC | 22:58 | |
| *** pblaho has quit IRC | 22:59 | |
| zaro | jeblair: that's a _david_ which we will need. so an infra-core should restore. | 23:00 |
| *** gokrokve has joined #openstack-infra | 23:00 | |
| jeblair | zaro: how does that relate to https://review.openstack.org/#/c/69800 ? | 23:00 |
| *** sarob has joined #openstack-infra | 23:02 | |
| zaro | jeblair: opps. i must have forgotten about the on from _david_, so those are dups | 23:02 |
| *** gokrokve has quit IRC | 23:02 | |
| *** gokrokve has joined #openstack-infra | 23:02 | |
| *** banix has quit IRC | 23:03 | |
| jeblair | zaro: it says 65162 is replaced by 70014; should we remove 65162 from your list? | 23:03 |
| jeblair | zaro: which one should we review, yours or david's? | 23:03 |
| sdague | so this has a paired devstack change right now - https://review.openstack.org/#/c/72786/ | 23:03 |
| *** gokrokve has quit IRC | 23:03 | |
| sdague | but basically, pidstat turns out to not be very useful, so I'd like to get dstat into our collector list | 23:04 |
| sdague | which might be able to replace sysstat and pidstat | 23:04 |
| jeblair | zaro: is 2.8.1 released? | 23:04 |
| zaro | jeblair: yes, i forgot to update that. replace 65162 with 70014 | 23:04 |
| sdague | also, dstat supports plugins written in python ... which would mean we could more easily extend it for custom stats | 23:04 |
| jeblair | sdague: we're not doing anything automated with that atm, so i don't have a preference | 23:05 |
| zaro | i'm probably more active on this now, you probably better to review mine? | 23:05 |
| sdague | jeblair: cool, so my approach right now is run them all, then probably turn off sysstat and pidstat later | 23:06 |
| sdague | but it will need the d-g change above | 23:06 |
| *** sarob has quit IRC | 23:07 | |
| *** pcrews has quit IRC | 23:07 | |
| jeblair | zaro: so that leaves your comment about 2.8.1; what do you mean by that? don't we already have a custom 2.8 with the change_owner patch? | 23:08 |
| *** dstanek has joined #openstack-infra | 23:09 | |
| *** dstanek has quit IRC | 23:09 | |
| *** harlowja has joined #openstack-infra | 23:09 | |
| nibalizer | haha sorry to whomever runs the puppetmaster, I've accidentally sent you a few cert requests when debugging locally | 23:10 |
| *** dane_ has quit IRC | 23:10 | |
| zaro | jeblair: my patches for change_owner was not merged to our custom 2.8 branch yet. they were abandoned. | 23:11 |
| *** sarob has joined #openstack-infra | 23:12 | |
| jeblair | zaro: what war is -dev running? | 23:12 |
| *** masayukig has joined #openstack-infra | 23:12 | |
| zaro | jeblair: it's running 2.8 | 23:12 |
| jeblair | zaro: how are you testing the WIP stuff? | 23:12 |
| zaro | jeblair: i hand patched and built it on review-dev | 23:13 |
| jeblair | zaro: okay, so it seems like we have several major steps: | 23:14 |
| zaro | jeblair: or maybe built on my machine then deployed on review-dev | 23:14 |
| jeblair | a) finish the build infrastructure | 23:14 |
| jeblair | b) merge needed local patches for 2.8 | 23:14 |
| jeblair | c) have puppet deploy a jenkins-built openstack/2.8 war on review-dev | 23:14 |
| jeblair | d) merge any remaining config changes that can be safely applied before the upgrade | 23:14 |
| jeblair | e) perform the upgrade | 23:14 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: restrict all queries to the relevant file https://review.openstack.org/72793 | 23:15 |
| jeblair | f) merge config changes that can only be applied after the upgrade | 23:15 |
| jeblair | [eol] | 23:15 |
| sdague | clarkb: ^^^ | 23:15 |
| sdague | I'm hoping that helps on the ES side a little | 23:15 |
| *** senthil has quit IRC | 23:15 | |
| zaro | jeblair: i believe a) is done just need to get those changes restored. | 23:15 |
| clarkb | sdague: thanks, one small nit then we can push that through | 23:16 |
| *** sarob has quit IRC | 23:16 | |
| jeblair | zaro: right; i'd consider that done once they are merged | 23:16 |
| zaro | jeblair: i though infra-core was waiting on go ahead to do step b. | 23:16 |
| *** masayukig has quit IRC | 23:16 | |
| *** cadenzajon has quit IRC | 23:17 | |
| zaro | jeblair: ok, i see we want to merge now so once i get those changes in good change i'll bug you guys again. | 23:17 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: restrict all queries to the relevant file https://review.openstack.org/72793 | 23:18 |
| *** esker has quit IRC | 23:18 | |
| jeblair | zaro: it's in a branch so it should be safe to merge; i think we want to have tho whole build infra in place before we upgrade so that we can maintain it | 23:18 |
| sdague | clarkb: that's what I get for bulk processing these all by eye | 23:18 |
| jeblair | zaro: (i don't want to start running 2.8, find we need to patch something, and then find out we don't have a build system) | 23:18 |
| sdague | and, holy crap, do we have variance | 23:18 |
| clarkb | sdague: :( | 23:18 |
| sdague | I need to write a hacking bit for these so that we are consistent in how they are structured | 23:18 |
| sdague | message: first | 23:19 |
| sdague | break on AND | 23:19 |
| sdague | pick if it's at the beginning or end | 23:19 |
| zaro | jeblair: ok. thanks. | 23:19 |
| anteaya | nibalizer: are you saying that the 25000 lines of code in that patch comprises modules from puppetlabs? | 23:20 |
| *** dcramer_ has quit IRC | 23:20 | |
| anteaya | nibalizer: or that you need 25000 lines of code to bring in modules from puppetlabs? | 23:20 |
| nibalizer | oh | 23:21 |
| nibalizer | no that patch should be abandoned | 23:21 |
| nibalizer | i can push a new version of what im working on in just a few minutes | 23:22 |
| anteaya | the 25000 lines of code patch? | 23:22 |
| fungi | anteaya: those were the source code for modules from the puppetforge. i pointed him at our install_modules.sh script where they can just be listed and sourced from the forge directly | 23:22 |
| nibalizer | ya | 23:22 |
| anteaya | ah, I missed that, thank you | 23:22 |
| * anteaya breaths a sigh of relief | 23:22 | |
| *** krtaylor has quit IRC | 23:22 | |
| *** pcrews has joined #openstack-infra | 23:23 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add devstack-gate unit test job https://review.openstack.org/64582 | 23:23 |
| nibalizer | anteaya: im still warming up to the codebase | 23:24 |
| nibalizer | win 5 | 23:25 |
| anteaya | nibalizer: I understand | 23:25 |
| anteaya | glad you were open to the redirection | 23:26 |
| *** zul has quit IRC | 23:26 | |
| harlowja | clarkb that mysql user permission change went in right, should be active (so that tests can use non-conflicts dbs) | 23:27 |
| clarkb | harlowja: did it? | 23:27 |
| clarkb | looks like it did | 23:27 |
| harlowja | seems so | 23:27 |
| clarkb | harlowja: so yeah about 30 minutes after when it merged you should be good on the static slaves | 23:27 |
| clarkb | then whenever image builds happen this evening you should be good on the single use slaves | 23:27 |
| harlowja | hmmm, interesting, k | 23:27 |
| harlowja | clarkb i was trying it out, seems like the tests i have aren't detecting mysql running, wonder if someting went bad? | 23:28 |
| harlowja | the unit tests are starting to skip mysql , so wondering if something went wrong | 23:28 |
| fungi | harlowja: what's the job name? | 23:28 |
| harlowja | http://logs.openstack.org/72/72572/2/check/gate-taskflow-python27 | 23:28 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Remove push refs to gerrit feature https://review.openstack.org/68723 | 23:28 |
| harlowja | u can search for skip, and u'll see that its skipping past the mysql tests, which happens when mysql isn't found | 23:28 |
| harlowja | so wondering if something went boo-boo | 23:29 |
| harlowja | *or had a boo-boo | 23:29 |
| harlowja | lol | 23:29 |
| fungi | 23:20:02.928 | Building remotely on precise10 | 23:29 |
| fungi | i'll check that slave out | 23:29 |
| *** ryanpetrello has quit IRC | 23:29 | |
| harlowja | k, np | 23:29 |
| harlowja | or i broke something (entirely possible) | 23:29 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Remove deprecated logstash max_inflight_requests https://review.openstack.org/72794 | 23:29 |
| harlowja | but i was trying to setup and use a alternate db per test, to avoid the race issues | 23:29 |
| fungi | harlowja: mysqld is running and listening on 127.0.0.1:3306 | 23:30 |
| harlowja | hmmm | 23:30 |
| harlowja | k | 23:30 |
| fungi | i'll make sure puppet agent isn't stuck | 23:30 |
| harlowja | same user and stuff right | 23:30 |
| harlowja | 'openstack_citest' | 23:30 |
| *** whoops has quit IRC | 23:31 | |
| *** mfer has joined #openstack-infra | 23:31 | |
| *** dhellmann is now known as dhellmann_ | 23:32 | |
| harlowja | let me repoke, see if i can cause it to fail with a more useful message, since the skipTest messages aren't shown in the webpage results (Afaik) | 23:32 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Bringing up a PuppetDB node https://review.openstack.org/72795 | 23:33 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Remove hardcoded config assumptions, cleanup variables https://review.openstack.org/66072 | 23:33 |
| clarkb | harlowja: yes same user and password, but you should have the ability to add and drop different DBs | 23:33 |
| harlowja | right | 23:33 |
| nibalizer | anteaya: ^ | 23:33 |
| nibalizer | i think we still need to have our own copy of the puppetdb codes | 23:34 |
| nibalizer | since I have to reach in there and rip stuff out or wrap it in if blocks | 23:34 |
| nibalizer | and im open to removing the 'extra' crap in the module if you want to do it that way | 23:34 |
| *** ryanpetrello has joined #openstack-infra | 23:35 | |
| fungi | clarkb: harlowja: user openstack_citest has drop privs on db openstack_citest according to my queries against mysql.db | 23:36 |
| *** mfer has quit IRC | 23:36 | |
| *** markwash has quit IRC | 23:36 | |
| anteaya | nibalizer: 66072 is a nice size | 23:36 |
| clarkb | fungi: it should have them globally | 23:37 |
| clarkb | fungi: create as well | 23:37 |
| * fungi checks | 23:37 | |
| *** thuc has quit IRC | 23:37 | |
| nibalizer | anteaya: thats not actually mine, its the one im depending on | 23:37 |
| *** mrda is now known as mrda_away | 23:37 | |
| nibalizer | i understand the need to make these small, but the puppetdb module is a big fat blob that we need to stick in there, somehow | 23:37 |
| *** thedodd has quit IRC | 23:37 | |
| *** thuc has joined #openstack-infra | 23:37 | |
| nibalizer | im happy to split off ALL logic/code changes into a separte commit from the one that brings in the puppetdb module, would that work? | 23:37 |
| anteaya | nibalizer: yes, I see AaronGr is the original author of 66072 | 23:38 |
| fungi | clarkb: harlowja: openstack_citest is N in mysql.user.drop_priv | 23:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Add devstack-gate unit test job https://review.openstack.org/64582 | 23:38 |
| anteaya | nibalizer: can we try it? | 23:38 |
| fungi | clarkb: harlowja: maybe puppet agent got stuck. i'll see if it's behind | 23:38 |
| anteaya | nibalizer: it really helps the reviewers review went changes are nice and small | 23:39 |
| anteaya | s/went/when | 23:39 |
| harlowja | fungi 'N' as in NO? | 23:39 |
| harlowja | ;) | 23:39 |
| * fungi nos | 23:39 | |
| fungi | nods | 23:39 |
| harlowja | k :) | 23:39 |
| fungi | notice: /Stage[main]/Jenkins::Slave/Database_grant[openstack_citest@localhost]/privileges: current_value , should be create, drop (noop) | 23:40 |
| fungi | looks like it's not updated for some reason? | 23:40 |
| harlowja | hmmm | 23:40 |
| clarkb | :( | 23:40 |
| *** julim has quit IRC | 23:40 | |
| fungi | funny thing is, there was no hung puppet agent run. cron has been terminating normally i think | 23:41 |
| fungi | aha! | 23:41 |
| fungi | err: /Stage[main]/Jenkins::Slave/Database_grant[openstack_citest@localhost]/privileges: change from to create, drop failed: The privileges 'create,drop' are not valid. Did you mean 'create_priv,drop_priv'? | 23:41 |
| fungi | typo! ;) | 23:41 |
| nibalizer | anteaya: what do you want to try? splitting the puppetdb module import into its own change? | 23:41 |
| *** denis_makogon has quit IRC | 23:41 | |
| *** thuc has quit IRC | 23:42 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove deprecated logstash max_inflight_requests https://review.openstack.org/72794 | 23:42 |
| jeblair | win 22 | 23:42 |
| fungi | always a winner | 23:42 |
| *** oubiwann_ has quit IRC | 23:42 | |
| *** fbo is now known as fbo_away | 23:43 | |
| anteaya | nibalizer: sure, let's start there | 23:43 |
| *** prad has quit IRC | 23:43 | |
| *** dizquierdo has quit IRC | 23:44 | |
| nibalizer | anteaya: im also open to any ideas you have on how to bring it in in a way thats good | 23:45 |
| nibalizer | logically, adding one line 'puppetlabs-inifile' to install_modules.sh and commiting 2500 lines of puppetdb module to git are doing the same thing | 23:46 |
| harlowja | intersting, fungi, should the following happen, 'OperationalError: (OperationalError) source database "template1" is being accessed by other users' | 23:46 |
| lifeless | harlowja: postgresql ? | 23:46 |
| harlowja | in my tests without lock for postgres, connecting to template1 (maybe i shouldn't connect to that) | 23:46 |
| harlowja | lifeless ya | 23:46 |
| lifeless | harlowja: there's an implicit lock on that when you make a new db | 23:46 |
| harlowja | hmmm | 23:47 |
| lifeless | harlowja: can't do two create db's concurrently. | 23:47 |
| harlowja | ah, so still need some sort of process lock | 23:47 |
| harlowja | durn | 23:47 |
| lifeless | I can dig up the workaround code from LP's parallel testsuite if you like | 23:47 |
| harlowja | sure | 23:47 |
| * harlowja was hoping to remove any process/file locks | 23:47 | |
| lifeless | abandon all hope ye who enter here | 23:48 |
| harlowja | def | 23:48 |
| clarkb | wait really? that is :( | 23:49 |
| *** ryanpetrello has quit IRC | 23:49 | |
| anteaya | nibalizer: well we don't really need the puppetdb module to be in git, since it is already under the control of puppet labs | 23:49 |
| anteaya | we don't keep dependencies in our repos | 23:49 |
| nibalizer | anteaya: im using it stock right now and I can advise that we'll need to change it | 23:49 |
| anteaya | we name them, and import them | 23:50 |
| *** thomasem has joined #openstack-infra | 23:50 | |
| anteaya | nibalizer: sounds good, who will you be advising | 23:50 |
| anteaya | and who is we? | 23:50 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct MySQL DB priv names for citest grants https://review.openstack.org/72799 | 23:50 |
| fungi | clarkb: harlowja: ^ | 23:50 |
| anteaya | do you work with puppetlabs? | 23:50 |
| nibalizer | idunno im stariting to think of myself as being on the openstack-puppet team | 23:50 |
| anteaya | good | 23:50 |
| lifeless | harlowja: one bit is here - http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/testing/pgsql.py#L288 | 23:50 |
| anteaya | that works for me | 23:51 |
| lifeless | harlowja: ok yeah, I've paged it in | 23:51 |
| lifeless | harlowja: what we did was this: | 23:51 |
| harlowja | lifeless durn, a spinlock | 23:51 |
| nibalizer | i dont work for pupeptlabs, but im in the puppet community | 23:51 |
| anteaya | nibalizer: cool | 23:51 |
| anteaya | glad to have your help | 23:51 |
| lifeless | - on parallel test setup, we single threaded create a template db with the right schema | 23:51 |
| *** gothicmindfood has quit IRC | 23:51 | |
| nibalizer | let me do some more testing, but im pretty confident that the puppetdb module from pupetlabs cant integrate into openstack-infra/config stock | 23:51 |
| lifeless | - then each worker has a unique template, so when tests need a new DB there is no cross-thread contention | 23:51 |
| harlowja | lifeless does testr have any equivalent of that? | 23:52 |
| lifeless | harlowja: you need a spin lock because other folk (e.g. you :P) can cause the contention, so just assuming that a test specific lock would work is insufficiently paranoid | 23:52 |
| *** gothicmindfood has joined #openstack-infra | 23:52 | |
| lifeless | harlowja: the way I'd do it today is to use van.pg and bring up a pg cluster per thread, totally isolated. | 23:52 |
| jeblair | nibalizer: would you consider patching upstream puppetdb to fix the deficiencies? | 23:53 |
| harlowja | lifeless ya, i could imagine it being possible if there was a way for testr to call a routine before it starts doing all work, something like onRun(thread-am) | 23:53 |
| harlowja | *process-am | 23:53 |
| harlowja | that i could then setup these dbs, before it splits into child processes | 23:54 |
| clarkb | initial logstash 1.3.3 testing is promising. I have a very minimal diff between the conf file I am using for it and the existing conf file | 23:54 |
| jeblair | clarkb: ++ | 23:54 |
| harlowja | fungi thx for the patch, will wait on that before trying this undo of locks again, although seems like i might still need one around the postgres part | 23:54 |
| *** mriedem has quit IRC | 23:55 | |
| lifeless | harlowja: huh, easiest way is just a global fixture object in the test suite | 23:55 |
| lifeless | harlowja: remember that testr backends are distinct processes, so globals are not shared | 23:55 |
| lifeless | harlowja: So I'm saying set it up in each child process | 23:55 |
| clarkb | jeblair: fungi: if you guys don't oppose this plan I would like to grab 16 and manually do the upgrade (have it use the elasticsearch_http output for es compat) and see how it does running in production | 23:56 |
| lifeless | harlowja: in LP the schema create time was ridiculously slow | 23:56 |
| clarkb | * how it does running logstash 1.3.3 | 23:56 |
| *** dkliban is now known as dkliban_afk | 23:56 | |
| lifeless | harlowja: so we needed that just-once optimisation | 23:56 |
| harlowja | lifeless sure, but each child process tries to run a postgres create database XYZ, and then this conflict happens | 23:56 |
| lifeless | harlowja: no. See van.pg which I referenced above. | 23:56 |
| jeblair | clarkb: go for it | 23:57 |
| nibalizer | jeblair: absolutely, thats deff the right approach, but as long as puppetdb module wants to use puppetlabs-firewall and opesntack-infra/config uses kinda home-grown firewall/iptables mangment its gonna be hard to have them coexist | 23:57 |
| nibalizer | the patches are essentially flags to turn off behavior | 23:57 |
| harlowja | lifeless k, let me checkout van.pg | 23:57 |
| *** weshay has quit IRC | 23:58 | |
| fungi | clarkb: i think that sounds capital | 23:59 |
| lifeless | harlowja: I can put a sketch of a patch together fpr you if you like | 23:59 |
| *** chris_johnson is now known as wchrisj|away | 23:59 | |
| harlowja | lifeless if u want, no rush i think | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!