| *** jnoller has quit IRC | 00:00 | |
| *** gokrokve has quit IRC | 00:07 | |
| lifeless | fungi: that seems excessive | 00:22 |
|---|---|---|
| lifeless | fungi: login, have a poke | 00:22 |
| *** dcramer_ has joined #openstack-infra | 00:23 | |
| lifeless | fungi: checking that the MTU was correctly reduced would be useful I suspect | 00:24 |
| lifeless | fungi: as that can cause -massive- slowdowns | 00:24 |
| *** ok_delta__ has joined #openstack-infra | 00:27 | |
| *** xchu has joined #openstack-infra | 00:36 | |
| *** ok_delta has quit IRC | 00:37 | |
| *** gokrokve has joined #openstack-infra | 00:40 | |
| *** jcooley_ has joined #openstack-infra | 00:53 | |
| sdague | jnoller_, jeblair: http://ubuntuone.com/4WAxy4VBGTCAuIktKrSxXr and http://ubuntuone.com/1iYufRBFGYNQ4TAiStICx1 - current state of things. Some very slow rax nodes in there. | 00:54 |
| *** gokrokve has quit IRC | 00:55 | |
| *** david-lyle has joined #openstack-infra | 00:56 | |
| *** ociuhandu has quit IRC | 01:06 | |
| *** thuc has joined #openstack-infra | 01:07 | |
| *** thuc_ has joined #openstack-infra | 01:08 | |
| *** thuc has quit IRC | 01:11 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Update Storyboard https://review.openstack.org/69150 | 01:14 |
| *** chuck__ has quit IRC | 01:28 | |
| *** senk1 has quit IRC | 01:30 | |
| *** pcrews has joined #openstack-infra | 01:30 | |
| *** mcrodermang is now known as mriedem | 01:32 | |
| *** senk has joined #openstack-infra | 01:33 | |
| *** senk1 has joined #openstack-infra | 01:37 | |
| *** senk has quit IRC | 01:38 | |
| *** dkliban has joined #openstack-infra | 01:46 | |
| *** senk1 has quit IRC | 02:02 | |
| *** thuc has joined #openstack-infra | 02:03 | |
| *** thuc_ has quit IRC | 02:06 | |
| jog0 | so the e-r bot isn't commenting on gerrit anymore | 02:07 |
| *** thuc has quit IRC | 02:08 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make functional tests work again https://review.openstack.org/69154 | 02:10 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make functional tests work again https://review.openstack.org/69154 | 02:11 |
| *** gokrokve has joined #openstack-infra | 02:17 | |
| *** gokrokve has quit IRC | 02:21 | |
| jog0 | woops found the bug | 02:22 |
| openstackgerrit | lifeless proposed a change to openstack-infra/devstack-gate: Capture disk space available at the start of a run https://review.openstack.org/69156 | 02:22 |
| openstackgerrit | lifeless proposed a change to openstack-infra/devstack-gate: Pretty up the post-run df call https://review.openstack.org/69157 | 02:22 |
| *** persia__ has quit IRC | 02:23 | |
| *** persia__ has joined #openstack-infra | 02:23 | |
| *** persia__ is now known as persia | 02:23 | |
| *** gokrokve_ has joined #openstack-infra | 02:24 | |
| *** yaguang has joined #openstack-infra | 02:24 | |
| *** gokrokve_ has quit IRC | 02:28 | |
| *** morganfainberg|z is now known as morganfainberg | 02:33 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Fix gerrit leave comment capability https://review.openstack.org/69158 | 02:34 |
| *** dstufft has quit IRC | 02:37 | |
| *** dstufft has joined #openstack-infra | 02:37 | |
| *** pcrews has quit IRC | 02:40 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Keep logs in $BASE instead of $WORKSPACE https://review.openstack.org/68782 | 02:44 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Use /opt/git instead of workspace cache https://review.openstack.org/68990 | 02:44 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Fix gerrit leave comment capability https://review.openstack.org/69158 | 02:49 |
| *** wenlock has joined #openstack-infra | 03:00 | |
| *** pcrews has joined #openstack-infra | 03:02 | |
| *** CaptTofu has joined #openstack-infra | 03:03 | |
| *** chandankumar_ has joined #openstack-infra | 03:05 | |
| *** pcrews has quit IRC | 03:08 | |
| *** morganfainberg is now known as morganfainberg|z | 03:08 | |
| *** chandankumar_ has quit IRC | 03:10 | |
| *** AlexF_ has joined #openstack-infra | 03:22 | |
| *** jasondotstar has joined #openstack-infra | 03:24 | |
| *** AlexF_ has quit IRC | 03:28 | |
| *** dims has joined #openstack-infra | 03:38 | |
| *** UtahDave has joined #openstack-infra | 03:46 | |
| *** david-lyle has quit IRC | 04:00 | |
| *** david-lyle has joined #openstack-infra | 04:02 | |
| *** mriedem has quit IRC | 04:03 | |
| *** jhesketh_ has quit IRC | 04:05 | |
| *** jasondotstar has quit IRC | 04:07 | |
| *** dkliban has quit IRC | 04:12 | |
| *** praneshp has quit IRC | 04:15 | |
| *** praneshp has joined #openstack-infra | 04:16 | |
| *** UtahDave has quit IRC | 04:19 | |
| *** boris-42 has joined #openstack-infra | 04:23 | |
| *** weshay has quit IRC | 04:24 | |
| *** ok_delta__ has quit IRC | 04:27 | |
| *** UtahDave has joined #openstack-infra | 04:37 | |
| *** morganfainberg|z is now known as morganfainberg | 04:39 | |
| *** yamahata_ has quit IRC | 04:42 | |
| *** david-lyle_ has quit IRC | 04:42 | |
| *** miqui has quit IRC | 04:43 | |
| *** yamahata_ has joined #openstack-infra | 04:43 | |
| *** miqui has joined #openstack-infra | 04:43 | |
| *** david-lyle_ has joined #openstack-infra | 04:43 | |
| *** UtahDave has quit IRC | 04:51 | |
| *** medberry has joined #openstack-infra | 04:57 | |
| *** medberry has joined #openstack-infra | 04:57 | |
| *** medberry is now known as Guest57266 | 04:57 | |
| *** UtahDave has joined #openstack-infra | 04:59 | |
| *** jcooley_ has quit IRC | 05:21 | |
| *** markwash has joined #openstack-infra | 05:22 | |
| *** CaptTofu has quit IRC | 05:27 | |
| *** CaptTofu has joined #openstack-infra | 05:27 | |
| *** CaptTofu has quit IRC | 05:32 | |
| *** matsuhashi has joined #openstack-infra | 05:32 | |
| *** yanghe has joined #openstack-infra | 05:33 | |
| *** coolsvap has joined #openstack-infra | 05:38 | |
| *** morganfainberg is now known as morganfainberg|z | 05:39 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files https://review.openstack.org/48783 | 05:46 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Support lazy resolving of include yaml tags https://review.openstack.org/63580 | 05:46 |
| *** matsuhashi has quit IRC | 05:48 | |
| *** matsuhashi has joined #openstack-infra | 05:48 | |
| *** matsuhashi has quit IRC | 05:52 | |
| *** matsuhashi has joined #openstack-infra | 05:52 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Support recursive processing of paths https://review.openstack.org/68895 | 05:55 |
| *** wenlock has quit IRC | 06:04 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files https://review.openstack.org/48783 | 06:04 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Add tests for YamlParser and patch 2.6 minidom https://review.openstack.org/63579 | 06:04 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Support lazy resolving of include yaml tags https://review.openstack.org/63580 | 06:04 |
| *** matsuhashi has quit IRC | 06:06 | |
| *** matsuhashi has joined #openstack-infra | 06:10 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 06:15 | |
| *** matsuhashi has quit IRC | 06:21 | |
| *** markwash has quit IRC | 06:22 | |
| *** markwash has joined #openstack-infra | 06:22 | |
| *** matsuhas_ has joined #openstack-infra | 06:23 | |
| *** praneshp has left #openstack-infra | 06:25 | |
| *** jcooley_ has joined #openstack-infra | 06:29 | |
| *** crank has quit IRC | 06:34 | |
| *** matsuhas_ has quit IRC | 06:38 | |
| *** markwash has quit IRC | 06:38 | |
| *** matsuhashi has joined #openstack-infra | 06:39 | |
| *** crank has joined #openstack-infra | 06:41 | |
| *** jcooley_ has quit IRC | 06:45 | |
| jnoller_ | sdague: this is after the pvhvm images? | 06:48 |
| jnoller_ | sdague: if so, let me know - I need to go chase some people down and figure out how to replicate the numbers | 06:49 |
| *** jnoller_ has quit IRC | 06:49 | |
| *** DinaBelova_ is now known as DinaBelova | 07:13 | |
| *** CaptTofu has joined #openstack-infra | 07:28 | |
| *** CaptTofu has quit IRC | 07:33 | |
| *** rlandy has joined #openstack-infra | 07:36 | |
| *** Ryan_Lane has joined #openstack-infra | 07:48 | |
| *** lttrl has joined #openstack-infra | 07:50 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 07:55 | |
| *** Ryan_Lane has quit IRC | 07:56 | |
| *** lttrl has quit IRC | 08:01 | |
| *** denis_makogon_ has joined #openstack-infra | 08:03 | |
| *** denis_makogon_ has quit IRC | 08:20 | |
| *** yolanda_ has joined #openstack-infra | 08:27 | |
| *** DinaBelova is now known as DinaBelova_ | 08:36 | |
| *** DinaBelova_ is now known as DinaBelova | 08:47 | |
| *** DinaBelova is now known as DinaBelova_ | 08:57 | |
| *** DinaBelova_ is now known as DinaBelova | 09:00 | |
| *** matsuhashi has quit IRC | 09:03 | |
| *** flaper87|afk is now known as flaper87 | 09:05 | |
| *** matsuhashi has joined #openstack-infra | 09:09 | |
| *** DinaBelova is now known as DinaBelova_ | 09:16 | |
| *** CaptTofu has joined #openstack-infra | 09:29 | |
| *** yaguang has quit IRC | 09:34 | |
| *** CaptTofu has quit IRC | 09:34 | |
| *** DinaBelova_ is now known as DinaBelova | 09:54 | |
| *** yanghe has left #openstack-infra | 10:03 | |
| *** DinaBelova is now known as DinaBelova_ | 10:24 | |
| *** UtahDave has quit IRC | 10:27 | |
| *** UtahDave has joined #openstack-infra | 10:35 | |
| *** coolsvap has quit IRC | 10:47 | |
| *** max_lobur has joined #openstack-infra | 11:10 | |
| *** xchu has quit IRC | 11:16 | |
| *** matsuhashi has quit IRC | 11:17 | |
| *** max_lobur has left #openstack-infra | 11:20 | |
| *** max_lobur has joined #openstack-infra | 11:20 | |
| *** max_lobur has left #openstack-infra | 11:20 | |
| *** wenlock has joined #openstack-infra | 11:23 | |
| *** wenlock has quit IRC | 11:27 | |
| *** CaptTofu has joined #openstack-infra | 11:30 | |
| *** CaptTofu has quit IRC | 11:34 | |
| *** matsuhashi has joined #openstack-infra | 11:40 | |
| sdague | jnoller_ no that's before | 11:49 |
| *** matsuhashi has quit IRC | 11:53 | |
| *** matsuhashi has joined #openstack-infra | 11:54 | |
| *** matsuhashi has quit IRC | 11:55 | |
| *** matsuhas_ has joined #openstack-infra | 11:55 | |
| *** thuc has joined #openstack-infra | 12:05 | |
| *** thuc has quit IRC | 12:14 | |
| *** dkliban has joined #openstack-infra | 12:32 | |
| *** vogxn has joined #openstack-infra | 12:33 | |
| *** matsuhas_ has quit IRC | 12:33 | |
| *** vogxn has quit IRC | 12:34 | |
| *** matsuhashi has joined #openstack-infra | 12:34 | |
| *** CaptTofu has joined #openstack-infra | 12:34 | |
| *** matsuhashi has quit IRC | 13:05 | |
| *** matsuhashi has joined #openstack-infra | 13:06 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Add gates for API projects and operations-guide https://review.openstack.org/67394 | 13:07 |
| *** dims has quit IRC | 13:07 | |
| *** dkliban has quit IRC | 13:16 | |
| *** matsuhashi has quit IRC | 13:18 | |
| *** CaptTofu has quit IRC | 13:26 | |
| *** CaptTofu has joined #openstack-infra | 13:27 | |
| *** CaptTofu has quit IRC | 13:32 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Publish database-api https://review.openstack.org/69194 | 13:33 |
| *** yolanda_ has quit IRC | 13:37 | |
| *** chuck__ has joined #openstack-infra | 13:40 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Fix check_projects_yaml_alphabetized.sh https://review.openstack.org/69195 | 13:46 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Fix check_projects_yaml_alphabetized.sh https://review.openstack.org/69195 | 13:52 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Fix check_projects_yaml_alphabetized.sh https://review.openstack.org/69195 | 14:00 |
| *** ociuhandu has joined #openstack-infra | 14:01 | |
| *** Ajaeger has joined #openstack-infra | 14:02 | |
| Ajaeger | fungi, mordred: gating in openstack-infra/config is broken due to check-alphabetized. Please review my fix: https://review.openstack.org/#/c/69195/ | 14:04 |
| *** gokrokve has joined #openstack-infra | 14:12 | |
| *** Ajaeger has quit IRC | 14:12 | |
| *** ociuhandu has quit IRC | 14:13 | |
| *** markwash has joined #openstack-infra | 14:18 | |
| *** DinaBelova_ is now known as DinaBelova | 14:25 | |
| anteaya | a commiter submitting a patch to neutronclient is working with the belief that mox is out of date and that projects should use mock or mox3: https://review.openstack.org/#/c/67359/ | 14:28 |
| anteaya | I have been working with the understanding that mox is the direction, to move away from mock to support python3 compatibility | 14:28 |
| anteaya | anyone with greater knowledge in this area care to comment? | 14:29 |
| anteaya | Alex_Gaynor dhellmann ^^ | 14:29 |
| *** mriedem has joined #openstack-infra | 14:29 | |
| sdague | anteaya: it's actually the other way around | 14:40 |
| sdague | mock is in python3 | 14:40 |
| anteaya | I need to checkout a new brain | 14:50 |
| anteaya | this one is used up | 14:50 |
| anteaya | so we have been moving from mox to mock | 14:50 |
| anteaya | for python3 compatibility | 14:50 |
| anteaya | so this patch should go from mox to mock and not bother with mox3, is that accurate? | 14:51 |
| *** dcramer_ has quit IRC | 14:54 | |
| *** mattoliverau has quit IRC | 14:54 | |
| anteaya | well that is what I wrote in the comment, let's see what other people think | 15:04 |
| *** gokrokve has quit IRC | 15:07 | |
| *** markwash has quit IRC | 15:15 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Orginized test requirements https://review.openstack.org/69200 | 15:25 |
| *** bknudson has joined #openstack-infra | 15:25 | |
| *** DinaBelova is now known as DinaBelova_ | 15:29 | |
| mordred | I'm going to ninja approve Ajaeger's patch. | 15:33 |
| *** Ajaeger has joined #openstack-infra | 15:35 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix check_projects_yaml_alphabetized.sh https://review.openstack.org/69195 | 15:38 |
| *** dcramer_ has joined #openstack-infra | 15:44 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Update documentation https://review.openstack.org/69202 | 15:46 |
| *** DinaBelova_ is now known as DinaBelova | 15:47 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Update documentation https://review.openstack.org/69202 | 15:49 |
| *** _ruhe is now known as ruhe | 15:54 | |
| *** wenlock has joined #openstack-infra | 16:02 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Update Storyboard jobs https://review.openstack.org/69204 | 16:04 |
| Ajaeger | mordred: any idea why this one still fails - this seems to be the version of the check from before your change: http://logs.openstack.org/87/62187/5/check/check-projects-yaml-alphabetized/7a5eb95/console.html | 16:04 |
| Ajaeger | https://review.openstack.org/69204 - fails also (still old version of the alphabetized check) - how often are the Jenkins nodes updated? When should a recheck be done? I guess I wait an hour and recheck... | 16:08 |
| *** luqas has joined #openstack-infra | 16:09 | |
| *** dizquierdo has joined #openstack-infra | 16:12 | |
| *** pcrews has joined #openstack-infra | 16:15 | |
| *** mestery has quit IRC | 16:16 | |
| *** mestery has joined #openstack-infra | 16:16 | |
| *** luqas has quit IRC | 16:19 | |
| *** dizquierdo_ has joined #openstack-infra | 16:25 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 16:27 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 16:33 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Update Storyboard jobs https://review.openstack.org/69204 | 16:35 |
| *** dizquierdo has quit IRC | 16:35 | |
| *** pcrews has quit IRC | 16:35 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 16:36 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 16:37 | |
| ruhe | Ajaeger: https://review.openstack.org/69204 is mine. how should i re-trigger it, once the alphabetized check issue is resolved? "recheck no bug", or there is a specific bug already filed for this? | 16:38 |
| mordred | Ajaeger: yeah - puppet needs to run on the slaves, which is on a cron | 16:38 |
| clarkb | ruhe tests will run again when you push a new patchset | 16:39 |
| Ajaeger | ruhe: recheck no bug will do it - but if you push a new patchset with my changes, I hope it will be fine ;) | 16:40 |
| *** shardy_afk is now known as shardy | 16:40 | |
| ruhe | Ajaeger: clarkb: thank you | 16:41 |
| Ajaeger | ruhe: since you pushed again: wait for the puppet runs first and then "recheck no bug" | 16:43 |
| Ajaeger | clarkb, mordred: how often is puppet run? | 16:43 |
| clarkb | every ten minutes with git updating every 15 | 16:44 |
| Ajaeger | clarkb: in that case 69204 should have been using the updated slave script... | 16:46 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 16:46 | |
| Ajaeger | have to drop off now, bye | 16:46 |
| *** Ajaeger has quit IRC | 16:46 | |
| clarkb | 69204 appears to be a config change | 16:47 |
| mordred | clarkb: yeah. we had a bug in the check-projects-yaml-alphabetized job | 16:48 |
| clarkb | oh I see bug in script. I bet a slave hasnt been updating and now I remember we use single use slavesnow | 16:48 |
| mordred | the fix is landed now, but the slave scripts don't seem to be update everywhere | 16:48 |
| clarkb | image updates once a day | 16:48 |
| mordred | clarkb: hang on - we're running check-projects-yaml-alphabetized on devstack slaves? | 16:49 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Orginized test requirements https://review.openstack.org/69200 | 16:50 |
| clarkb | mordred: no on bare-precise slaves | 16:50 |
| clarkb | single use no root slaves | 16:50 |
| mordred | gotcha. well - that means I'm going to have to just fix the job for real - because the current design is a bit broken. one sec | 16:51 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make the alphabetized check self-gating https://review.openstack.org/69208 | 16:56 |
| mordred | clarkb: I believe that will fix the gate on the config repo | 16:56 |
| dstufft | mordred: heh, the Openstack Infra (Zuul etc) is being used as a basis for a proposal for Python.org | 16:56 |
| mordred | dstufft: yay! | 16:56 |
| mordred | dstufft: you know - it's possible we could work together on something like that :) | 16:57 |
| dstufft | mordred: I'm not the one doing the proposing, Nick Coghlan is. I believe he plans on talking about it at PyCon | 16:57 |
| dstufft | http://www.python.org/dev/peps/pep-0462/ is his draft so far | 16:57 |
| clarkb | he was at lca and listened to the indoctrination :) | 16:58 |
| *** nicedice_ has joined #openstack-infra | 16:59 | |
| *** yolanda_ has joined #openstack-infra | 17:00 | |
| *** AlexF_ has joined #openstack-infra | 17:02 | |
| *** AlexF_ has quit IRC | 17:02 | |
| mordred | dstufft: how do I respond to something in that? he's got a technical challege listed that I think a little additional info might help him sort out | 17:02 |
| *** nicedice has quit IRC | 17:03 | |
| mordred | specifically, zuul is actually not integrated with jenkins at all, but rather sends build signals over gearman using the python gear library - so it should be prettys simple to write a gearman plugin for buildbot (which is the actual work that needs done) | 17:03 |
| dstufft | mordred: ncoghlan@gmail.com | 17:03 |
| dstufft | you can CC me if you don't want to email him out of the blue | 17:04 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 17:07 | |
| jeblair | mordred: cc me as well; i will be responding too | 17:07 |
| mordred | jeblair: k. I'm only going to cover teh "you just need a gearman driver for buildbot" part | 17:09 |
| jeblair | ++ | 17:09 |
| dstufft | mordred: jeblair if any of y'all are going to PyCon, I believe he plans on trying to hash this out at the language summit to get some sort of agreement on what should be done, or at least discussing it there | 17:11 |
| mordred | dstufft: Ill be at the language summit | 17:11 |
| dstufft | https://mail.python.org/pipermail/python-committers/2014-January/002976.html there's the thread where he mentioned it, I think you have to have a commit bit to join that mailing list though | 17:11 |
| *** dizquierdo_ has quit IRC | 17:12 | |
| jeblair | dstufft: i wasn't planning on attending the language summit, but i can make time for it and would like to be available to help discussions. do i need an invite for that? | 17:12 |
| dstufft | jeblair: I'm not sure, I've never actually been at PyCon :| | 17:13 |
| mordred | jeblair: nick is actually usually the one who hands out invitations | 17:13 |
| jeblair | mordred: ok, cool | 17:13 |
| mordred | jeblair: so, let him know you'd be happy to show up and he'd probably be more than happy to give you a thing | 17:13 |
| *** thuc has joined #openstack-infra | 17:19 | |
| *** thuc_ has joined #openstack-infra | 17:20 | |
| *** yolanda_ has quit IRC | 17:22 | |
| mordred | jeblair: while you're around - could you give a quick eyeball to https://review.openstack.org/#/c/69208/ ? | 17:23 |
| *** thuc has quit IRC | 17:23 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Update documentation https://review.openstack.org/69211 | 17:24 |
| mordred | jeblair: gate is currently broken for openstack-infra/config changes, that should fix it - but I don't want to break more things by trying to fix things | 17:24 |
| *** Ajaeger has joined #openstack-infra | 17:24 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Added documentation for REST API layer. https://review.openstack.org/69212 | 17:24 |
| jeblair | mordred: how come that patch passed then? | 17:26 |
| mordred | jeblair: the slave it ran on is from before the breaking patch landed | 17:26 |
| Ajaeger | jeblair: It executes the installed version of the patch, not the one in the repo | 17:26 |
| mordred | jeblair: there are some slaves out there which have the new broken patch, and afaict, none of them yet have the final fixed version of the script installed in slave_scripts | 17:27 |
| Ajaeger | jeblair: it executes " /usr/local/jenkins/slave_scripts/check_projects_yaml_alphabetized.sh" - and that's distributed via puppet | 17:27 |
| Ajaeger | mordred: this patch https://review.openstack.org/#/c/67394/ passed with the new version, so some seem to have it | 17:28 |
| mordred | Ajaeger: excellent | 17:28 |
| mordred | Ajaeger: in general, I think "does the slave have teh right version of the script" is a terrible gate to play :) | 17:29 |
| mordred | and that one failed with an old version | 17:29 |
| Ajaeger | ruhe: for https://review.openstack.org/#/c/69204/: Best to rebase the patch now. It should have the new patch in it and then pass. Wait half an hour (?) to have Zuul updated | 17:30 |
| Ajaeger | mordred: yeah, it really surprised me that of the 5 triggers I did only two passed ;( | 17:30 |
| ruhe | Ajaeger: cool, thank you | 17:30 |
| Ajaeger | mordred: Just one thing: Since you changed the path: Will this now not fail for everybody that is not at current HEAD? | 17:31 |
| mordred | jeblair: I believe I need to force sumit the patch, or else we're just going to play good-slave-roulette | 17:31 |
| jeblair | mordred: wfm | 17:31 |
| mordred | Ajaeger: nope. we merge changes on top of other cahnges, so as soon as it lands, and as soon as the jjb trigger runs to update the jobs, it should just work everywhere | 17:31 |
| Ajaeger | mordred: So, if https://review.openstack.org/#/c/62187/ gets rechecked, it will fail since tools/... does not exist - and neeeds a rebase first? | 17:31 |
| Ajaeger | mordred, so if I recheck 62187 now, it will rebase first? | 17:32 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make the alphabetized check self-gating https://review.openstack.org/69208 | 17:33 |
| mordred | no, we have to land the patch first ... one sec | 17:33 |
| openstackgerrit | A change was merged to openstack-infra/config: Make the alphabetized check self-gating https://review.openstack.org/69208 | 17:33 |
| mordred | there. that's done. now we need to wait 20-30 minutes for the jenkins jobs to update everywhere. then everything shoudl Just Work | 17:33 |
| * mordred stepping away for a few | 17:33 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 17:39 | |
| *** ok_delta has joined #openstack-infra | 17:39 | |
| *** fallenpegasus has joined #openstack-infra | 17:39 | |
| *** dims has joined #openstack-infra | 17:43 | |
| *** yassine has joined #openstack-infra | 17:45 | |
| *** dims has quit IRC | 17:45 | |
| *** dims has joined #openstack-infra | 17:46 | |
| *** jasondotstar has joined #openstack-infra | 17:47 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Update documentation https://review.openstack.org/69211 | 17:57 |
| *** luqas has joined #openstack-infra | 17:58 | |
| *** thuc has joined #openstack-infra | 18:00 | |
| *** thuc_ has quit IRC | 18:02 | |
| *** thuc_ has joined #openstack-infra | 18:02 | |
| *** thuc has quit IRC | 18:05 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Update Storyboard jobs https://review.openstack.org/69204 | 18:07 |
| *** chuck__ has quit IRC | 18:09 | |
| *** AlexF_ has joined #openstack-infra | 18:16 | |
| *** crank has quit IRC | 18:18 | |
| *** crank has joined #openstack-infra | 18:24 | |
| *** ruhe is now known as _ruhe | 18:24 | |
| *** luqas has joined #openstack-infra | 18:32 | |
| *** jbryce has quit IRC | 18:34 | |
| *** sdague has quit IRC | 18:35 | |
| *** sdague has joined #openstack-infra | 18:36 | |
| *** jbryce has joined #openstack-infra | 18:36 | |
| *** praneshp has joined #openstack-infra | 18:44 | |
| *** AlexF_ has quit IRC | 18:46 | |
| *** Ajaeger has quit IRC | 18:49 | |
| *** afazekas has quit IRC | 19:01 | |
| *** dcramer_ has quit IRC | 19:11 | |
| *** sdake has quit IRC | 19:15 | |
| *** sarob has joined #openstack-infra | 19:25 | |
| *** sarob has quit IRC | 19:32 | |
| *** sarob has joined #openstack-infra | 19:32 | |
| *** sarob has quit IRC | 19:34 | |
| *** sarob has joined #openstack-infra | 19:34 | |
| *** DinaBelova is now known as DinaBelova_ | 19:41 | |
| *** thuc has joined #openstack-infra | 19:43 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 19:43 | |
| *** thuc_ has quit IRC | 19:46 | |
| *** alexpilotti has joined #openstack-infra | 19:46 | |
| *** thuc has quit IRC | 19:47 | |
| clarkb | the openstack integrated gate is empty right now | 19:50 |
| sdague | clarkb: yep | 19:51 |
| sdague | clarkb: you poking at things a little bit, or just checking in? | 19:52 |
| *** AlexF_ has joined #openstack-infra | 19:53 | |
| fungi | just spotted that myself | 19:54 |
| openstackgerrit | Brant Knudson proposed a change to openstack/requirements: Update oauthlib requirement to at least 0.6 https://review.openstack.org/67900 | 19:56 |
| sdague | I think we still need to change a couple things on the window algorithm based on what I was seeing yesterday | 19:56 |
| sdague | 1) it would be nice to increase the window on any job success, not just when it gets to the head | 19:57 |
| *** sarob has quit IRC | 19:57 | |
| sdague | we have this train problem now where a slow rax node will stack up 8 changes behind it, so effectively there will be 2 d-g nodes used by the gate | 19:57 |
| fungi | right now it's based on succeeding changes, not succeeding jobs | 19:57 |
| sdague | fungi: yeh, succeeding changes would be fine | 19:58 |
| sdague | that would have increased the window | 19:58 |
| sdague | I actually meant changes :) | 19:58 |
| sdague | but if you have A .... H | 19:58 |
| fungi | sdague: a succeeding change is one with no failing jobs, a failing change is one with at least one completed job which failed | 19:59 |
| sdague | and B, C, D, E come back success, we should extend the window | 19:59 |
| sdague | fungi: yep | 19:59 |
| *** sarob has joined #openstack-infra | 19:59 | |
| sdague | but we aren't processing that until merge, right? | 19:59 |
| fungi | "completed" being key here | 19:59 |
| fungi | well, that was the first part of the problem sdtatement | 19:59 |
| sdague | I was definitely not seeing behavior that would indicate that we were | 19:59 |
| sdague | right, hence, something that should change :) | 20:00 |
| fungi | changes with no completed jobs (or even no started jobs) are succeeding | 20:00 |
| sdague | so we are missing a state? | 20:00 |
| sdague | to understand succeed complete? | 20:00 |
| fungi | the second part of the problem is as you point out, right now the behavior is triggered when a change reports, which it can't do until it reaches the gate position | 20:01 |
| sdague | well we trigger an event on a fail | 20:01 |
| sdague | so maybe need to add it on a success | 20:02 |
| sdague | the other issue, I think we are treating merge conflicts as fails | 20:02 |
| fungi | so yes, i think we lack a discrete state for "all jobs completed successfully but waiting for the change ahead of it" | 20:02 |
| sdague | which we shouldn't be for the window | 20:02 |
| fungi | we don't trigger it on a faIL | 20:02 |
| fungi | we trigger it when a failing change reports | 20:02 |
| sdague | a merge conflict and the stuff behind it shouldn't trigger window shrink | 20:02 |
| sdague | fungi: so the window changes don't happen until the head is processed? | 20:03 |
| fungi | right now, correct | 20:03 |
| sdague | if you look at our test node usage you can see a very distinctive sawtooth that's now created with the current model | 20:03 |
| fungi | each change passes through the gate position and reports as it does so | 20:03 |
| sdague | fungi: right, so for the window we need to account for these things sooner than that, otherwise we self starve really easily | 20:04 |
| fungi | if we begin triggering window changes on not-yet-reported change states, we're going to grow it a bunch following gate resets | 20:04 |
| sdague | fungi: why? | 20:05 |
| * jog0 is shocked to see the gate is empty | 20:06 | |
| fungi | if we still have slow nodes causing one change in front to delay reporting, then each change which succeeds all jobs behind it will cause the window to grow | 20:06 |
| *** luqas has quit IRC | 20:06 | |
| sdague | fungi: yep | 20:06 |
| sdague | then if we get a reset somewhere in the pipeline, we cut in half | 20:06 |
| sdague | jog0: we actually started processing pretty well | 20:07 |
| fungi | alternatively, if the changes complete all their jobs sequentially, we never reach there | 20:07 |
| sdague | ?? | 20:07 |
| fungi | basically saying that if we determine window size from changes which complete their jobs before they report, we get an inconsistency where a single change in the queue has the potential to count toward window calculations more than once | 20:09 |
| jog0 | sdague: yeah all that work is finally paying off | 20:09 |
| sdague | fungi: only if we also triggered a failure | 20:09 |
| sdague | which would cut the pipeline in half | 20:09 |
| sdague | which I think is actually fine | 20:09 |
| sdague | the system is about using the most resources we think we can manage without killing ourselves | 20:10 |
| sdague | the current state, with slow rax nodes taking 90 minutes to report in, very quickly leads to 6hrs of backup on a Saturday | 20:10 |
| fungi | but yeah i think we're mainly lacking a distinct state for "all jobs completed successfully but not yet ready to report" | 20:10 |
| sdague | yep | 20:11 |
| sdague | there is basically a practical issues of time to report a success | 20:11 |
| sdague | plus I think the current growth factors mean we end up defaulting to min window all the time | 20:12 |
| sdague | and chunking to these 90 minute merges | 20:12 |
| sdague | because you don't grow the window until you start merging | 20:13 |
| sdague | so you don't have anything that started | 20:13 |
| fungi | part of the problem is jobs being too lengthy, and part is slaves which are running too slowly, i think | 20:13 |
| sdague | yeh, well I think processing the changes when finished will actually alleviate that | 20:14 |
| fungi | because otherwise changes would be completing their jobs more quickly and more sequentially | 20:14 |
| sdague | yeh, well everything is great when everything is faster :) | 20:14 |
| fungi | well, it's not just that, but the two compound one another | 20:15 |
| sdague | yep | 20:15 |
| sdague | which is why I was thinking about the midstream reporting, as I think that would address the fact that we starve out | 20:16 |
| fungi | also, getting to where we can have zuul mark jobs failing before they finish running (for example by reporting from the subunit stream) will help a lot of this i tink | 20:16 |
| fungi | think | 20:16 |
| sdague | fungi: maybe | 20:16 |
| sdague | though I'm not actually convinced | 20:16 |
| sdague | especially as the troublesome tests are actually kind of near the end | 20:17 |
| sdague | so I no longer think that will save us much | 20:17 |
| fungi | not all, but most of the backup i've seen has been jobs running a really long time because the particular way in which they failed causes a cascading effect throughout subsequent tests within that job making it go for far longer than it should | 20:17 |
| sdague | I've not seen one of those in a while | 20:18 |
| sdague | but I could believe it | 20:18 |
| fungi | i was seeing quite a few which at least looked that way week before last | 20:18 |
| sdague | yeh, we're a lot better now | 20:18 |
| sdague | oh, there was that swift job | 20:18 |
| sdague | we also should pull merge conflict jobs out of the window for length calculations | 20:19 |
| fungi | invariably when i'd see a bunch of changes with all their jobs completed waiting on one job on a change at the gate position, and i'd pull the console log for that job, it would have been running way longer than it should and marking test after test as fail | 20:19 |
| fungi | sdague: yeah, we at least already discussed having the window calculations count only testable changes as active | 20:20 |
| sdague | fungi: well, that wasn't the case yesterday | 20:20 |
| sdague | I think another good zuul optimization would be to throw out merge conflict changes if there is nothing in the queue ahead from that project | 20:20 |
| sdague | because I've seen plenty of that, where a change has to wait to get to the top of the queue to get a merge conflict report back to the user | 20:21 |
| sdague | and there was nothing else in the queue from that project | 20:21 |
| fungi | right, that's been discussed as well. there are still ways to make the mergeability determination more effective, but i think those are backburnered until we get git ref generation distributed | 20:22 |
| sdague | yep | 20:22 |
| *** yolanda_ has joined #openstack-infra | 20:22 | |
| sdague | hey, you know where I'd go looking to add the grenade logs to logstash? | 20:22 |
| sdague | I'd sort of like to get those there before the gate bug day tomorrow | 20:23 |
| fungi | since anything besides trivial "am i actually merging against head when this is failing" tests require additional merge operations, which will slow down zuul | 20:23 |
| sdague | yep | 20:23 |
| fungi | sdague: i think there's just a list of files the log processor knows to retrieve | 20:24 |
| sdague | fungi: know where that is? | 20:24 |
| sdague | greping has failed me | 20:24 |
| fungi | i'll have to go look | 20:24 |
| *** dkliban has joined #openstack-infra | 20:24 | |
| fungi | sdague: openstack-infra/config:modules/openstack_pr | 20:26 |
| fungi | oject/files/logstash/jenkins-log-client.yaml | 20:26 |
| fungi | (you can keep the extra newline as a token of my appreciation) | 20:26 |
| * fungi needs to step away for a bit and weekend | 20:27 | |
| *** yolanda_ has quit IRC | 20:27 | |
| *** sdake has joined #openstack-infra | 20:30 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 20:35 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Copy libvirt log file after tempest run https://review.openstack.org/61892 | 20:39 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Split config from projects list https://review.openstack.org/62187 | 20:48 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Unlaunchpadify projects.yaml https://review.openstack.org/62189 | 20:49 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Track direct-release projects in projects.yaml https://review.openstack.org/62190 | 20:49 |
| openstackgerrit | Monty Taylor proposed a change to openstack/requirements: Add python-openstackclient https://review.openstack.org/64562 | 20:50 |
| openstackgerrit | Monty Taylor proposed a change to openstack/requirements: psutil >= 1.1.0 is actually on PyPI https://review.openstack.org/68946 | 20:55 |
| *** Ryan_Lane has joined #openstack-infra | 20:55 | |
| *** hogepodge has quit IRC | 21:00 | |
| *** mgagne has joined #openstack-infra | 21:03 | |
| *** alexpilotti has quit IRC | 21:05 | |
| *** sarob has quit IRC | 21:21 | |
| *** david-lyle has quit IRC | 21:42 | |
| *** marun has joined #openstack-infra | 21:47 | |
| *** praneshp_ has joined #openstack-infra | 21:52 | |
| *** praneshp has quit IRC | 21:54 | |
| *** praneshp_ is now known as praneshp | 21:54 | |
| *** marun has quit IRC | 21:56 | |
| *** marun has joined #openstack-infra | 22:02 | |
| *** AlexF_ has quit IRC | 22:03 | |
| sdague | hmmmm I wonder why the head of the gate is still waiting on d-g nodes | 22:03 |
| *** b3nt_pin has joined #openstack-infra | 22:03 | |
| *** flaper87 is now known as flaper87|afk | 22:09 | |
| *** yassine has quit IRC | 22:16 | |
| *** yassine has joined #openstack-infra | 22:16 | |
| clarkb | sdague: its possible it grabbed nodes that failed and zuul retriggered them | 22:20 |
| clarkb | when that happens the retriggers go to the back of the queue :/ | 22:20 |
| sdague | clarkb: so it sat in gate for 1.5hrs, with apparently plenty of ready nodes before it allocated for the neutron isolated job, which it just did | 22:21 |
| clarkb | sdague: I am not in a position to grab logs but the scenario I describe could have happened repeatedly | 22:22 |
| sdague | ok no prob, if it's helpful, here is the zuul json from that time - http://dague.net/testing/2014-01-26.17-00.json | 22:22 |
| *** sdake has quit IRC | 22:25 | |
| *** yassine has quit IRC | 22:26 | |
| *** AaronGr_Zzz is now known as AaronGr | 22:28 | |
| *** marun has quit IRC | 22:32 | |
| lyxus | Hi Everyone, I have sent a Service Account Request however I provided the wrong public key. SHould I send another message or can I update it here | 22:35 |
| clarkb | lyxus: please send another message, it is sunday afternoon for most of us so a key here is likely to get lost in scrollback tomorrow | 22:37 |
| lyxus | clarkb, ok will do that right away ! Thanks ! | 22:38 |
| *** hashar has joined #openstack-infra | 22:45 | |
| *** senk has joined #openstack-infra | 22:47 | |
| *** fallenpegasus has quit IRC | 22:53 | |
| *** HenryG has quit IRC | 23:20 | |
| *** HenryG has joined #openstack-infra | 23:22 | |
| *** senk has quit IRC | 23:37 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Update documentation https://review.openstack.org/69211 | 23:54 |
| *** UtahDave has quit IRC | 23:56 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1262153 https://review.openstack.org/69242 | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!