| *** saschpe has quit IRC | 00:00 | |
| *** dcramer_ has joined #openstack-infra | 00:00 | |
| *** mgagne has quit IRC | 00:01 | |
| *** Trozz has joined #openstack-infra | 00:03 | |
| *** nkinder has quit IRC | 00:03 | |
| fungi | i'm continuing to look at it crossways and hope it'll shape up, but if it goes on much longer i'll push up a revert | 00:04 |
|---|---|---|
| boris-42 | mordred hi could you help Rally team to add first functional testing gate?) https://review.openstack.org/#/c/82617/ | 00:04 |
| mordred | boris-42: not right now - but I'll look at it a little later | 00:05 |
| boris-42 | mordred thanks=) | 00:05 |
| *** nkinder has joined #openstack-infra | 00:06 | |
| *** saschpe has joined #openstack-infra | 00:07 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add a very basic functional test https://review.openstack.org/82952 | 00:08 |
| *** atiwari has quit IRC | 00:09 | |
| jeblair | i say very basic, but it tests quite a lot actually, mostly because it's hard to test a little at this point | 00:13 |
| *** krtaylor has joined #openstack-infra | 00:20 | |
| *** dcramer_ has quit IRC | 00:22 | |
| *** sabari has quit IRC | 00:23 | |
| *** nkinder has quit IRC | 00:23 | |
| *** mestery has quit IRC | 00:23 | |
| *** matsuhashi has joined #openstack-infra | 00:27 | |
| *** thuc has joined #openstack-infra | 00:29 | |
| *** dcramer_ has joined #openstack-infra | 00:29 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/meetbot: Prevents meetbot from starting meetings in non-meeting channels https://review.openstack.org/82898 | 00:31 |
| clarkb | is ^ a problem? | 00:32 |
| clarkb | also did we revert the switchover to region b? | 00:32 |
| anteaya | clarkb: are you asking me about 82898? | 00:33 |
| clarkb | yes | 00:33 |
| anteaya | yes | 00:33 |
| anteaya | in -neutron this morning | 00:33 |
| anteaya | someone started a meeting | 00:33 |
| *** colinmcnamara has quit IRC | 00:33 | |
| *** morganfainberg is now known as morganfainberg_Z | 00:33 | |
| anteaya | so markmcclain asked me if there was a way to prevent it | 00:33 |
| *** thuc has quit IRC | 00:33 | |
| anteaya | so I went to #meetbot on oftc and this was suggested to me | 00:34 |
| *** yamahata has quit IRC | 00:34 | |
| *** dcramer_ has quit IRC | 00:34 | |
| pleia2 | so I thought about this some today, I wonder if there are teams who have meetings in their channels and want to retain the feature | 00:35 |
| anteaya | I'm open to changing the conditional | 00:35 |
| anteaya | I can do `if "neutron" in channel: return` | 00:35 |
| * pleia2 nods | 00:36 | |
| fungi | there are definitely some who do... one stackforge project (i forget which) mentioned using it in their channel for daily scrum meetings | 00:36 |
| anteaya | if we want to only target neutron | 00:36 |
| * anteaya changes the conditional | 00:36 | |
| jeblair | pleia2: well, we created the meetings channels to force people to use them, and schedule meetings centrally | 00:36 |
| *** SumitNaiksatam has quit IRC | 00:36 | |
| pleia2 | jeblair: yeah, but I think some do little less formal meetings and things | 00:36 |
| jeblair | we could revisit that of course, but that's the current intent... | 00:36 |
| clarkb | anteaya: roger I missed that | 00:36 |
| jeblair | pleia2: if there are openstack projects that do that, then we should have a discussion as a project about how to handle that.. | 00:37 |
| *** mriedem has joined #openstack-infra | 00:37 | |
| jeblair | pleia2: if there are stackforge projects that do that, i'm not particularly concerned | 00:37 |
| * pleia2 nods | 00:37 | |
| fungi | clarkb: we haven't reverted yet, but i'm starting to think it might be warranted (though not sure how we work out the cause of so many nova boot requests failing to complete) | 00:37 |
| fungi | other than asking hpcloud to look at their nova error logs, i guess | 00:38 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/meetbot: Prevents meetbot from starting meetings in openstack-neutron https://review.openstack.org/82898 | 00:38 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revert "Stop using hpcloud az1-az3 in favor of region-b" https://review.openstack.org/82954 | 00:39 |
| anteaya | so there is an option that just prevents meetings in neutron | 00:39 |
| anteaya | vote for your favourite | 00:39 |
| jeblair | anteaya: that was the same change | 00:39 |
| anteaya | I changed the conditional: if "neutron" in channel: | 00:40 |
| jeblair | anteaya: yes, but you updated the same change in gerrit so it is not possible to "vote for your favorite" unless your favorite is the most recent patchset. | 00:40 |
| anteaya | oh I didnt' know that | 00:41 |
| anteaya | fungi had previously merged a change from a stack of changes of mine | 00:41 |
| anteaya | so I erronously thought you could vote for any one of the three patchsets | 00:41 |
| pabelanger | fungi, I am noticing you are also getting the same network slowness from hpcloud-region-b: http://nodepool.openstack.org/image.log.2014-02-24 | 00:41 |
| anteaya | I didn't realize you were limited to just the latest one | 00:42 |
| pabelanger | I can't seem to get more then 150 kB/s for apt updates | 00:42 |
| *** gokrokve has joined #openstack-infra | 00:42 | |
| fungi | pabelanger: glad to know it's not just us... well, "glad" isn't quite right, but you know what i mean | 00:42 |
| pabelanger | fungi, Ya, trying to figure out if I was doing something wrong. Trying to see if there is some internal mirror someplace in region-b | 00:43 |
| anteaya | I'll let that steep until tomorrow, if there is consenseus with jeblair view, I can offer patchset 2 back up | 00:43 |
| anteaya | jeblair's view | 00:43 |
| *** rediskin has left #openstack-infra | 00:43 | |
| fungi | pabelanger: we're currently getting a lot of instance boot requests failing to complete | 00:43 |
| fungi | pabelanger: in addition to that less-problematic issue (from our standpoint) | 00:44 |
| pabelanger | fungi, just on region-b? | 00:45 |
| fungi | pabelanger: primarily anyway, yes | 00:45 |
| *** UtahDave has quit IRC | 00:45 | |
| fungi | pabelanger: we just merged a change a few hours ago to switch from the old hpcloud availability zones to region b, and have had lots of nodes there timing out after the boot call | 00:46 |
| *** dcramer_ has joined #openstack-infra | 00:46 | |
| fungi | far more than we anticipated anyway | 00:46 |
| pabelanger | fungi, Ya, see that today | 00:46 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for heat bug 1297560 https://review.openstack.org/82955 | 00:50 |
| uvirtbot | Launchpad bug 1297560 in heat "*tempest-dsvm-neutron-heat-slow fails with WaitConditionTimeout" [Undecided,New] https://launchpad.net/bugs/1297560 | 00:50 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Include provider names in timeout messages https://review.openstack.org/82959 | 00:55 |
| jeblair | fungi: ^ (and it's not on the crazy long multinode branch) | 00:55 |
| fungi | now you're just giving me excuses to not get around to the rest of the patches in your crazy long branch yet | 00:58 |
| fungi | not that it's that crazy long, i've just been slack on review volume recently | 00:59 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-dev/hacking: Add a tool that can fix the import order https://review.openstack.org/68988 | 01:00 |
| fungi | greptatsic | 01:01 |
| fungi | tastic | 01:01 |
| fungi | something | 01:01 |
| *** imcsk8_lap has quit IRC | 01:06 | |
| *** reed has quit IRC | 01:09 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Stop using hpcloud az1-az3 in favor of region-b" https://review.openstack.org/82954 | 01:11 |
| *** changbl has joined #openstack-infra | 01:12 | |
| *** matsuhashi has quit IRC | 01:12 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: WIP, getting train tracks, DO NOT MERGE https://review.openstack.org/82960 | 01:13 |
| *** matsuhashi has joined #openstack-infra | 01:14 | |
| *** sdake_ has quit IRC | 01:15 | |
| *** sdake has quit IRC | 01:16 | |
| *** krotscheck has quit IRC | 01:21 | |
| *** krotscheck has joined #openstack-infra | 01:21 | |
| *** rfolco has joined #openstack-infra | 01:23 | |
| *** weshay has quit IRC | 01:25 | |
| *** nosnos has joined #openstack-infra | 01:27 | |
| *** khyati has quit IRC | 01:29 | |
| *** krtaylor has quit IRC | 01:34 | |
| *** sdake has joined #openstack-infra | 01:35 | |
| *** sdake_ has joined #openstack-infra | 01:35 | |
| *** imcsk8_lap has joined #openstack-infra | 01:35 | |
| *** nkinder has joined #openstack-infra | 01:43 | |
| *** alexpilotti has quit IRC | 01:47 | |
| *** rfolco has quit IRC | 01:48 | |
| *** wenlock has joined #openstack-infra | 01:49 | |
| *** wenlock has quit IRC | 01:53 | |
| *** yjiang has joined #openstack-infra | 01:54 | |
| *** yamahata has joined #openstack-infra | 01:58 | |
| *** VijayTripathi has quit IRC | 01:59 | |
| *** matsuhashi has quit IRC | 02:00 | |
| *** wenlock has joined #openstack-infra | 02:03 | |
| *** mgagne has joined #openstack-infra | 02:03 | |
| *** matsuhashi has joined #openstack-infra | 02:08 | |
| *** thuc has joined #openstack-infra | 02:11 | |
| *** saschpe has quit IRC | 02:24 | |
| *** VijayTripathi has joined #openstack-infra | 02:25 | |
| *** harlowja is now known as harlowja_away | 02:27 | |
| *** saschpe has joined #openstack-infra | 02:27 | |
| *** thuc has quit IRC | 02:27 | |
| clarkb | krotscheck: jeblair: is there an existing plan for doing search in storyboard? | 02:28 |
| *** thuc has joined #openstack-infra | 02:28 | |
| clarkb | curious to know if you plan on doing something like sphinx or elasticsearch or $otherthing | 02:28 |
| pabelanger | fungi, here is my issue I think: https://github.com/openstack-infra/config/commit/cbd64b61751bf22d2955ffb7f531a02947a97edb | 02:30 |
| *** vkozhukalov has joined #openstack-infra | 02:31 | |
| pabelanger | testing the logic now | 02:31 |
| *** Alexandra is now known as alex-lunch | 02:31 | |
| *** jeckersb is now known as jeckersb_gone | 02:32 | |
| *** thuc has quit IRC | 02:32 | |
| *** zhiyan_ is now known as zhiyan | 02:32 | |
| *** dcramer_ has quit IRC | 02:36 | |
| clarkb | pabelanger: the syncs were causing problems? | 02:36 |
| fungi | pabelanger: interesting. you think the sync calls at the end of one or more of those scripts is causing your problem, or did we miss adding one somewhere? | 02:36 |
| pabelanger | clarkb, fungi no, I have not been calling sync. | 02:37 |
| pabelanger | I am just adding it now, and testing | 02:37 |
| clarkb | ah | 02:37 |
| fungi | aHH | 02:37 |
| pabelanger | yes | 02:37 |
| pabelanger | just found that commit | 02:37 |
| clarkb | my eyes are mostly back to normal now | 02:37 |
| pabelanger | after looking at source | 02:37 |
| fungi | we did indeed add it for... reasons | 02:37 |
| pabelanger | that's my guess, snapshot is happening too fast, corrupting image | 02:38 |
| *** khyati has joined #openstack-infra | 02:38 | |
| *** krtaylor has joined #openstack-infra | 02:38 | |
| *** BadCub01 has quit IRC | 02:39 | |
| *** nati_ueno has quit IRC | 02:42 | |
| *** thomasem has joined #openstack-infra | 02:53 | |
| *** imcsk8_lap has quit IRC | 02:55 | |
| *** gyee has quit IRC | 02:56 | |
| mordred | clarkb: I was thinking sphinxsearch | 02:58 |
| mordred | clarkb: I should ping some of the sphinxsearch guys and see if they want to do that for us... | 02:58 |
| pabelanger | fungi, clarkb: Yup, that was it. I needed to call sync | 03:01 |
| *** todd_dsm has joined #openstack-infra | 03:03 | |
| lifeless | there seem to be a lot of queued nodes atm | 03:04 |
| lifeless | or is it just me ? | 03:04 |
| lifeless | queued jobs, I mean | 03:04 |
| *** dcramer_ has joined #openstack-infra | 03:05 | |
| *** mriedem has quit IRC | 03:06 | |
| *** todd_dsm has quit IRC | 03:08 | |
| *** zns has quit IRC | 03:12 | |
| *** emagana has joined #openstack-infra | 03:13 | |
| *** matsuhashi has quit IRC | 03:15 | |
| *** thuc has joined #openstack-infra | 03:17 | |
| *** thuc has quit IRC | 03:19 | |
| *** thuc has joined #openstack-infra | 03:20 | |
| pabelanger | lifeless, I believe there is a backup from rolling back from hpcloud-region-b | 03:21 |
| mordred | clarkb, pabelanger, fungi: should I or anyone else communicate anything to hp? | 03:22 |
| pabelanger | mordred, well, I only started using hpcloud today, so my feedback is pretty limited. | 03:24 |
| pabelanger | but am seeing slow transfer rates to ubuntu archive for updates | 03:24 |
| mordred | pabelanger: welcome to the jungle | 03:24 |
| *** emagana has quit IRC | 03:24 | |
| pabelanger | I already talked to support about some internal mirror if there was one, they are getting back to me | 03:24 |
| *** thuc has quit IRC | 03:27 | |
| *** nosnos has quit IRC | 03:28 | |
| clarkb | lifeless: yes I think you are correct | 03:29 |
| clarkb | mordred: possibly, I wasn't around for much of the afternoon fun, but it sounded like node boot time was terrible | 03:29 |
| *** khyati has quit IRC | 03:36 | |
| *** gokrokve has quit IRC | 03:37 | |
| *** gokrokve_ has joined #openstack-infra | 03:38 | |
| *** emagana has joined #openstack-infra | 03:39 | |
| *** dkliban has quit IRC | 03:39 | |
| *** thomasem has quit IRC | 03:39 | |
| *** jepoy_ has joined #openstack-infra | 03:42 | |
| *** gokrokve_ has quit IRC | 03:42 | |
| *** alex-lunch is now known as Alexandra | 03:43 | |
| *** wchrisj has quit IRC | 03:43 | |
| *** jepoy has quit IRC | 03:44 | |
| *** blamar has quit IRC | 03:46 | |
| *** khyati has joined #openstack-infra | 03:46 | |
| *** thuc has joined #openstack-infra | 03:48 | |
| *** mestery has joined #openstack-infra | 03:51 | |
| *** emagana has quit IRC | 03:51 | |
| *** blamar has joined #openstack-infra | 03:56 | |
| *** blamar has quit IRC | 03:57 | |
| *** khyati has quit IRC | 03:58 | |
| *** blamar has joined #openstack-infra | 03:58 | |
| *** nati_ueno has joined #openstack-infra | 04:02 | |
| *** thuc_ has joined #openstack-infra | 04:03 | |
| *** thuc has quit IRC | 04:06 | |
| *** gokrokve has joined #openstack-infra | 04:07 | |
| *** todd_dsm has joined #openstack-infra | 04:10 | |
| *** blamar has quit IRC | 04:10 | |
| *** gokrokve has quit IRC | 04:13 | |
| *** sabari has joined #openstack-infra | 04:13 | |
| *** lcheng has joined #openstack-infra | 04:14 | |
| *** david-lyle has joined #openstack-infra | 04:16 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:18 | |
| *** khyati has joined #openstack-infra | 04:18 | |
| *** mihgen has joined #openstack-infra | 04:22 | |
| *** dolphm has quit IRC | 04:25 | |
| *** matsuhashi has joined #openstack-infra | 04:25 | |
| *** SumitNaiksatam has quit IRC | 04:25 | |
| *** mestery has quit IRC | 04:25 | |
| *** dolphm has joined #openstack-infra | 04:27 | |
| *** vkozhukalov has quit IRC | 04:28 | |
| *** pcrews has quit IRC | 04:28 | |
| *** mrodden has quit IRC | 04:30 | |
| *** nosnos has joined #openstack-infra | 04:36 | |
| *** todd_dsm has quit IRC | 04:39 | |
| *** blamar has joined #openstack-infra | 04:46 | |
| *** basha has joined #openstack-infra | 04:48 | |
| *** todd_dsm has joined #openstack-infra | 04:49 | |
| openstackgerrit | MORITA Kazutaka proposed a change to openstack-infra/config: Add Jenkins jobs for swift3 https://review.openstack.org/82986 | 04:54 |
| *** imcsk8_lap has joined #openstack-infra | 04:56 | |
| *** imcsk8_lap has quit IRC | 04:57 | |
| *** _nadya_ has joined #openstack-infra | 05:02 | |
| *** _nadya_ has quit IRC | 05:03 | |
| *** amcrn has quit IRC | 05:08 | |
| *** harlowja_at_home has joined #openstack-infra | 05:14 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Clean up a few errant tabs in setup.cfg https://review.openstack.org/82694 | 05:15 |
| *** skraynev_afk is now known as skraynev | 05:22 | |
| *** thuc_ has quit IRC | 05:26 | |
| *** thuc has joined #openstack-infra | 05:27 | |
| *** harlowja_at_home has quit IRC | 05:27 | |
| *** _nadya_ has joined #openstack-infra | 05:30 | |
| *** thuc has quit IRC | 05:31 | |
| *** talluri has joined #openstack-infra | 05:39 | |
| *** emagana has joined #openstack-infra | 05:40 | |
| *** mgagne1 has joined #openstack-infra | 05:45 | |
| *** mgagne has quit IRC | 05:46 | |
| *** nicedice has quit IRC | 05:51 | |
| *** ildikov_ has quit IRC | 05:52 | |
| *** e0ne has joined #openstack-infra | 05:55 | |
| *** nati_ueno has quit IRC | 05:57 | |
| *** _nadya_ has quit IRC | 06:00 | |
| *** khyati has quit IRC | 06:04 | |
| *** Ryan_Lane has joined #openstack-infra | 06:05 | |
| *** gokrokve has joined #openstack-infra | 06:08 | |
| *** Ryan_Lane has quit IRC | 06:11 | |
| *** sdake has quit IRC | 06:12 | |
| *** sdake_ has quit IRC | 06:12 | |
| *** gokrokve has quit IRC | 06:13 | |
| *** khyati has joined #openstack-infra | 06:14 | |
| *** saju_m has joined #openstack-infra | 06:19 | |
| *** sdake has joined #openstack-infra | 06:24 | |
| *** sdake has quit IRC | 06:24 | |
| *** sdake has joined #openstack-infra | 06:24 | |
| *** tnurlygayanov_ has joined #openstack-infra | 06:27 | |
| *** sdake_ has joined #openstack-infra | 06:27 | |
| *** tnurlygayanov has quit IRC | 06:27 | |
| *** mrda is now known as mrda_away | 06:31 | |
| *** matsuhashi has quit IRC | 06:33 | |
| *** VijayTripathi has quit IRC | 06:34 | |
| *** matsuhashi has joined #openstack-infra | 06:35 | |
| *** thuc has joined #openstack-infra | 06:37 | |
| *** ichavero|zZz has joined #openstack-infra | 06:41 | |
| *** thuc has quit IRC | 06:42 | |
| *** ichavero|zZz is now known as imcsk8_lap | 06:42 | |
| imcsk8_lap | hello are you guys habing problems with name resolution on the jenkins jobs?? i'm having them: http://logs.openstack.org/53/82953/1/check/gate-packstack-python27/11a0d3c/console.html | 06:43 |
| imcsk8_lap | (having) | 06:43 |
| *** mgagne1 has quit IRC | 06:44 | |
| *** afazekas has quit IRC | 06:46 | |
| *** e0ne has quit IRC | 06:47 | |
| *** e0ne has joined #openstack-infra | 06:48 | |
| *** jlibosva has joined #openstack-infra | 06:49 | |
| *** amcrn has joined #openstack-infra | 06:51 | |
| *** e0ne has quit IRC | 06:51 | |
| *** khyati has quit IRC | 07:00 | |
| *** vkozhukalov has joined #openstack-infra | 07:02 | |
| *** matsuhas_ has joined #openstack-infra | 07:03 | |
| *** matsuhas_ has quit IRC | 07:04 | |
| *** jcoufal has joined #openstack-infra | 07:04 | |
| *** matsuhas_ has joined #openstack-infra | 07:06 | |
| *** matsuhashi has quit IRC | 07:07 | |
| *** ildikov_ has joined #openstack-infra | 07:11 | |
| *** saju_m has quit IRC | 07:13 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added paging to list endpoints https://review.openstack.org/79757 | 07:17 |
| SergeyLukjanov | imcsk8_lap, hey, it's a known issue - https://bugs.launchpad.net/openstack-ci/+bug/1270382 | 07:18 |
| uvirtbot | Launchpad bug 1270382 in openstack-ci "Name resolution failure on gate job" [Medium,Confirmed] | 07:18 |
| *** wenlock has quit IRC | 07:20 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: DataBase Token Storage added https://review.openstack.org/82101 | 07:20 |
| *** afazekas has joined #openstack-infra | 07:21 | |
| *** afazekas has quit IRC | 07:25 | |
| *** sabari has quit IRC | 07:35 | |
| *** markmcclain has joined #openstack-infra | 07:37 | |
| *** lcheng has quit IRC | 07:37 | |
| *** afazekas has joined #openstack-infra | 07:40 | |
| *** Alexandra is now known as alex-gone | 07:41 | |
| *** yolanda has joined #openstack-infra | 07:46 | |
| *** mihgen has quit IRC | 07:50 | |
| *** johnthetubaguy has joined #openstack-infra | 07:51 | |
| *** johnthetubaguy1 has joined #openstack-infra | 07:54 | |
| *** johnthetubaguy has quit IRC | 07:54 | |
| *** medieval1 has quit IRC | 07:57 | |
| *** flaper87|afk is now known as flaper87 | 07:57 | |
| *** markmcclain has quit IRC | 07:57 | |
| *** mihgen has joined #openstack-infra | 07:57 | |
| *** matsuhas_ has quit IRC | 07:59 | |
| *** medieval1 has joined #openstack-infra | 08:01 | |
| *** vkozhukalov has left #openstack-infra | 08:02 | |
| *** matsuhashi has joined #openstack-infra | 08:04 | |
| *** gokrokve has joined #openstack-infra | 08:08 | |
| *** gokrokve has quit IRC | 08:12 | |
| *** mihgen has quit IRC | 08:13 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added prod convenience invocation https://review.openstack.org/82571 | 08:16 |
| *** pblaho has joined #openstack-infra | 08:18 | |
| *** hashar has joined #openstack-infra | 08:23 | |
| *** jgallard has joined #openstack-infra | 08:31 | |
| *** jgallard has quit IRC | 08:33 | |
| *** jgallard has joined #openstack-infra | 08:33 | |
| *** emagana has quit IRC | 08:33 | |
| *** chandankumar_ has joined #openstack-infra | 08:42 | |
| *** dizquierdo has joined #openstack-infra | 08:43 | |
| *** rlandy has joined #openstack-infra | 08:48 | |
| *** amotoki has quit IRC | 08:51 | |
| *** saschpe has quit IRC | 08:54 | |
| *** jcoufal has quit IRC | 08:59 | |
| *** jcoufal has joined #openstack-infra | 09:00 | |
| *** saschpe has joined #openstack-infra | 09:03 | |
| *** jpich has joined #openstack-infra | 09:03 | |
| *** mihgen has joined #openstack-infra | 09:06 | |
| *** yassine has joined #openstack-infra | 09:08 | |
| *** yassine has quit IRC | 09:08 | |
| *** yassine has joined #openstack-infra | 09:08 | |
| *** derekh has joined #openstack-infra | 09:11 | |
| *** AJaeger has quit IRC | 09:12 | |
| *** gokrokve has joined #openstack-infra | 09:13 | |
| *** gokrokve has quit IRC | 09:13 | |
| *** gokrokve has joined #openstack-infra | 09:14 | |
| *** gokrokve has quit IRC | 09:16 | |
| *** gokrokve has joined #openstack-infra | 09:17 | |
| *** AJaeger has joined #openstack-infra | 09:17 | |
| *** gokrokve has quit IRC | 09:21 | |
| *** andreaf has joined #openstack-infra | 09:22 | |
| *** johnthetubaguy1 has quit IRC | 09:23 | |
| *** ominakov has joined #openstack-infra | 09:24 | |
| *** dizquierdo has quit IRC | 09:27 | |
| *** hashar has quit IRC | 09:28 | |
| *** Trozz has quit IRC | 09:28 | |
| *** Trozz has joined #openstack-infra | 09:29 | |
| *** Trozz_ has joined #openstack-infra | 09:30 | |
| *** Trozz_ has quit IRC | 09:30 | |
| *** Trozz has quit IRC | 09:30 | |
| *** Trozz has joined #openstack-infra | 09:31 | |
| *** Trozz_ has joined #openstack-infra | 09:31 | |
| *** mich5225 has joined #openstack-infra | 09:32 | |
| *** Trozz has quit IRC | 09:33 | |
| *** Trozz_ has quit IRC | 09:33 | |
| *** mich5225 has quit IRC | 09:33 | |
| *** Trozz has joined #openstack-infra | 09:33 | |
| *** hashar has joined #openstack-infra | 09:35 | |
| *** e0ne has joined #openstack-infra | 09:39 | |
| *** Trozz has quit IRC | 09:39 | |
| *** Trozz has joined #openstack-infra | 09:40 | |
| *** derekh has quit IRC | 09:42 | |
| *** derekh has joined #openstack-infra | 09:42 | |
| *** Trozz has joined #openstack-infra | 09:43 | |
| *** pbelanyi has joined #openstack-infra | 09:44 | |
| *** gokrokve has joined #openstack-infra | 09:44 | |
| *** Trozz has quit IRC | 09:45 | |
| *** bauzas has joined #openstack-infra | 09:51 | |
| *** yjiang|2 has joined #openstack-infra | 10:00 | |
| *** yjiang has quit IRC | 10:00 | |
| *** jp_at_hp has joined #openstack-infra | 10:01 | |
| *** AJaeger has quit IRC | 10:03 | |
| *** matsuhashi has quit IRC | 10:09 | |
| *** AJaeger has joined #openstack-infra | 10:12 | |
| *** yamahata has quit IRC | 10:15 | |
| *** dizquierdo has joined #openstack-infra | 10:17 | |
| *** chandankumar_ has quit IRC | 10:22 | |
| *** dkliban has joined #openstack-infra | 10:23 | |
| *** Trozz has joined #openstack-infra | 10:31 | |
| *** Trozz has quit IRC | 10:34 | |
| *** yjiang|2 has quit IRC | 10:35 | |
| *** matsuhashi has joined #openstack-infra | 10:39 | |
| *** Trozz has joined #openstack-infra | 10:41 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: DB api refactoring https://review.openstack.org/83027 | 10:41 |
| *** Trozz has quit IRC | 10:42 | |
| *** Trozz has joined #openstack-infra | 10:44 | |
| *** Trozz has quit IRC | 10:45 | |
| *** hashar has quit IRC | 10:45 | |
| *** hashar has joined #openstack-infra | 10:48 | |
| ttx | sdague: Hi! could you +2/APRV https://review.openstack.org/#/c/81084/ before I start -2ing all requirements changes ? | 10:49 |
| *** jepoy_ has quit IRC | 10:53 | |
| *** jepoy has joined #openstack-infra | 10:53 | |
| sdague | ttx: done | 10:54 |
| *** che-arne has joined #openstack-infra | 10:54 | |
| sdague | -2 away! | 10:54 |
| *** hashar has quit IRC | 10:54 | |
| ttx | I'm going through them -1ing the ones I think we could consider, -2ing the ones we probably should freeze | 10:54 |
| *** hashar has joined #openstack-infra | 10:54 | |
| ttx | sdague: do you happen to know how functional ">=1.6.1,<2.0,>=2.1" is @ https://review.openstack.org/#/c/79796/ ? | 10:55 |
| sdague | that I don't know | 10:55 |
| sdague | I've seen != | 10:55 |
| ttx | sdague: will ask mordred about it | 10:55 |
| sdague | ttx: works for me | 10:56 |
| *** jepoy_ has joined #openstack-infra | 10:57 | |
| sdague | so testing it on the command line, it does what you think it might do, even if it doesn't make logical sense from a math perspective | 10:57 |
| ttx | sdague: maybe I should just +2 https://review.openstack.org/#/c/65863/3 and make it pass just below the bar. Opinion ? | 10:58 |
| *** jepoy has quit IRC | 10:58 | |
| sdague | I could go either way on that. Realistically, stuff isn't going to use it until juno | 10:58 |
| ttx | ok, can wait a week | 10:58 |
| sdague | I honestly think it's ok that certain parts of our process slow down during release. Getting people to focus on the core release is goodness, instead of running full speed ahead on stuff that's juno only. | 11:00 |
| sdague | so something like this has almost no chance of negatively impacting the release technically, but it's fine to wait as well | 11:00 |
| *** lcostantino has joined #openstack-infra | 11:01 | |
| ttx | yep, -2ed | 11:01 |
| *** jgallard has quit IRC | 11:10 | |
| ttx | sdague: about https://review.openstack.org/#/c/81373/4 -- is the gate using the lower bound or the upper bound in testing ? Your comment seems to suggest it uses the upper bound ? | 11:11 |
| sdague | so... it depends | 11:12 |
| sdague | basically if the requirement is satisfied by 12.04 packages, we use that | 11:12 |
| sdague | otherwise it's a pip install | 11:12 |
| sdague | which means it's going to be the newest version | 11:12 |
| ttx | sdague: I guess &2.04 has a psutil1 in that case | 11:13 |
| ttx | 12.04* | 11:13 |
| sdague | probably | 11:13 |
| sdague | we can see in a log | 11:13 |
| ttx | sdague: ok thx | 11:13 |
| sdague | http://logs.openstack.org/96/81896/7/gate/gate-tempest-dsvm-full/ea6ae39/logs/pip-freeze.txt.gz | 11:14 |
| sdague | or not... maybe neutron needs it? | 11:14 |
| sdague | no, it's removed there as well | 11:15 |
| sdague | so what concerns me about this is it was being brought in because of some rather complicated incubator code | 11:16 |
| *** jepoy_ has quit IRC | 11:16 | |
| sdague | which is why it had a version lock | 11:16 |
| sdague | but we're not good about ever getting past version locks | 11:16 |
| sdague | so I really want to avoid them | 11:16 |
| *** jepoy has joined #openstack-infra | 11:16 | |
| sdague | and instead have the code written to support multiple versions | 11:16 |
| ttx | Looks like glance wants psutil | 11:17 |
| ttx | and neutron | 11:17 |
| sdague | I guess they take from the distro | 11:17 |
| ttx | got pulled from neutron, so maybe it's only in glance | 11:19 |
| *** saju_m has joined #openstack-infra | 11:20 | |
| *** jepoy has quit IRC | 11:21 | |
| *** e0ne has quit IRC | 11:23 | |
| *** Trozz has joined #openstack-infra | 11:23 | |
| *** e0ne has joined #openstack-infra | 11:23 | |
| *** dkliban_ has joined #openstack-infra | 11:27 | |
| *** dkliban has quit IRC | 11:27 | |
| *** basha_ has joined #openstack-infra | 11:27 | |
| Mithrandir | hmm, so, for j-j-b, would people be ok with extending it so it can manage plugins and such too? | 11:28 |
| *** ociuhandu has quit IRC | 11:29 | |
| *** e0ne has quit IRC | 11:29 | |
| *** mwagner_ is now known as mwagner_notHere | 11:29 | |
| *** Trozz has joined #openstack-infra | 11:29 | |
| *** basha has quit IRC | 11:29 | |
| *** basha_ is now known as basha | 11:29 | |
| *** ominakov has quit IRC | 11:30 | |
| *** dkliban_ is now known as dkliban | 11:31 | |
| *** Trozz has quit IRC | 11:36 | |
| ihrachys | just got dns failure at https://jenkins04.openstack.org/job/gate-oslo.messaging-pep8/63/console when resolving git.openstack.org. Any ideas? | 11:36 |
| sdague | Bug 1270382 - Name resolution failure on gate job - it's being worked on by jeblair | 11:39 |
| uvirtbot | Launchpad bug 1270382 in openstack-ci "Name resolution failure on gate job" [Medium,Confirmed] https://launchpad.net/bugs/1270382 | 11:39 |
| ihrachys | ok, tnx | 11:39 |
| sdague | rax has intermittent dns issues | 11:39 |
| sdague | root cause unknown | 11:39 |
| *** yamahata has joined #openstack-infra | 11:40 | |
| ihrachys | sdague: safe to recheck, right? | 11:40 |
| sdague | ihrachys: yes | 11:41 |
| imcsk8_lap | ihrachys: hello, one quick question. how do i trigger the recheck? | 11:42 |
| sdague | imcsk8_lap: leave a comment that is | 11:43 |
| sdague | recheck bug ###### | 11:43 |
| sdague | where ##### is the bug number | 11:43 |
| ihrachys | imcsk8_lap: https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures | 11:43 |
| *** hashar has quit IRC | 11:43 | |
| ihrachys | imcsk8_lap: the link is posted by jenkins in reviews when they fail in gate | 11:43 |
| imcsk8_lap | saju_m, ihrachys thanks! | 11:43 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Fix 009 migration https://review.openstack.org/83050 | 11:44 |
| *** gokrokve has quit IRC | 11:45 | |
| *** gokrokve has joined #openstack-infra | 11:46 | |
| *** gokrokve has quit IRC | 11:50 | |
| *** jcoufal has quit IRC | 11:52 | |
| *** ociuhandu has joined #openstack-infra | 11:53 | |
| *** wenlock has joined #openstack-infra | 11:55 | |
| *** jcoufal has joined #openstack-infra | 11:55 | |
| *** rfolco has joined #openstack-infra | 11:56 | |
| *** gokrokve has joined #openstack-infra | 11:57 | |
| *** bogdando has quit IRC | 11:58 | |
| *** wenlock has quit IRC | 11:58 | |
| *** e0ne has joined #openstack-infra | 12:00 | |
| *** mwagner_lap has quit IRC | 12:00 | |
| *** Trozz has joined #openstack-infra | 12:02 | |
| *** Trozz has quit IRC | 12:03 | |
| *** wchrisj has joined #openstack-infra | 12:04 | |
| *** e0ne has quit IRC | 12:04 | |
| *** ArxCruz has joined #openstack-infra | 12:04 | |
| *** e0ne has joined #openstack-infra | 12:05 | |
| *** yamahata has quit IRC | 12:06 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/meetbot: Prevents meetbot from starting meetings in non-meeting channels https://review.openstack.org/82898 | 12:06 |
| *** basha has quit IRC | 12:06 | |
| anteaya | SergeyLukjanov: care to share your thoughts with me? | 12:09 |
| SergeyLukjanov | anteaya, hey | 12:09 |
| anteaya | can you expand on your concerns so that I can understand them? | 12:09 |
| SergeyLukjanov | anteaya, okay, I'll try to explain:) | 12:10 |
| *** e0ne has quit IRC | 12:10 | |
| SergeyLukjanov | anteaya, #1 is that it's quite useful IMO to run meetings in project channels | 12:10 |
| *** matsuhashi has quit IRC | 12:10 | |
| SergeyLukjanov | anteaya, #2 is that blacklisting channels will not help in general | 12:10 |
| *** bogdando has joined #openstack-infra | 12:10 | |
| SergeyLukjanov | someone could start meeting in #.*meeting.* | 12:11 |
| anteaya | I'm not sure I agree with your use of the term blacklisting | 12:11 |
| *** matsuhashi has joined #openstack-infra | 12:11 | |
| anteaya | so regarding running meetings in project channels | 12:11 |
| SergeyLukjanov | anteaya, do you like whitelisting? :) | 12:11 |
| anteaya | my understanding of jim's point last night is that he feels that running meetings in project channels is a bad idea | 12:11 |
| anteaya | since that is why we have the meeting channels | 12:12 |
| SergeyLukjanov | shame on me, I've skipped reading scrollback today's morning | 12:12 |
| anteaya | mind you I am paraphrasing, but that is the take away I had regarding jim's points from last nigh | 12:12 |
| sdague | SergeyLukjanov: I'm with jeblair on that | 12:12 |
| sdague | the reason for the meeting channels is so there is a "public square" where meetings happen | 12:12 |
| *** pdmars has joined #openstack-infra | 12:13 | |
| anteaya | there was some discussion to not worry about what the stackforge projects do | 12:13 |
| *** matsuhas_ has joined #openstack-infra | 12:13 | |
| *** matsuhashi has quit IRC | 12:13 | |
| SergeyLukjanov | anteaya, agreed | 12:13 |
| anteaya | in that case, I am open to changing the conditional to also checking if "stackforge" is in the channel topic | 12:13 |
| SergeyLukjanov | sdague, ++ for public square | 12:14 |
| *** e0ne has joined #openstack-infra | 12:14 | |
| * persia really appreciates being able to review multiple project meeting backscroll as a way to see what is happening in projects for which the channels are too high volume to follow directly, which only works with *-meeting-* channels. | 12:14 | |
| SergeyLukjanov | my main concern is this it doesn't solve root issue - anyone could start meeting | 12:14 |
| SergeyLukjanov | anteaya, and I agree with you that we don't like to store lists of chairs in infra | 12:15 |
| SergeyLukjanov | so, I think I'm going to +2 your change | 12:15 |
| anteaya | thanks | 12:15 |
| SergeyLukjanov | because of lack of solutions | 12:15 |
| SergeyLukjanov | suggestions* | 12:15 |
| anteaya | anyone can start a meeting in a meeting channel | 12:15 |
| anteaya | which is the design, yes | 12:16 |
| anteaya | since chairs for meetings are to be decided by the folks holding the meeting | 12:16 |
| anteaya | upon checking the list I have for channels meetbot logs to eavesdrop: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/eavesdrop.pp#n31 | 12:16 |
| anteaya | I don't see any stackforge channels there | 12:16 |
| sdague | anteaya: murano ? | 12:17 |
| *** dims_ has quit IRC | 12:18 | |
| SergeyLukjanov | anteaya, murano, climate, dns, sdks(?) | 12:18 |
| *** dcramer_ has quit IRC | 12:19 | |
| anteaya | ah so they don't put "stackforge" in the channel topic | 12:20 |
| sdague | anteaya: when did neutron run a meeting in their channel? | 12:20 |
| sdague | has this been a real issue? | 12:20 |
| anteaya | yesterday | 12:20 |
| fungi | right, there's currently no "easy" way to discern stackforge from non-stackforge channels since lots of them have names which start with #openstack- | 12:20 |
| anteaya | someone showed up and started a meeting | 12:21 |
| anteaya | mark asked me is there a way to prevent it in future | 12:21 |
| *** dims_ has joined #openstack-infra | 12:21 | |
| anteaya | I didn't think it would become what it is | 12:21 |
| sdague | interesting | 12:21 |
| anteaya | now they were probably not paying attention to what channel they were in | 12:21 |
| anteaya | and ended the meeting when it was brought to their attention | 12:22 |
| sdague | fungi: something is really weird with nodepool btw | 12:22 |
| anteaya | but I was asked could something be done, so I tried | 12:22 |
| *** e0ne_ has joined #openstack-infra | 12:23 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/devstack-gate: Correcting the right path to mysql log in Fedora 19 https://review.openstack.org/79967 | 12:23 |
| *** e0ne has quit IRC | 12:23 | |
| fungi | sdague: agreed, looks like maybe there aren't as many operable nodes as it thinks. i wonder if we confused it by shuffling quota between old and new hpcloud regions and back again last night | 12:24 |
| fungi | checking for stale nodes now | 12:24 |
| *** e0ne has joined #openstack-infra | 12:25 | |
| *** aysyd has joined #openstack-infra | 12:26 | |
| *** david-lyle has quit IRC | 12:27 | |
| fungi | somehow we ended up with a ton of py3k-precise nodes in a ready state in hpcloud for many hours now | 12:28 |
| *** e0ne_ has quit IRC | 12:28 | |
| sdague | this would be one of those places where some monitoring would be nice for sanity, so it doesn't require a person to alert folks that everything is wonky | 12:28 |
| fungi | out of curiosity, what would your wonkameter for this particular issue look like? | 12:29 |
| *** jcoufal has quit IRC | 12:30 | |
| fungi | the py3k-precise nodes do actually appear to be getting used, i think nodepool has merely misestimated the ratio of them it needs to keep in hpcloud | 12:30 |
| sdague | so the thing that sticks out is the number of ready nodes vs. queued | 12:30 |
| *** jcoufal has joined #openstack-infra | 12:30 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Add the second tripleo provider https://review.openstack.org/83057 | 12:30 |
| fungi | i'm checking the logs now to see if it still thinks it's supposed to be adding a bunch of them | 12:31 |
| fungi | most of those ready nodes are in that state for about 10.75 hours, so i think nodepool spun up the majority when the quota for the old hpcloud azs was reintroduced last night | 12:34 |
| fungi | that roughly coincides with the overall rise in building/ready nodes and workers lines on the graphs | 12:36 |
| fungi | then the comments left by the auto-abandon cron job triggering check pipeline runs plus the periodic jobs firing at the same time around 6.5 hours ago accounts for the spike in waiting jobs which never subsided because we're working through them too slowly | 12:37 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Moving to soft delete https://review.openstack.org/83058 | 12:37 |
| fungi | picking a ready py3k-precise node at random from that time period, it's up, reachable via ssh and in jenkins waiting to grab a job, so i think we're really just looking at an early miscalculation on node types needed by nodepool, probably due to there having been so many jobs waiting on py3k-precise nodes prior to that (because of the situation with all the build timeouts in region b) | 12:39 |
| sdague | ok | 12:40 |
| fungi | anyway, i can speed things up by just killing off a bunch of the py3k-precise nodes we don't currently need so we can use that quota for more necessary purposes | 12:40 |
| sdague | sure | 12:40 |
| *** dprince has joined #openstack-infra | 12:41 | |
| sdague | just seems we could come up with what normal operating would be for nodepool, and see if we are staying inside the lines with normal over time | 12:42 |
| *** jepoy has joined #openstack-infra | 12:42 | |
| fungi | i agree, it does seem that way... curious whether it actually turns out to be the case | 12:43 |
| *** dkranz has joined #openstack-infra | 12:44 | |
| fungi | anyway, i'm deleting any nodes which have been waiting in a ready state for >3 hours... that gets us back about 300 nodes of available capacity for other purposes | 12:44 |
| fungi | presumably nodepool will recalculate what it actually needs based on current demand now | 12:44 |
| sdague | fungi: sure, but we'll never know unless we try :) | 12:45 |
| fungi | i believe, looking at the logs, that it was already doing so... it just doesn't have a provision yet for "i have too many of *this* type, i should delete some) | 12:45 |
| fungi | and we were reclaiming available slots in the quota too slowly because we don't normally use very many py3k-precise nodes compared to other types | 12:46 |
| *** jeckersb_gone is now known as jeckersb | 12:47 | |
| openstackgerrit | MORITA Kazutaka proposed a change to openstack-infra/config: Add Jenkins jobs for swift3 https://review.openstack.org/82986 | 12:47 |
| *** hashar has joined #openstack-infra | 12:49 | |
| *** jepoy has quit IRC | 12:50 | |
| *** jepoy has joined #openstack-infra | 12:51 | |
| *** jaypipes has quit IRC | 12:51 | |
| *** thomasem has joined #openstack-infra | 12:52 | |
| *** gokrokve has quit IRC | 12:53 | |
| *** gokrokve has joined #openstack-infra | 12:53 | |
| *** mrodden has joined #openstack-infra | 12:53 | |
| *** lcheng has joined #openstack-infra | 12:55 | |
| *** jaypipes has joined #openstack-infra | 12:56 | |
| fungi | we've also got a little over 70 nodes which have been in a delete state for more than 3 hours, so i'm trying to more forcibly deleting them now | 12:57 |
| *** thuc has joined #openstack-infra | 12:57 | |
| *** gokrokve has quit IRC | 12:57 | |
| *** jhesketh__ has quit IRC | 12:58 | |
| sdague | cool | 12:58 |
| *** thuc_ has joined #openstack-infra | 12:58 | |
| *** thuc has quit IRC | 13:02 | |
| *** bknudson has joined #openstack-infra | 13:03 | |
| *** wenlock has joined #openstack-infra | 13:03 | |
| *** unicell has quit IRC | 13:04 | |
| *** mfer has joined #openstack-infra | 13:04 | |
| *** weshay has joined #openstack-infra | 13:05 | |
| *** e0ne_ has joined #openstack-infra | 13:07 | |
| *** gokrokve has joined #openstack-infra | 13:07 | |
| *** matsuhas_ has quit IRC | 13:07 | |
| *** jgallard has joined #openstack-infra | 13:07 | |
| *** CaptTofu has quit IRC | 13:07 | |
| *** CaptTofu has joined #openstack-infra | 13:08 | |
| *** johnthetubaguy has joined #openstack-infra | 13:09 | |
| *** CaptTofu has quit IRC | 13:09 | |
| *** CaptTofu has joined #openstack-infra | 13:09 | |
| *** e0ne has quit IRC | 13:10 | |
| *** johnthetubaguy1 has joined #openstack-infra | 13:12 | |
| *** mwagner_lap has joined #openstack-infra | 13:13 | |
| *** johnthetubaguy has quit IRC | 13:13 | |
| *** CaptTofu has quit IRC | 13:14 | |
| *** CaptTofu_ has joined #openstack-infra | 13:14 | |
| *** julim has joined #openstack-infra | 13:16 | |
| *** adalbas has joined #openstack-infra | 13:16 | |
| *** basha has joined #openstack-infra | 13:17 | |
| *** mihgen has quit IRC | 13:17 | |
| *** yamahata has joined #openstack-infra | 13:18 | |
| *** zhiyan is now known as zhiyan_ | 13:19 | |
| *** thomasem has quit IRC | 13:19 | |
| *** zhiyan_ is now known as zhiyan | 13:19 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Fix wait_for_tarball 'None' error https://review.openstack.org/83065 | 13:20 |
| *** lcheng has quit IRC | 13:20 | |
| *** bknudson has quit IRC | 13:23 | |
| *** unicell has joined #openstack-infra | 13:25 | |
| *** signed8bit has joined #openstack-infra | 13:31 | |
| *** yassine_ has joined #openstack-infra | 13:32 | |
| *** mrodden has quit IRC | 13:35 | |
| *** nosnos has quit IRC | 13:35 | |
| *** afazekas_ has joined #openstack-infra | 13:36 | |
| *** afazekas has quit IRC | 13:39 | |
| *** jeckersb is now known as jeckersb_gone | 13:39 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: be extra explicit for people reading logs https://review.openstack.org/83071 | 13:46 |
| sdague | ttx: ^^^ | 13:47 |
| *** jeckersb_gone is now known as jeckersb | 13:47 | |
| ttx | commented | 13:49 |
| *** nkinder has quit IRC | 13:49 | |
| sdague | ttx: so the issue is, that failure specifies the devstack log only if devstack is the load | 13:50 |
| sdague | but d-g is used for things besides devstack | 13:50 |
| sdague | gate_hook is a variable | 13:50 |
| sdague | and is replaceable | 13:51 |
| *** sahumada_ has joined #openstack-infra | 13:53 | |
| *** pcrews has joined #openstack-infra | 13:55 | |
| ttx | hah | 13:58 |
| *** skraynev is now known as skraynev_going_h | 13:59 | |
| *** skraynev_going_h is now known as skraynev_afk | 13:59 | |
| *** thuc_ has quit IRC | 14:00 | |
| ttx | sdague: ok +1ed -- "exit code" should also hint at "look just above" | 14:00 |
| *** thuc has joined #openstack-infra | 14:01 | |
| sdague | so, honestly if "please see relevant logs" and 3 lines up is "logs will be in..." aren't enough | 14:03 |
| sdague | then I'm not sure what will be | 14:03 |
| *** eharney has joined #openstack-infra | 14:04 | |
| *** prad_ has joined #openstack-infra | 14:04 | |
| *** thuc has quit IRC | 14:05 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Prepare for Juno dev https://review.openstack.org/83075 | 14:05 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Enable sahara as default in gate https://review.openstack.org/83076 | 14:05 |
| *** thuc has joined #openstack-infra | 14:05 | |
| *** thuc has quit IRC | 14:06 | |
| SergeyLukjanov | ttx, sdague, could you please take a look at https://review.openstack.org/#/c/81058/? | 14:06 |
| *** imcsk8_lap has quit IRC | 14:06 | |
| *** thuc has joined #openstack-infra | 14:07 | |
| *** imcsk8_lap has joined #openstack-infra | 14:07 | |
| sdague | sure, but putting the ? in breaks the url :) | 14:07 |
| ttx | SergeyLukjanov: still cleaning up I see | 14:07 |
| SergeyLukjanov | sdague, hmm, it works ok in my irc client, will know | 14:08 |
| SergeyLukjanov | ttx, yeah, I have several CRs for final cleanup | 14:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove extra g722 sounds https://review.openstack.org/82360 | 14:09 |
| SergeyLukjanov | and I've opened CRs for enabling someday in J sahara in gate | 14:09 |
| SergeyLukjanov | I think it's the last non-approved cleanup https://review.openstack.org/#/c/79548/ | 14:10 |
| *** jgrimm has joined #openstack-infra | 14:11 | |
| sdague | xchat puts the ? in the url | 14:12 |
| *** sandywalsh has joined #openstack-infra | 14:12 | |
| sdague | because it's a valid character | 14:13 |
| *** talluri has quit IRC | 14:13 | |
| SergeyLukjanov | sdague, I'm using textual | 14:13 |
| SergeyLukjanov | https://review.openstack.org/#/c/81058/?a | 14:13 |
| SergeyLukjanov | it's parsed as url | 14:13 |
| SergeyLukjanov | https://review.openstack.org/#/c/81058/a? | 14:14 |
| SergeyLukjanov | https://review.openstack.org/#/c/81058/a=b? | 14:14 |
| *** basha has quit IRC | 14:14 | |
| SergeyLukjanov | heh, it just always ignore ? in the end of url :) | 14:14 |
| sdague | SergeyLukjanov: in xchat I can click them | 14:14 |
| *** CaptTofu_ has quit IRC | 14:15 | |
| SergeyLukjanov | sdague, mee too, but my client doesn't include trailing ? to the link, nice | 14:15 |
| sdague | well, it's valid | 14:15 |
| *** CaptTofu has joined #openstack-infra | 14:16 | |
| *** dcramer_ has joined #openstack-infra | 14:16 | |
| *** amotoki has joined #openstack-infra | 14:18 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove savanna from logstash/jenkins-log-client.yaml https://review.openstack.org/79548 | 14:20 |
| *** CaptTofu has quit IRC | 14:20 | |
| SergeyLukjanov | fungi, thank you sir! | 14:20 |
| *** thuc has quit IRC | 14:23 | |
| *** thuc has joined #openstack-infra | 14:23 | |
| *** thuc has quit IRC | 14:28 | |
| *** rcleere has joined #openstack-infra | 14:30 | |
| *** e0ne has joined #openstack-infra | 14:36 | |
| *** alexpilotti has joined #openstack-infra | 14:39 | |
| *** e0ne_ has quit IRC | 14:39 | |
| *** branen_ has quit IRC | 14:40 | |
| *** chandan_kumar has quit IRC | 14:42 | |
| *** nkinder has joined #openstack-infra | 14:42 | |
| *** sahumada_ has quit IRC | 14:43 | |
| *** skraynev_afk is now known as skraynev | 14:44 | |
| *** chandankumar_ has joined #openstack-infra | 14:44 | |
| jeblair | sdague: can we get an idea of the iops needed for devstack runs from the dstat logs we have? | 14:45 |
| jeblair | sdague: also, good morning :) | 14:45 |
| sdague | jeblair: good morning | 14:46 |
| *** chandankumar_ has quit IRC | 14:46 | |
| sdague | lets see what's in a log... | 14:46 |
| *** mestery has joined #openstack-infra | 14:46 | |
| sdague | http://logs.openstack.org/05/82905/5/check/check-tempest-dsvm-full/7c0869b/logs/dstat.txt.gz | 14:47 |
| jeblair | so looks like we can get disk throughput at least | 14:47 |
| sdague | yeh | 14:47 |
| *** chandan_kumar has joined #openstack-infra | 14:47 | |
| sdague | we can probably enable other options | 14:47 |
| sdague | what are you looking for ? | 14:47 |
| jeblair | cool, i'll try to take a median of that... | 14:48 |
| *** chandan_kumar has quit IRC | 14:48 | |
| jeblair | sdague: i'm talking with someone who might have some cloud to give us | 14:48 |
| *** chandan_kumar has joined #openstack-infra | 14:48 | |
| sdague | -r, --io | 14:48 |
| sdague | enable I/O request stats (read, write requests) | 14:48 |
| sdague | we could turn that on as well | 14:48 |
| sdague | if it was useful | 14:48 |
| *** thedodd has joined #openstack-infra | 14:48 | |
| *** reed has joined #openstack-infra | 14:48 | |
| jeblair | cool, i'll throw that in a d-g change, and at least we'll have that run (or, well, 16 runs or whatever :) | 14:48 |
| *** ominakov has joined #openstack-infra | 14:48 | |
| sdague | I think we are running the dstat defaults.... just 'cuz no one really had other bits they wanted | 14:48 |
| sdague | yeh, actually in devstack change | 14:49 |
| sdague | if there is anything else in dstat you think would be useful for either diagnose or scaling, please just propose it | 14:49 |
| sdague | dstat seems pretty light impact on the system, so it's fine to collect that | 14:50 |
| jeblair | sdague: ack, thx | 14:50 |
| *** CaptTofu has joined #openstack-infra | 14:50 | |
| sdague | ls | 14:51 |
| sdague | grrrr | 14:51 |
| jeblair | no such file or directory | 14:52 |
| sdague | :) | 14:52 |
| *** hashar has quit IRC | 14:52 | |
| *** david-lyle has joined #openstack-infra | 14:53 | |
| *** emagana has joined #openstack-infra | 14:55 | |
| *** chandan_kumar has quit IRC | 14:55 | |
| jeblair | sdague: https://review.openstack.org/83083 | 14:56 |
| *** chandan_kumar has joined #openstack-infra | 14:56 | |
| sdague | probably worth adding +2 | 14:58 |
| *** sweston has quit IRC | 14:58 | |
| *** emagana has quit IRC | 15:00 | |
| *** julim_ has joined #openstack-infra | 15:00 | |
| *** julim has quit IRC | 15:01 | |
| openstackgerrit | Nathan Kinder proposed a change to openstack-infra/config: Add the openstack-security-notes project https://review.openstack.org/73157 | 15:01 |
| *** kmartin has quit IRC | 15:02 | |
| Mithrandir | hmm, is dbailey on irc? I'd really like something like https://review.openstack.org/#/c/48783/17 to move forward. | 15:02 |
| *** chandan_kumar has quit IRC | 15:03 | |
| *** yamahata has quit IRC | 15:03 | |
| *** VijayTripathi has joined #openstack-infra | 15:03 | |
| jeblair | Mithrandir: i don't think he is normally :( | 15:03 |
| *** atiwari has joined #openstack-infra | 15:04 | |
| *** thuc has joined #openstack-infra | 15:04 | |
| Mithrandir | hmm, is Khai Do on irc, then? I think dbailey might be spurred into action if he gets a reply to the question there | 15:05 |
| Mithrandir | or somebody else could bring it forward. | 15:05 |
| *** thuc_ has joined #openstack-infra | 15:05 | |
| *** mestery has quit IRC | 15:05 | |
| jeblair | Mithrandir: zaro is khai do | 15:05 |
| *** weshay has quit IRC | 15:05 | |
| Mithrandir | jeblair: great, thanks. | 15:05 |
| *** yamahata has joined #openstack-infra | 15:06 | |
| Mithrandir | zaro: any chance you could respond to dbailey's question at the end of https://review.openstack.org/#/c/48783/17 ? | 15:06 |
| Mithrandir | zaro: I'd really like to see something like that feature go in, so.. :-) | 15:06 |
| anteaya | Mithrandir: he might not be around for an hour or so | 15:06 |
| anteaya | he is on PST | 15:06 |
| *** hashar has joined #openstack-infra | 15:06 | |
| fungi | Mithrandir: darraghb seems to be /join'd on weekdays 09:00-17:00 utc (ireland i think) | 15:06 |
| Mithrandir | anteaya: right, hopefully he reads backlog. | 15:07 |
| Mithrandir | fungi: thanks. :-) | 15:07 |
| *** mihgen has joined #openstack-infra | 15:07 | |
| fungi | oh, though last time i saw him in #openstack-dev was a year ago, nevermind | 15:07 |
| Mithrandir | oh, ok | 15:07 |
| Mithrandir | that's a while ago indeed. | 15:07 |
| anteaya | Mithrandir: he usually does I think | 15:07 |
| *** vhoward has joined #openstack-infra | 15:07 | |
| fungi | unless he changed his nick, 2013-04-26 17:46:21 was the last time he /part'd | 15:07 |
| jeblair | sdague: ha, our median disk read throughput is 0. :) | 15:08 |
| sdague | yeh, not surprising | 15:08 |
| *** thuc has quit IRC | 15:09 | |
| jeblair | we read nothing 2/3 of the time | 15:09 |
| *** yamahata has quit IRC | 15:09 | |
| *** chandan_kumar has joined #openstack-infra | 15:09 | |
| sdague | once we get the cirros image into page cache that takes away a lot of our reads | 15:09 |
| *** yamahata has joined #openstack-infra | 15:10 | |
| jeblair | our median write throughput is 16M/s | 15:10 |
| *** thuc_ has quit IRC | 15:12 | |
| *** thuc has joined #openstack-infra | 15:13 | |
| *** UtahDave has joined #openstack-infra | 15:13 | |
| *** yolanda has quit IRC | 15:13 | |
| *** mflobo has joined #openstack-infra | 15:15 | |
| *** sweston has joined #openstack-infra | 15:16 | |
| *** wchrisj has quit IRC | 15:18 | |
| *** lcostantino has quit IRC | 15:18 | |
| jeblair | sdague: also https://review.openstack.org/83100 to add memory stats | 15:19 |
| *** markmc has joined #openstack-infra | 15:19 | |
| *** UtahDave has left #openstack-infra | 15:20 | |
| *** weshay has joined #openstack-infra | 15:21 | |
| sdague | jeblair: right, volumes :) | 15:23 |
| jeblair | mean and median cpu usage values are about 35%, max 96% | 15:26 |
| jeblair | maybe when i'm done with this script, i could tack it onto devstack-gate to output a summary for each job | 15:26 |
| sdague | sure | 15:27 |
| sdague | I think that would be handy | 15:27 |
| *** bknudson has joined #openstack-infra | 15:30 | |
| zaro | morning | 15:31 |
| mordred | sdague: reading scrollback... | 15:31 |
| mordred | 11:12:43 sdague | basically if the requirement is satisfied by 12.04 packages, we use that | 15:31 |
| mordred | 11:12:52 sdague | otherwise it's a pip install | 15:31 |
| mordred | just to bring it up again - I'd REALLY like that ^^ to stop being the answer at some point | 15:31 |
| sdague | mordred: sure | 15:31 |
| sdague | it's just lower priority right now | 15:32 |
| openstackgerrit | gordon chung proposed a change to openstack-infra/devstack-gate: fix trove merge https://review.openstack.org/83103 | 15:32 |
| sdague | honestly, that may be part of the big drag down knock out test matrix conversation at summit | 15:33 |
| *** johnthetubaguy1 is now known as johnthetubaguy | 15:33 | |
| sdague | because with multinode, mongo jobs, max and min versions | 15:33 |
| sdague | we're talking about a lot of new combos | 15:33 |
| mordred | yah | 15:33 |
| sdague | that people would like to see | 15:33 |
| mordred | btw - I don't care about min versions, for the record, and I don't think we should either | 15:34 |
| sdague | mordred: why? | 15:34 |
| mordred | because I don't think it provides value to | 15:34 |
| fungi | it's also not feasible to implement | 15:35 |
| zaro | Mithrandir: lookin at it now | 15:35 |
| mordred | that too | 15:35 |
| sdague | it doesn't provide value to know that we work with sqla 0.7? | 15:35 |
| mordred | nope | 15:35 |
| mordred | not at all | 15:35 |
| sdague | ok | 15:35 |
| *** mihgen has quit IRC | 15:36 | |
| jeblair | does in provide value to know that we don't work with it? | 15:36 |
| fungi | at least as long as we're using pip and pip continues to lack any real dependency solver, we can't just say "install the lowest versions of stuff" unless we manually spider the entire dependency tree first (and need to figure out what we do about things which specify no min version too, including on transitive deps) | 15:36 |
| clarkb | mordred jeblair fungi sdague et al holger is looking for feedback on tox bug 150 so that it can possibly be fixed in 1.7.1. I am indifferent to the suggested fix as it requires updating all tox.ini files and we can use 1.7.0 today of we do that | 15:36 |
| uvirtbot | Launchpad bug 150 in baz "change arch_escape_classes" [Medium,New] https://launchpad.net/bugs/150 | 15:36 |
| sdague | ok, I'm trying to reconcile that with putting min versions global requirements | 15:36 |
| jeblair | because i think it would be great of the graphite devs knew that graphite doesn't work with the min version of django they specify | 15:36 |
| sdague | fungi: I'm not saying it's trivial | 15:37 |
| clarkb | any chance you guys can catch up on the thread and provide feedback if you feel more strongly? | 15:37 |
| *** lcheng has joined #openstack-infra | 15:37 | |
| mordred | jeblair: ok. good point - let me rephrase the thing I'm trying to say | 15:37 |
| *** jepoy_ has joined #openstack-infra | 15:37 | |
| Mithrandir | zaro: great, thanks. | 15:38 |
| *** emagana has joined #openstack-infra | 15:38 | |
| mordred | which is that I _do_ think there is value in asserting that we don't work with version X - I think that given the way pip's dep solver works (per fungi) the work to try to infer which old versions we might not work with anymore is going to outweigh that value from an automated system | 15:39 |
| sdague | if we actually don't care at all about anything less than latest, we can just fix that with adding -U to the pip install in devstack right? | 15:39 |
| mordred | so while I'm fine with someone filing a bug saying "we dont' work with sqla 0.7 anymore, we should bump the lower bound" - I think trying to discover that will be almost imposibble | 15:40 |
| sdague | mordred: I don't think we'd discover a perfect answer | 15:40 |
| mordred | sdague: well, -U has some interesting semantics ... but yeah | 15:40 |
| sdague | but I do think we'd discover a rough answer that was useful | 15:40 |
| clarkb | https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror is the link to the tox bug | 15:40 |
| mordred | sdague: k. I'm not opposed to rough answer | 15:41 |
| clarkb | I intend on commenting as soon as I am actually awake | 15:41 |
| *** jepoy has quit IRC | 15:41 | |
| clarkb | but more feedback is better :) | 15:41 |
| mordred | I just don't want to try to gate on somethign that looks like the collected set of mins | 15:41 |
| fungi | keep in mind the corollary to my assertion is that we *could* potentially test this by either not using pip to discover and install the deps we need, or by implementing a good dep solver in pip | 15:41 |
| *** mflobo has quit IRC | 15:41 | |
| sdague | it would be interesting how many shadow deps are still in there | 15:41 |
| fungi | the latter of which, i think, would be of great benefit to a lot of people, not just openstack | 15:42 |
| sdague | we've gotten more and more explicit over time | 15:42 |
| sdague | there is also the issue around python with native bindings | 15:42 |
| sdague | the pip -U is going to add a ton of time to gate jobs | 15:42 |
| sdague | 3 minutes to build numpy, for instance | 15:42 |
| fungi | wheels | 15:42 |
| mordred | we have wheels | 15:42 |
| clarkb | sdague we are logging "shadow" deps now | 15:42 |
| sdague | for everything? | 15:43 |
| mordred | sdague: yes | 15:43 |
| clarkb | we know what isnt listed | 15:43 |
| mordred | the mirror has wheels for everything | 15:43 |
| sdague | ok | 15:43 |
| fungi | sdague: we do a pip freeze | 15:43 |
| mordred | and we build them for the platforms we devstack on | 15:43 |
| fungi | though mordred... with modern pip should we be doing pip list instead of pip freeze? | 15:43 |
| mordred | fungi: probably | 15:43 |
| *** saju_m has quit IRC | 15:43 | |
| fungi | it seems to pick up things freeze doesn't, sometimes, from what i've seen | 15:43 |
| dstufft | they are for different purposes | 15:44 |
| mordred | sdague: anyway - my main point in opening this can of worms, is that I think some-things-from-apt-some-things-from-pip is a thing we should get away from | 15:44 |
| sdague | well, now you make me curious | 15:44 |
| fungi | dstufft: is there a good writeup somewhere for what their respective purposes are? | 15:44 |
| dstufft | pip freeze is used to create a requirements file to "freeze" an environment | 15:44 |
| sdague | Sean Dague proposed a change to openstack-dev/devstack: attempt pip install -U https://review.openstack.org/83106 | 15:44 |
| mordred | espeically since dstufft tells us it's a use case completely unsuppoed by him | 15:44 |
| dstufft | pip list is more of a UX thing | 15:44 |
| dstufft | er UI | 15:44 |
| mordred | sdague: so - the biggest problem with -U | 15:44 |
| anteaya | clarkb: after tox files are updated, is 1.7.1 better for us in any way, compared to 1.7.0? | 15:45 |
| dstufft | we only exclude a few things from pip freeze | 15:45 |
| mordred | sdague: is that it changes how transitive ordering is going to work :) | 15:45 |
| dstufft | mostly becuase they are things you probably don't want frozen | 15:45 |
| fungi | dstufft: ahh, setuptools being one example | 15:45 |
| dstufft | like pip itself, setuptools | 15:45 |
| dstufft | distribute | 15:45 |
| dstufft | I think are the main 3 | 15:45 |
| sdague | mordred: sure, well lets see what happens | 15:45 |
| zaro | Mithrandir: i've commented, but the change is in abandoned state. | 15:45 |
| sdague | it will at least be data in the explosion | 15:45 |
| fungi | makes sense. so from a logging perspective, i think we *do* want to switch to pip list | 15:45 |
| mordred | sdague: since later declared deps will -U their transitive deps, whereas without -U, the transitive deps of later deps that were installed transitively by earlier declared deps will not re-install | 15:45 |
| mordred | sdague: ++ | 15:45 |
| clarkb | anteaya: don't know yet, they are looking for feedback on the proposed fix. IMO helps preserve the old behavior with new config so it is a reasonable translation compared to the alternative of embedding a shell script | 15:46 |
| dstufft | I think we also ignore some stdlib stuff that used to be third party modules and still have packaging metadata in the stdlib for them | 15:46 |
| dstufft | like argparse in 2.7 or something | 15:46 |
| Mithrandir | zaro: I'm happy to revive it if it doesn't get picked up. | 15:46 |
| anteaya | clarkb: k | 15:46 |
| dstufft | fungi: yea if you want something for diagnostics pip log is reasonable | 15:46 |
| mordred | sdague: oh - you _will_ see a slower build there - because I dont' believe we're configuring the mirror to pick up the wheel dirs | 15:46 |
| Mithrandir | I don't know what the usual procedure for that is, I'd like to move it forward, but not step on toes. | 15:46 |
| dstufft | also yes, mve away from half apt to half pip | 15:46 |
| mordred | but the data on what would get installed and if it woudl work will still be good | 15:46 |
| * clarkb goes ahead and tries to write up a comment now | 15:46 | |
| clarkb | before hpk afks | 15:47 |
| anteaya | clarkb: I am logged into lp and don't have permission to view: https://bugs.launchpad.net/baz/+bug/150 | 15:47 |
| uvirtbot | Launchpad bug 150 in baz "change arch_escape_classes" [Medium,New] | 15:47 |
| clarkb | anteaya: that is the wrong bug | 15:48 |
| dstufft | mordred: do you use the same wheel house between consecutive runs? | 15:48 |
| clarkb | tox hosts bugs on bitbucket, I dropped a proper link not too long ago in sb | 15:48 |
| anteaya | ah urvibot served that up | 15:48 |
| mordred | dstufft: I don't follow the question? | 15:48 |
| dstufft | mordred: I assume you're building wheels on the mirror with `pip wheel` ? | 15:48 |
| mordred | dstufft: yes! | 15:48 |
| anteaya | yes, I got the bitbucket one, just wanted to ensure I wasn't missing something | 15:48 |
| dstufft | mordred: that has a destination directory (the wheel house) | 15:49 |
| anteaya | uvirbot is being too helpful | 15:49 |
| dstufft | do you use the same one between runs | 15:49 |
| mordred | I do not believe we're using the same wheelhouse | 15:49 |
| mordred | although I don't think we're opposed to | 15:49 |
| dstufft | I think if you set --find-links to that wheel house, iyou won't rebuild wheels you already have built | 15:49 |
| jeblair | that is one really mixed metaphor, btw. | 15:50 |
| mordred | yes - I know we do that during the run - so taht we don't double-build wheels for repeated deps | 15:50 |
| mordred | jeblair: right/ | 15:50 |
| mordred | ? | 15:50 |
| dstufft | mordred: well just saying if you do that between consecutive runs it shouldn't inflate your mirror run time too much | 15:51 |
| dstufft | since most/all of your wheels should already be built then | 15:52 |
| *** wchrisj has joined #openstack-infra | 15:52 | |
| mordred | dstufft: indeed | 15:52 |
| *** mgagne has joined #openstack-infra | 15:58 | |
| *** mestery has joined #openstack-infra | 16:01 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc https://review.openstack.org/52080 | 16:01 |
| clarkb | I responded to tox bug 150, please do chime in if you want | 16:01 |
| uvirtbot | Launchpad bug 150 in baz "change arch_escape_classes" [Medium,New] https://launchpad.net/bugs/150 | 16:01 |
| zaro | Mithrandir: i think you'll need to get the owner to un-abandone | 16:01 |
| clarkb | Mithrandir: zaro: a gerrit admin can restore changes, what change needs restoring? | 16:02 |
| Mithrandir | clarkb: https://review.openstack.org/#/c/48783/17 | 16:03 |
| Mithrandir | and thanks. | 16:03 |
| zaro | clarkb: Mithrandir: review in question is https://review.openstack.org/#/c/48783/ | 16:03 |
| * Mithrandir goes to the shop to find some grub to eat | 16:03 | |
| *** pblaho has quit IRC | 16:04 | |
| *** wchrisj has quit IRC | 16:05 | |
| anteaya | for some reason git clone git://git.openstack.org/openstack-infra/gerritlib hangs for me | 16:06 |
| anteaya | git farm is quiet: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=2 | 16:06 |
| *** wchrisj has joined #openstack-infra | 16:07 | |
| anteaya | https:// works though | 16:07 |
| fungi | anteaya: are you able to clone other repositories over git:// protocol? might be something blocking that port outbound | 16:09 |
| *** Sukhdev has joined #openstack-infra | 16:10 | |
| *** _nadya_ has joined #openstack-infra | 16:10 | |
| *** sahumada has joined #openstack-infra | 16:11 | |
| *** wchrisj has quit IRC | 16:12 | |
| *** mgagne1 has joined #openstack-infra | 16:12 | |
| *** mgagne has quit IRC | 16:12 | |
| *** skraynev is now known as skraynev_afk | 16:14 | |
| *** weshay has quit IRC | 16:14 | |
| *** weshay has joined #openstack-infra | 16:15 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/gerritlib: Add empty-commit flag when creating a project with no commits https://review.openstack.org/83116 | 16:15 |
| anteaya | fungi: good question | 16:15 |
| anteaya | yes just cloned with: git clone git://git.openstack.org/openstack/api-site | 16:16 |
| dhellmann | can I get another pbr-core member to take a look at https://review.openstack.org/#/c/82356/ please? I would like to cut a release before our RC period starts. | 16:16 |
| jeblair | sdague: dstat is python! | 16:16 |
| jeblair | sdague: so, "16E", as in, 16 exabytes, comes up as a value in network throughput far too often for me to believe. | 16:18 |
| *** Ryan_Lane has joined #openstack-infra | 16:18 | |
| clarkb | jeblair: we should run elasticsearch on those VMs :) | 16:19 |
| sdague | jeblair: yes it is | 16:19 |
| *** maxbit has joined #openstack-infra | 16:19 | |
| sdague | it also has a well defined plugin api | 16:19 |
| fungi | anteaya: interesting... i just successfully cloned gerritlib via git protocol directly from each of the 5 git farm backends | 16:19 |
| sdague | jeblair: presumably there is an overrun somewhere | 16:19 |
| anteaya | fungi: let's assume derp until we here otherwise | 16:20 |
| anteaya | hear | 16:20 |
| jeblair | sdague: yeah, i'm looking to see if it's using the 16 or 32 bit counters | 16:20 |
| jeblair | i mean 32 or 64 | 16:20 |
| anteaya | and thanks for checking the git farm | 16:20 |
| *** e0ne_ has joined #openstack-infra | 16:21 | |
| clarkb | anteaya: anyone can reach the backends as well. Just git clone git://git01.openstack.org:29418/openstack-infra/gerritlib for example | 16:21 |
| clarkb | I really should set up a plan to switch the backend ports to the non offset port numbers | 16:21 |
| anteaya | clarkb: ah thanks | 16:22 |
| *** khyati has joined #openstack-infra | 16:23 | |
| openstackgerrit | A change was merged to openstack/requirements: Remove obsolete python-savannaclient https://review.openstack.org/81084 | 16:23 |
| *** e0ne has quit IRC | 16:24 | |
| *** boris-42 has quit IRC | 16:24 | |
| *** SumitNaiksatam has joined #openstack-infra | 16:24 | |
| * SergeyLukjanov reading scrollback | 16:27 | |
| *** hogepodge has quit IRC | 16:27 | |
| *** rfolco has quit IRC | 16:29 | |
| *** rfolco has joined #openstack-infra | 16:29 | |
| *** bauzas has quit IRC | 16:29 | |
| *** hogepodge has joined #openstack-infra | 16:29 | |
| ttx | sdague: another cleanup in requireemnts: https://review.openstack.org/#/c/75371/ | 16:30 |
| *** jepoy_ has quit IRC | 16:30 | |
| sdague | ttx: sure | 16:30 |
| *** jepoy has joined #openstack-infra | 16:30 | |
| openstackgerrit | A change was merged to openstack-infra/release-tools: Fix wait_for_tarball 'None' error https://review.openstack.org/83065 | 16:30 |
| clarkb | sdague: re https://review.openstack.org/#/c/83071/1 any desire to implement jeblair's suggestion before I approve? | 16:30 |
| *** afazekas_ has quit IRC | 16:30 | |
| zaro | clarkb: if you are in reviewing mode, could you please take a look at https://review.openstack.org/#/c/52080 ? | 16:31 |
| pabelanger | jeblair, I found a side affect of setting min-ready = 0, once everything is setup and processing. If a new job is submitted into zuul, zuul will error out with lost jobs: | 16:31 |
| pabelanger | 2014-03-26 16:24:58,241 ERROR zuul.Gearman: Job <gear.Job 0x7fdce013c650 handle: None name: build:gate-kickstandproject-ci-puppet-puppet-syntax unique: 44980fb4bb104d81bb9ce37a0d45dddf> is not registered with Gearman | 16:31 |
| pabelanger | doh | 16:31 |
| pabelanger | sorry, should have been pastebin URL | 16:31 |
| sdague | clarkb: sure, I can fix that | 16:31 |
| pabelanger | http://pastebin.com/DuPmZt1B | 16:31 |
| clarkb | zaro: I may have intentionally avoided that one. I feel like if I review it I will -2 it | 16:32 |
| clarkb | zaro: but there is clearly interest in it so a -2 is probably not fair | 16:32 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: be extra explicit for people reading logs https://review.openstack.org/83071 | 16:33 |
| clarkb | zaro: it appears to exploit a possible bug in jenkins to do a thing that JJB arguably shouldn't do | 16:33 |
| *** jasondotstar has joined #openstack-infra | 16:33 | |
| jeblair | pabelanger: yeah, the job isn't listed in gearman until a worker shows up and registers it. once it is, it stays there, regardless of whether any workers are registered | 16:34 |
| *** luisg has quit IRC | 16:34 | |
| jeblair | pabelanger: zuul has that check to make sure that if you typo a job name, you don't end up with jobs stuck waiting around forever | 16:34 |
| pabelanger | jeblair, ya, so the chicken / egg issue I guess | 16:34 |
| *** jepoy has quit IRC | 16:35 | |
| *** freyes has joined #openstack-infra | 16:35 | |
| jeblair | pabelanger: we were discussing that it would be nice to have a timeout in zuul, so if zuul enqueues a job and nothing runs it after, say, 8 hours, zuul considers it failed | 16:35 |
| jeblair | pabelanger: if that existed, then i think it would be safer for zuul to skip the 'does job exist' check... | 16:35 |
| jeblair | pabelanger: or we could make that check optional in a config file | 16:35 |
| *** rlandy has quit IRC | 16:35 | |
| pabelanger | nice | 16:36 |
| *** boris-42 has joined #openstack-infra | 16:36 | |
| *** wchrisj has joined #openstack-infra | 16:36 | |
| *** Ryan_Lane has quit IRC | 16:36 | |
| sdague | mordred: pip install -U fails in new and unexpected ways - http://logs.openstack.org/06/83106/1/check/check-tempest-dsvm-full/a0312f3/logs/devstacklog.txt.gz | 16:37 |
| sdague | because of an inability to find olso.config after doing that | 16:37 |
| zaro | clarkb: it's been around for a while so we should accept or reject, don't want to ignore. | 16:37 |
| *** kmartin has joined #openstack-infra | 16:38 | |
| * anteaya goes for a walk | 16:38 | |
| zaro | infra is opposed to new gerrit change screen, correct? | 16:39 |
| pleia2 | good morning | 16:39 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/storyboard: Mention load_projects in dev install https://review.openstack.org/83129 | 16:39 |
| zaro | from WIP plugin doc: Old change screen doesn't support JS API. So that the popup dialog is not shown if "Work In Progress" and "Ready In Review" buttons are used and no comments can be provided. New change screen should be used for best experience with WIP Workflow plugin | 16:39 |
| zaro | jeblair, fungi , clarkb ^ | 16:40 |
| fungi | fun! | 16:40 |
| *** melwitt has joined #openstack-infra | 16:41 | |
| boris-42 | fungi hi | 16:41 |
| boris-42 | fungi could you pls help with merging this patch https://review.openstack.org/#/c/82617/ =) | 16:42 |
| jeblair | zaro: well, that's not going to work | 16:42 |
| boris-42 | fungi we would like to add our first functional test=) | 16:42 |
| Mithrandir | clarkb: re the "don't overwrite job descs", I'd rather do that by having the token be configurable and have token = None be "don't add anything and ignore description" | 16:45 |
| *** bhuvan has joined #openstack-infra | 16:45 | |
| derekh | I've been trying to get a handle on job failures and run times for tripleo jobs http://goodsquishy.com/downloads/tripleo-jobs.html | 16:45 |
| derekh | then jogo pointed me at graphite | 16:46 |
| *** atiwari has quit IRC | 16:46 | |
| Mithrandir | clarkb: though, as I note in the change, I think the real fix is to have a "notes" plugin where you can add random notes. | 16:46 |
| Mithrandir | and not abuse the description. | 16:46 |
| jeblair | derekh: i may be able to help there | 16:46 |
| derekh | the plot I've been trying doesn't give the times I expect http://graphite.openstack.org/render/?width=586&height=308&from=-36hours&target=movingAverage(stats.timers.zuul.pipeline.check-tripleo.resident_time.mean,%20%2712hours%27) | 16:46 |
| derekh | jeblair: cool, also I wondering if there is any precedence for ci jobs sending their own statsd stats? | 16:47 |
| *** pbelanyi has quit IRC | 16:47 | |
| jeblair | derekh: i think times are in milliseconds, so you need to scale by .00001666666666666666 to get minutes | 16:47 |
| derekh | jeblair: I havn't used graphite (or statsd) befor, but am willing to read up if pointed in the right direction | 16:48 |
| *** hashar has quit IRC | 16:48 | |
| *** signed8bit has quit IRC | 16:48 | |
| derekh | jeblair: ahh ok, so maybe the times are correct | 16:48 |
| jeblair | derekh: sorry 0.00001666666666666666 (graphite really wants that leading 0) | 16:48 |
| jeblair | http://graphite.openstack.org/render/?width=586&height=308&from=-36hours&target=movingAverage%28scale%28stats.timers.zuul.pipeline.check-tripleo.resident_time.mean,0.00001666666666666666%29,%20%2712hours%27%29 | 16:48 |
| jeblair | derekh: so that looks something like that ^ | 16:48 |
| openstackgerrit | A change was merged to openstack-infra/config: Add check-rally-install-* template and jobs https://review.openstack.org/82617 | 16:49 |
| *** mestery has quit IRC | 16:49 | |
| jeblair | derekh: so changes mostly stay in that pipeline for around 1 hour | 16:49 |
| Mithrandir | it feels a lot like https://xkcd.com/1172/ | 16:49 |
| derekh | jeblair: ahh that looks better, so that hour is included in the total time | 16:49 |
| jeblair | derekh: (resident time is time change spent in queue, which should be ~= max(runtimes of all jobs) if the system is not busy, or much higher if it is busy and waiting on nodes) | 16:50 |
| *** jasondotstar has quit IRC | 16:50 | |
| *** vkozhukalov has joined #openstack-infra | 16:51 | |
| derekh | jeblair: so is there any timer available that doesn't include the time spent in the queue waiting? | 16:51 |
| *** luisg has joined #openstack-infra | 16:51 | |
| jeblair | derekh: yes | 16:52 |
| jeblair | derekh: http://graphite.openstack.org/render/?width=586&height=308&from=-36hours&target=movingAverage%28scale%28stats.timers.zuul.pipeline.check-tripleo.job.check-tripleo-undercloud-precise.SUCCESS.mean,0.00001666666666666666%29,%20%2712hours%27%29 | 16:52 |
| *** luisg has quit IRC | 16:52 | |
| jeblair | derekh: that's run time of successful check-tripleo-undercloud-precise jobs in the check-tripleo pipeline | 16:52 |
| mordred | sdague: wow. that is both new and unexpected | 16:53 |
| jeblair | derekh: you can put '*' in place of success to get success and failure on two lines, and you can use the 'summarize' them to include both values on one line | 16:54 |
| jeblair | s/them/function/ | 16:54 |
| *** vkozhukalov has left #openstack-infra | 16:54 | |
| sdague | mordred: given the other issues around namespace packages vs. install -e, I'm not entirely surprised | 16:55 |
| derekh | jeblair: great I'll dig a big more and see what I can pull together, I may be wrong but I'm thinking I'll leave the Failures out for this since they could happen at any stage and scew results | 16:55 |
| *** gokrokve has quit IRC | 16:55 | |
| jeblair | derekh: yeah, that's part of why they're broken out like that; same thing for the pipeline name -- that way we can include gate runs but exclude check runs... | 16:56 |
| jeblair | derekh: there's a change in review that will let us specify "the integrated gate" for these kinds of stats, which will be very helpful | 16:56 |
| *** sabari2 has joined #openstack-infra | 16:57 | |
| *** chandan_kumar has quit IRC | 16:57 | |
| *** harlowja_away is now known as harlowja | 16:58 | |
| mordred | sdague: I still don't grok the namepsace package issue and I'm sad I was away when it came up | 16:58 |
| *** gokrokve_ has joined #openstack-infra | 16:58 | |
| sdague | I still don't grok it entirely either | 16:59 |
| openstackgerrit | Jason Kincl proposed a change to openstack-infra/jenkins-job-builder: Add support for Copy Artifact project copy permissions https://review.openstack.org/82551 | 16:59 |
| *** atiwari has joined #openstack-infra | 17:00 | |
| derekh | jeblair: ok thanks, btw the script to generate that HTML page is here https://review.openstack.org/#/c/81236/ if infra want it and thing it would be helpful, otherwise if I get what I'm looking for from graphite the script will just probably go away at some stage | 17:00 |
| *** jlibosva has quit IRC | 17:00 | |
| *** nati_ueno has joined #openstack-infra | 17:00 | |
| jogo | clarkb: nova log are now much smaller http://logs.openstack.org/30/68030/2/gate/gate-tempest-dsvm-full/39524fa/logs/ | 17:01 |
| *** sabari2 has quit IRC | 17:02 | |
| *** khyati has quit IRC | 17:02 | |
| *** sabari2 has joined #openstack-infra | 17:02 | |
| *** sabari2 is now known as sabari | 17:03 | |
| *** wchrisj has quit IRC | 17:05 | |
| *** krotscheck_ has joined #openstack-infra | 17:07 | |
| *** krotscheck has quit IRC | 17:08 | |
| *** krotscheck_ is now known as krotscheck | 17:08 | |
| openstackgerrit | Nadya Privalova proposed a change to openstack-infra/devstack-gate: Fix ceilometer-anotification and trove start https://review.openstack.org/83134 | 17:08 |
| *** mestery has joined #openstack-infra | 17:09 | |
| SergeyLukjanov | sdague, please, take a look at ^^ | 17:10 |
| *** mkoderer has quit IRC | 17:11 | |
| SergeyLukjanov | sdague, it means that trove isn't enabled in gate at all | 17:11 |
| mordred | really? whatever happend to those tests ... vipul? (I don't see hub_cap around) | 17:12 |
| *** lcostantino has joined #openstack-infra | 17:13 | |
| SergeyLukjanov | sdague, correction https://review.openstack.org/#/c/83103/ | 17:14 |
| SergeyLukjanov | mordred, lack of punctuation | 17:14 |
| SergeyLukjanov | mordred, https://review.openstack.org/#/c/83103/ | 17:14 |
| *** e0ne_ has quit IRC | 17:15 | |
| *** amcrn has quit IRC | 17:15 | |
| *** mkoderer has joined #openstack-infra | 17:15 | |
| fungi | SergeyLukjanov: yeah, that's definitely unfortunate. just double-checking that havana failure on it before i approve | 17:16 |
| mordred | SergeyLukjanov: that's impressive | 17:16 |
| *** sweston has quit IRC | 17:17 | |
| fungi | SergeyLukjanov: is that failure because havana trove doesn't work with postgres? worried about breaking stable/havana with that | 17:18 |
| mordred | fungi: should we have trove enabled in havana? | 17:18 |
| jeblair | fungi: hrm, the service addition should only apply to master.. | 17:18 |
| fungi | wasn't there a change up to fix the branch override issue which was causing trove to get started in havana even though it wasn't supposed to be? did that merge? if so, doesn't seem to be working | 17:19 |
| SergeyLukjanov | fungi, it should be only for master | 17:19 |
| * fungi goes on a review hunt | 17:19 | |
| jeblair | fungi: this one? https://review.openstack.org/#/c/82496/ | 17:19 |
| fungi | ahh, yep | 17:20 |
| fungi | +2'ing that again | 17:20 |
| fungi | okay, so nothing new there | 17:20 |
| clarkb | Mithrandir: ++ to both ideas | 17:21 |
| jeblair | separately, i wonder about: setup_localrc "new" "$GRENADE_OLD_BRANCH" | 17:21 |
| clarkb | jogo: woot thanks | 17:21 |
| jeblair | i mean, that looks really suspicious | 17:21 |
| clarkb | jogo: was that a collection of changes to do that? | 17:21 |
| clarkb | zaro: I agree JJB in general shouldn't ignore that change and hasnt which is why I have been able to | 17:22 |
| vipul | mordred: I believe a patch just went in a few days ago enabling trove in devstack-gate | 17:22 |
| vipul | it's likely we didn't actually notice it wasn't really working | 17:23 |
| mordred | ok. cool | 17:23 |
| * mordred is behind on the sitch | 17:23 | |
| jeblair | oh, no i understand why $GRENADE_OLD_BRANCH is there, that makes sense. we should comment that though. :) | 17:24 |
| jeblair | https://review.openstack.org/#/c/82496/ approved | 17:24 |
| ArxCruz | jeblair: clarkb https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh#L177 | 17:26 |
| *** _nadya_ has quit IRC | 17:26 | |
| ArxCruz | on my ssystem, running Fedora, my vms have 2 cpus | 17:27 |
| ArxCruz | nprocs shows and assign 2 | 17:27 |
| ArxCruz | but it's entering in this if | 17:27 |
| *** Ryan_Lane has joined #openstack-infra | 17:27 | |
| ArxCruz | changing > 3 to -gt 3 works | 17:27 |
| jeblair | ArxCruz: bash is hard! :) | 17:27 |
| ArxCruz | jeblair: small patch is welcome? :) | 17:27 |
| clarkb | ArxCruz: yup I think that is a valid bug | 17:28 |
| ArxCruz | okay | 17:28 |
| fungi | this is where we've been getting our dns advice from, right? https://twitter.com/utku/status/446956710502993920/photo/1 | 17:28 |
| jeblair | ArxCruz: yes, but before you do that, you may want to consider whether the patch should fix that bug, or make it do what it does now and always /= 2 | 17:28 |
| ArxCruz | jeblair: my ci is failing because, although it has 2 cpus, it always dividing by 2 because it's entering there | 17:29 |
| jogo | clarkb: just a change to oslo-incubator and a change to nova | 17:29 |
| ArxCruz | jeblair: using -gt it doesn't because check 2 -gt 3 | 17:29 |
| jeblair | ArxCruz: i don't think we've ever run tempest on <4 cpus; that check was in there because we wanted to avoid overloading the host | 17:29 |
| jogo | have patches out for cinder and ceilometer to do the same | 17:29 |
| jeblair | ArxCruz: so i'm wondering if using both of your cpus would do that | 17:29 |
| clarkb | ArxCruz: jeblair: right theory was with all of the services running + mysql + rabbit, there wasn't enough cpu time with tempest running on each cpu | 17:30 |
| jeblair | ArxCruz: anyway, whatever you think best -- we don't have the experience with <= 2 vcpu machines | 17:30 |
| clarkb | giving tempest half the cpus allows openstack and mysql and rabbit and so on to use the cpus too | 17:30 |
| zaro | clarkb: it needs your expertise. i'm the only one that +2ed-it | 17:30 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for heat bug 1297560 https://review.openstack.org/82955 | 17:30 |
| uvirtbot | Launchpad bug 1297560 in heat "*tempest-dsvm-neutron-heat-slow fails with WaitConditionTimeout" [Undecided,New] https://launchpad.net/bugs/1297560 | 17:30 |
| ArxCruz | jeblair: understand, I can test it and report for you guys my experiences | 17:31 |
| ArxCruz | but the code is bugged anyway right ? | 17:31 |
| jeblair | clarkb: we might want to -=2 cpus rather than /=2 | 17:31 |
| zaro | Ok. i'm gonna drop my investigation to WIP plugin. | 17:31 |
| jeblair | ArxCruz: definitely; should be fixed one way or the other. :) | 17:31 |
| ArxCruz | okay, cool :) | 17:31 |
| *** nicedice has joined #openstack-infra | 17:31 | |
| jeblair | cpu median 39 mean 40 max 99 | 17:32 |
| jeblair | percentages according to dstat ^ | 17:32 |
| clarkb | jeblair: max(1, numcpus-2) or similar | 17:32 |
| jeblair | ++ | 17:32 |
| clarkb | sdague: mtreinish ^ how does that sound to you guys? | 17:33 |
| mordred | when did I become the only person who thinks that the presentation laptops at linux events should actually run linux/ | 17:34 |
| mordred | ? | 17:34 |
| mordred | I mean, especially when said events are run by the linux foundation | 17:34 |
| clarkb | mordred: I am with you on that | 17:34 |
| clarkb | lca was actually reasonably good about it | 17:34 |
| mordred | lca are good humans | 17:35 |
| pleia2 | mordred: sigh | 17:35 |
| zaro | can you dual boot macbook into linux? | 17:36 |
| pleia2 | yeah | 17:37 |
| zaro | hmm, worth investigating i guess. | 17:37 |
| pleia2 | I love my non-mac laptop that doesn't need a silly dongle to connect to projectors :) | 17:37 |
| zaro | i'm with you on that :) | 17:38 |
| mordred | pleia2: ++ | 17:38 |
| jeblair | pleia2: i never got that -- surely someone at apple would have said "hey guys, people try to give presentations with our computers all the time and no one ever has the right dongle so they all start half an hour late with apoligies for how their computer doesn't work with the projector" | 17:39 |
| jeblair | "that looks bad for us" | 17:39 |
| jeblair | but i guess that's why i don't work at apple | 17:39 |
| mtreinish | clarkb: sounds good to me | 17:40 |
| clarkb | mordred: is there a current plan for https://review.openstack.org/#/c/51425 ? I don't want to rereview it until we are committed to getting it through, otherwise it is just chrun | 17:40 |
| *** gyee has joined #openstack-infra | 17:40 | |
| Mithrandir | jeblair: well, vga connector is large | 17:40 |
| clarkb | jeblair: on the flip side their machines are art | 17:40 |
| mtreinish | clarkb: what about max(2, numcpus-2)? | 17:41 |
| *** gokrokve_ has quit IRC | 17:41 | |
| clarkb | mtreinish: will that work on single cpu or two cpu machines? | 17:41 |
| mtreinish | I guess there wouldn't be a real difference there | 17:41 |
| *** lcostantino has quit IRC | 17:41 | |
| *** sweston has joined #openstack-infra | 17:41 | |
| jeblair | clarkb: "instead of the presentation i was going to give, let's admire my computer for 40 minutes" | 17:41 |
| mordred | if I tweet the following: | 17:41 |
| mtreinish | I guess not but do people actually run with parallel in those cases? | 17:41 |
| clarkb | jeblair: exactly | 17:41 |
| clarkb | mtreinish: they do if running devstack gate | 17:42 |
| mordred | At Linux events, MAYBE someone should consider using Linux for the presentation laptops. #yolo | 17:42 |
| mordred | do you think the #yolo tag is too much? | 17:42 |
| mtreinish | clarkb: well it depends on the env variables. There are still a few serial tempest jobs | 17:42 |
| mtreinish | err or maybe only 2 | 17:42 |
| clarkb | mordred: you only live once badump tssh | 17:42 |
| mordred | yeah. that was how I was reading it | 17:43 |
| *** prad__ has joined #openstack-infra | 17:44 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/devstack-gate: Altering the > to -gt to check number of processors https://review.openstack.org/83141 | 17:44 |
| Mithrandir | zaro,clarkb : https://gist.github.com/tfheen/9789098 btw | 17:45 |
| Mithrandir | for the "don't overwrite job descs" | 17:45 |
| Mithrandir | I think that's a massively simpler fix | 17:45 |
| *** prad_ has quit IRC | 17:45 | |
| Mithrandir | oh, yeah, tests/yamlparser/fixtures/complete001.xml needs updating for a new default string too | 17:45 |
| mtreinish | clarkb: yeah, you're probably right to do max(1, numprocs-2) though because I doubt anyone would ever know to use those env variables | 17:47 |
| clarkb | Mithrandir: ++ that is a small diff, fwiw I am fine deferring on this one | 17:47 |
| mtreinish | clarkb: or just manually set the concurrency to 1 in those cases | 17:48 |
| Mithrandir | mine's very lightly tested, but I think that's so much simpler than the other hole that's being dug | 17:48 |
| clarkb | fungi: by the way I think I did the initial group seed for all of the new projects we added on Friday | 17:49 |
| clarkb | fungi: if any other bugs pop up to add users, double check but it should be done | 17:49 |
| mtreinish | jeblair: did you see this?: http://lists.openstack.org/pipermail/openstack-dev/2014-March/030945.html | 17:50 |
| fungi | clarkb: awesome--thanks | 17:50 |
| mtreinish | I checked salv-orlando was correct the duplicate entries in the layout are just making 2 links to the same job | 17:50 |
| clarkb | fungi: at the time there were no bugs open, but pretty clear indications of who the group members should be so I went ahead and did it | 17:50 |
| *** morganfainberg_Z is now known as morganfainberg | 17:50 | |
| fungi | clarkb: works for me | 17:51 |
| *** VijayTripathi has quit IRC | 17:51 | |
| clarkb | mtreinish: interesting | 17:51 |
| *** prad__ has quit IRC | 17:51 | |
| SlickNik | SergeyLukjanov / mordred / fungi: Thanks for the catch on https://review.openstack.org/#/c/83103. | 17:53 |
| clarkb | mordred: did you see my comment about https://review.openstack.org/#/c/51425 ? | 17:54 |
| SlickNik | The trove services were running in the gate correctly since the individual services are called out in the ENABLED_SERVICES list. | 17:54 |
| *** lcostantino has joined #openstack-infra | 17:54 | |
| *** prad_ has joined #openstack-infra | 17:55 | |
| *** khyati_ has joined #openstack-infra | 17:55 | |
| *** _nadya_ has joined #openstack-infra | 17:55 | |
| SlickNik | SergeyLukjanov / mordred / fungi: http://logs.openstack.org/06/80706/10/check/check-tempest-dsvm-full/f34b5b9/logs/testr_results.html.gz | 17:55 |
| *** signed8bit has joined #openstack-infra | 17:55 | |
| *** amcrn has joined #openstack-infra | 17:55 | |
| *** prad_ has quit IRC | 17:55 | |
| jeblair | mtreinish: no, thanks | 17:55 |
| SlickNik | SergeyLukjanov / mordred / fungi: Looks like the incorrect concatenation of "ceilometer-anotificationtrove" affected ceilometer tests though, and so gordon found the issue. | 17:56 |
| anteaya | zaro: is there any ability to set WIP from the CLi with gerrit 2.8? | 17:56 |
| *** saju_m has joined #openstack-infra | 17:57 | |
| zaro | anteaya: if we use a vote for WIP then for sure it's doable thru CLI. | 17:58 |
| anteaya | if I create an initial patchset and want to mark it WIP, where does the vote come in? | 17:59 |
| zaro | anteaya: you would use the ssh command to set 'workflow' label to -1 | 18:00 |
| *** bhuvan_ has joined #openstack-infra | 18:00 | |
| * anteaya looks up the workflow label | 18:00 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-dev/pbr: Add support for nested requirements files https://review.openstack.org/83149 | 18:00 |
| *** yolanda has joined #openstack-infra | 18:01 | |
| clarkb | jeblair: fungi other irc ops, do we see it as a problem that openstackgerrit has founder in #satori https://review.openstack.org/#/c/79058/ or should we ignore that for nwo and let the autmation clean it up? | 18:02 |
| zaro | anteaya: that's a custom label that we will define for WIP, it's not in gerrit docs. | 18:02 |
| *** luisg has joined #openstack-infra | 18:02 | |
| jeblair | clarkb: the automation will clean it up | 18:02 |
| SergeyLukjanov | jeblair, +1 | 18:02 |
| *** luisg has quit IRC | 18:02 | |
| zaro | anteaya: labels are referred to as votes on gerit UI | 18:02 |
| *** bhuvan has quit IRC | 18:03 | |
| clarkb | ok, I will approve that change then bump the irc automation reviews to the top of my list | 18:03 |
| jeblair | clarkb: cool | 18:03 |
| zaro | Mithrandir: how about push your solution to review.o.o? | 18:03 |
| jeblair | clarkb: oh, it's not going to clean it up unless the channel is added to the channel list | 18:03 |
| jeblair | for access bot | 18:03 |
| anteaya | zaro: browsing https://gerrit-documentation.storage.googleapis.com/Documentation/2.8/index.html so far I can't see labels/votes or workflow | 18:04 |
| *** derekh has quit IRC | 18:04 | |
| zaro | anteaya: https://gerrit-documentation.storage.googleapis.com/Documentation/2.8/config-labels.html | 18:04 |
| clarkb | jeblair: are we worried about channels granting too many permissions? | 18:04 |
| jeblair | clarkb: er, what i mean is that there is now an additional step for new channels | 18:05 |
| *** dripton is now known as dripton_afk | 18:05 | |
| jeblair | clarkb: the accessbot has its own channel config file, so all new channels need to be added to it | 18:05 |
| clarkb | jeblair: oh right, you mentioned it needed to be double accounted until we had something better | 18:06 |
| jeblair | clarkb: that should be considered a review criteria for irc related changes right now | 18:06 |
| jeblair | clarkb: yeah; i kinda want to wait to see what Kiall reports back on that new bot... | 18:06 |
| clarkb | ok, I will -1 and leave a comment that it iwll need to be double accounted | 18:06 |
| clarkb | the IRC stuff did merge then? I know I reviewed some of it but not all of it | 18:06 |
| *** VijayTripathi has joined #openstack-infra | 18:06 | |
| jeblair | clarkb: because one of the solutions to that problem is "use a single bot with multiple plugins for everything" which can obviously read a single config file | 18:07 |
| *** jcoufal has quit IRC | 18:07 | |
| jeblair | clarkb: if we don't do that, then it becomes "try to make a single config file that our multiple bots can read" or something like that | 18:07 |
| clarkb | roger | 18:07 |
| *** dripton_afk is now known as dripton | 18:07 | |
| jeblair | or one file that gets parsed out into multiple ones, etc. | 18:07 |
| pabelanger | jeblair, fungi: should jobs in jenkins have "Check if ZMQ events should be published for this project." enabled for ZMQ events? | 18:08 |
| jeblair | pabelanger: we enable it globally in the main jenkins config so they don't have to | 18:08 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Don't enable savanna service for sahara https://review.openstack.org/81058 | 18:08 |
| pabelanger | jeblair, Ah, okay. I think that is part of my issue too | 18:09 |
| jeblair | pabelanger: we're going to try to move that to zuul/gearman instead of zmq soon too, so that'll be a little simpler | 18:09 |
| *** Sukhdev has quit IRC | 18:10 | |
| clarkb | jeblair: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/accessbot/channels.yaml#n94 is the meetbots section of that config unused? | 18:10 |
| clarkb | jeblair: grepping for meetbots as a key in the accessbot source doesn't return anything | 18:10 |
| zaro | anteaya: maybe this will also help https://review.openstack.org/#/c/70818/10/doc/source/gerrit.rst | 18:11 |
| anteaya | zaro: I'm not quite envisioning why we would need to vote -1 for that, it looks like we can makeup our own labels | 18:11 |
| jeblair | clarkb: no, meetbots get +O | 18:11 |
| anteaya | so I would be inclined to do a wip label with a 0 vote | 18:11 |
| Mithrandir | zaro: doing so now | 18:11 |
| * anteaya clicks | 18:11 | |
| Mithrandir | zaro: after I added some more code to make it more likely to actually work. :-P | 18:12 |
| pabelanger | jeblair, okay, cool. Sorry for all the questions, still learning the ropes on nodepool :) | 18:12 |
| jeblair | pabelanger: no problem, the documentation is terrible. :) | 18:12 |
| clarkb | anteaya: 0 is the default, we don't want WIP by default | 18:12 |
| jeblair | pabelanger: half of it is still in review. :) | 18:12 |
| openstackgerrit | Tollef Fog Heen proposed a change to openstack-infra/jenkins-job-builder: Make it possible to override the "managed by" tag https://review.openstack.org/83157 | 18:12 |
| clarkb | WIP is a thing you must explicitly change your state into | 18:12 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: DB api refactoring https://review.openstack.org/83027 | 18:12 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Fix 009 migration https://review.openstack.org/83050 | 18:12 |
| Mithrandir | zaro: ^ | 18:13 |
| Mithrandir | (83157) | 18:13 |
| anteaya | clarkb: I was going with the believe that using the label would convey that, if we made up our own wip label | 18:13 |
| anteaya | because apparently we can make up our won | 18:14 |
| anteaya | own | 18:14 |
| clarkb | anteaya: right we can make up our own labels | 18:14 |
| *** dstanek is now known as dstanek_afk | 18:14 | |
| anteaya | cool | 18:14 |
| clarkb | anteaya: approved and verified are non standard labels | 18:14 |
| clarkb | but each label comes with a default state: 0 | 18:14 |
| pabelanger | jeblair, ya, going to review it too | 18:14 |
| clarkb | so adding a different label doesn't solve the problem of still needing to vote to change the state | 18:14 |
| anteaya | oh | 18:14 |
| anteaya | the vote changes it from the default state to register the change in the label | 18:15 |
| clarkb | yes, voting workflow -1 says make this thing WIP | 18:15 |
| clarkb | you could have a WIP label but you would still have to vote on it to make the state change | 18:15 |
| anteaya | ah okay | 18:15 |
| *** dizquierdo has quit IRC | 18:16 | |
| anteaya | the fact that stream events is under permissions is going to be a change | 18:17 |
| anteaya | more 3rd party fun | 18:17 |
| clarkb | jeblair: I see there is a meetbots mask but not sure how that maps to the per channel config yet | 18:18 |
| clarkb | anteaya: we will probably just open it to registered users to preserve access to the feature | 18:18 |
| zaro | anteaya: yes, my change to 70818 documents that. | 18:18 |
| jeblair | clarkb: it's not a mask, it's an acl | 18:19 |
| * zaro afk for a few hrs. | 18:19 | |
| jeblair | clarkb: the acl lists are masks only if they are used with "mask:" in the channel config, otherwise they are absolute values | 18:20 |
| jeblair | clarkb: so operators get _exactly_ "+Afortv", and meetbots get exactly "+O" | 18:20 |
| *** mrodden has joined #openstack-infra | 18:20 | |
| clarkb | jeblair: roger, and I assume open_stack is the actually registered name for the openstack meetbot? | 18:20 |
| * clarkb pokes nickserv to find out | 18:21 | |
| jeblair | clarkb: but everybody else gets masked to "+AVOfortv" because that's the value of channel_op_mask which is set as the global mask | 18:21 |
| jeblair | clarkb: yes (open_stack) | 18:21 |
| *** dripton is now known as dripton_afk | 18:21 | |
| openstackgerrit | A change was merged to openstack/requirements: Remove oslo.sphinx from global requirements https://review.openstack.org/75371 | 18:22 |
| clarkb | yup confirmed with nickserv thanks | 18:22 |
| anteaya | clarkb: cool | 18:23 |
| anteaya | zaro: k, going to read through 70818 | 18:23 |
| *** thuc has quit IRC | 18:23 | |
| *** prad_ has joined #openstack-infra | 18:23 | |
| * Mithrandir sighs at python2.6 | 18:25 | |
| *** jgallard has quit IRC | 18:25 | |
| *** mriedem has joined #openstack-infra | 18:26 | |
| *** pcrews has quit IRC | 18:27 | |
| * clarkb completely changes the subject. Assuming we get working glance in our clouds and stop using hpcloud 1.0, we could do the crazy thing of switching to centos 7 beta for testing icehouse. But as I bring it up it seems like a silly thing to do | 18:28 | |
| clarkb | Mithrandir's comment sort of triggered the idea since we can drop python 26 when rhel7 happens | 18:28 |
| clarkb | but it hasn't happened yet | 18:28 |
| jeblair | clarkb: we can drop python2.6 when we stop supporting the last version of openstack that was released with python2.6 support | 18:29 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1297992 https://review.openstack.org/83162 | 18:29 |
| uvirtbot | Launchpad bug 1297992 in nova "deleting an interface hits a race condition and fails" [Undecided,New] https://launchpad.net/bugs/1297992 | 18:29 |
| jeblair | that's not exactly when rhel7 is released... | 18:29 |
| clarkb | jeblair: well, I meant for icehouse | 18:30 |
| *** yolanda has quit IRC | 18:30 | |
| clarkb | older versions will continue to be tested against python 2.6 | 18:30 |
| *** thuc has joined #openstack-infra | 18:30 | |
| jeblair | clarkb: i'm pretty sure icehouse will be released with support for python 2.6 | 18:30 |
| clarkb | jeblair: me too | 18:30 |
| *** e0ne has joined #openstack-infra | 18:30 | |
| *** pcrews has joined #openstack-infra | 18:30 | |
| jeblair | but hopefully we can remove it in juno | 18:31 |
| clarkb | since rhel7 doesn't appear to be happening by april | 18:31 |
| *** prad_ has quit IRC | 18:31 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: fix trove merge https://review.openstack.org/83103 | 18:32 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Use OVERRIDE_ZUUL_BRANCH in devstack-vm-gate.sh https://review.openstack.org/82496 | 18:32 |
| fungi | also, as discussed not long ago, our python 3.3 jobs should likely get replaced by python 3.4 jobs since that'll be the default py3k implementation in ubuntu 14.04 lts | 18:32 |
| fungi | right now we don't have anything which is part of the integrated release supporting 3.3 afaik, so ought to be able to do that without worrying about deprecation | 18:33 |
| jeblair | fungi: ++ | 18:33 |
| clarkb | fungi: the only possible gotcha is whether or not we want to support clients on 3.3, but considering ubuntu lts will be 3.4 I am not worried much about 3.3 | 18:33 |
| *** jpich has quit IRC | 18:33 | |
| *** prad_ has joined #openstack-infra | 18:34 | |
| fungi | clarkb: i also think the delta between 3.3 and 3.4 is smallish enough that we'd be unlikely to merge many changes which work with 3.4 and not 3.3 (though no doubt there may be some) | 18:34 |
| clarkb | fungi: agreed, I don';t think it is like the difference between 3.2 and 3.3 | 18:35 |
| dhellmann | has anyone else reported issues with missing /opt/stack/new/devstack on the neutron-large-ops job? http://logs.openstack.org/56/82356/2/check/gate-tempest-dsvm-neutron-large-ops/1ca16bc/console.html | 18:35 |
| jeblair | mtreinish, sdague: there is indeed a problem with the 'duplicate job' feature in zuul; we _do_ run multiple jobs, but zuul is only considering one of them when it reports, or even decides whether the change has failed | 18:36 |
| *** dstanek_afk has quit IRC | 18:36 | |
| jeblair | mtreinish, sdague: fixing that is non-trivial and makes me a little concerned about the complexity needed to do it correctly... | 18:37 |
| openstackgerrit | Tollef Fog Heen proposed a change to openstack-infra/jenkins-job-builder: Make it possible to override the "managed by" tag https://review.openstack.org/83157 | 18:37 |
| *** _sweston_ has joined #openstack-infra | 18:37 | |
| jeblair | mtreinish, sdague: how bad would it be to say that in this case, we need to instantiate the job multiple times in jjb? (so neutron-pg-1, neutron-pg-2, etc?) | 18:37 |
| jeblair | jogo: ^ | 18:37 |
| *** sweston has quit IRC | 18:38 | |
| *** prad_ has quit IRC | 18:38 | |
| jeblair | that might make analysis a little harder (like in er, etc) | 18:38 |
| anteaya | dhellmann: I have not seen that error before, no | 18:38 |
| clarkb | jeblair: it may make some of the rules a little more verbose, but hopefully not terribly so. and creating those jobs should be simple in JJB | 18:38 |
| dhellmann | thanks, anteaya, I'm not finding it in the list of rechecks | 18:39 |
| clarkb | jeblair: I think it is a decent plan of attack | 18:39 |
| anteaya | dhellmann: you are the first, congratulations | 18:39 |
| fungi | dhellmann: the workspace setup log for that job looks heinous too | 18:39 |
| * dhellmann doesn't feel lucky | 18:39 | |
| jeblair | dhellmann: | 18:39 |
| jeblair | 2014-03-26 17:02:46.573 | error: Failed to connect to 2001:4800:7813:516:3bc3:d7f6:ff04:aacb: Network is unreachable while accessing https://git.openstack.org/openstack/swift/info/refs | 18:39 |
| mtreinish | jeblair: I'm fine with using different jobs names | 18:39 |
| mtreinish | I'll write up a patch to do that now | 18:40 |
| clarkb | sdague: mtreinish mriedem btw https://review.openstack.org/#/c/79447/ merged on the 13th, we can merge a change to e-r to use all of the new goodness on Friday without data loss | 18:40 |
| dhellmann | jeblair: ah, I see that now | 18:40 |
| fungi | jeblair: looking at the console log there was no ipv6 global address/default route reported either when the job started | 18:40 |
| anteaya | could this be more git.o.o derp like I experienced earlier? | 18:40 |
| fungi | jeblair: which would be correct for hpcloud | 18:40 |
| clarkb | fungi: I think this is that supposed git bug which I haven't been able to properly track down | 18:41 |
| fungi | so that's got to be that git fallback behavior | 18:41 |
| jogo | jeblair: if we have only a few copies of a job it won't be so bad | 18:41 |
| fungi | clarkb: yeah | 18:41 |
| clarkb | fungi: lifeless and spamaps see it occasionally too | 18:41 |
| jogo | but if we have a lot it may be confusing | 18:41 |
| fungi | which suggests that maybe it had trouble reaching the ipv4 address it looked up for git.o.o first (or failed to look one up?) | 18:41 |
| jeblair | fascinating | 18:42 |
| fungi | cute... https://mail-index.netbsd.org/netbsd-users/2012/08/18/msg011301.html | 18:43 |
| clarkb | sdague: mtreinish mriedem I think I will take a stab at using the short_uuid today | 18:43 |
| clarkb | I feel like I need to write some code and that will be a good outlet | 18:43 |
| jeblair | dtroyer: oh hey, are you going to propose python-openstackclient as an official program? | 18:43 |
| mriedem | clarkb: ok, go nuts :) | 18:44 |
| mriedem | i've been in a class all morning so have no idea what's going on today | 18:44 |
| lifeless | fungi: I like your NS issue theory | 18:44 |
| dhellmann | fungi: should I open a bug against openstack-ci? I need to reverify my changeset anyway, but it's not clear this is a "bug" per se | 18:44 |
| fungi | dhellmann: sure, we can just mark it incomplete until someone is able to put together a more reproducible test case | 18:45 |
| dhellmann | ok | 18:45 |
| mordred | jeblair: I would so much prefer if dtroyer made python-openstackclient a thing | 18:45 |
| dtroyer | jeblair: I thought the SDK might have been further along by now, but maybe we should start with OSC? I think ultimately the client-side stuff should be close, if not together. | 18:45 |
| mordred | there's also the python-openstacksdk thing - perhaps teh two of them together are a thing? | 18:45 |
| mordred | dtroyer: jinx | 18:46 |
| dtroyer | mmrmmphmmrrmhh | 18:46 |
| dtroyer | so is me being the ptl of DevStack an issue to doing an OSC program? | 18:47 |
| mordred | dtroyer: I think python-openstackclient is something the project desperately needs - and I kinda thought most of the folks in the project agreed | 18:47 |
| lifeless | dtroyer: do you have the bandwidth, do you mean? | 18:47 |
| mordred | dtroyer: I don't think so - we have no rules about single PTL-ness | 18:47 |
| jeblair | i don't think we have limits on the number of ptl seats one can hold | 18:47 |
| dtroyer | no, is there something about not having one person wear more than one ptl hat? | 18:47 |
| dtroyer | ok | 18:47 |
| dhellmann | fungi: https://bugs.launchpad.net/openstack-ci/+bug/1298006 | 18:48 |
| uvirtbot | Launchpad bug 1298006 in openstack-ci "network sometimes unreachable accessing our git server" [Undecided,New] | 18:48 |
| fungi | dhellmann: thanks | 18:48 |
| jeblair | clarkb, mordred: should we open a ticket with hpcloud about the large number of servers that took more than 1 hour to launch? | 18:50 |
| jeblair | clarkb, mordred: or should we merge the nodepool logging change, restart it, bump up our regionb usage and get more data? | 18:50 |
| jeblair | and then open a ticket :) | 18:50 |
| mordred | we might want to do the second thing | 18:50 |
| anteaya | dtroyer: nothing in the election rules I have seen | 18:51 |
| fungi | i'm fine with making a second go of it | 18:51 |
| dhellmann | mordred: heads up, I'm going to do another pbr release when https://review.openstack.org/#/c/82356/ lands | 18:51 |
| fungi | jeblair: you did see the interesting miscalculation nodepool seems to have made after we switched back (in scrollback from before you were awake)? | 18:51 |
| mordred | dhellmann: ok | 18:51 |
| jeblair | fungi: not in detail | 18:52 |
| *** zhiyan is now known as zhiyan_ | 18:52 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs https://review.openstack.org/83172 | 18:52 |
| fungi | jeblair: current theory is that last night when we switched to region b and ended up starved for py3k nodes because nodepool kept asking region b to make some and then they'd eventually accumulate in building state for too long, the pileup of waiting pypy/python33 jobs caused nodepool to think it needed many more of that node type to service them when we switched back | 18:53 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs https://review.openstack.org/83172 | 18:53 |
| mtreinish | jeblair: ^^^ I think that should do it correctly | 18:54 |
| clarkb | mordred: jeblair ++ to the second thing | 18:54 |
| fungi | jeblair: so it made *a lot* of precise-py3k nodes, which were then hanging around in a ready state when i woke up this morning, and weren't going through them nearly fast enough to turn them over into other node types we actually needed | 18:54 |
| *** jamielennox is now known as jamielennox|away | 18:54 | |
| jeblair | fungi: ah. | 18:56 |
| fungi | jeblair: so working as designed, but something we should watch out for | 18:57 |
| fungi | jeblair: and maybe a wishlist feature for nodepool is to delete ready nodes it realizes it no longer needs so many of, so it can recycle their slots for node types which are actually in demand | 18:57 |
| jeblair | ++ | 18:57 |
| *** johnthetubaguy has quit IRC | 19:00 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: be extra explicit for people reading logs https://review.openstack.org/83071 | 19:00 |
| fungi | though the other problem, dealing with rare node starvation due to node builds taking too long (and building state counting toward the available number for a node type), is something i don't currently have good ideas for how we'd address | 19:01 |
| clarkb | mtreinish: mriedem sdague are the bug-number.json files currently in use by the unittests? any idea how those are generated so that I can update them with current field data? | 19:02 |
| clarkb | ah there they are in the tests so they are used | 19:03 |
| mtreinish | clarkb: one of the functional tests is supposed to read them in | 19:03 |
| clarkb | I know I can write a small python script that loads each one and adds the appropriate fields | 19:03 |
| mtreinish | but I think it's broken | 19:03 |
| *** thuc has quit IRC | 19:05 | |
| *** thedodd has quit IRC | 19:05 | |
| clarkb | mtreinish: I will do my best to make sure the json is correct then try to figure out if those tests need help | 19:05 |
| mordred | fungi: I was thinking this morning - maybe we don't need the precise3k nodes anymore - or the system-wide python3 stuff installed | 19:05 |
| mordred | since we made the tox.ini changes to run things with teh python in the virtualenv | 19:06 |
| mordred | we may have solved the underlying problem differently | 19:06 |
| fungi | mordred: well, we still need the interpreter | 19:06 |
| clarkb | mtreinish: actually that json is really really old | 19:06 |
| clarkb | mtreinish: we should maybe redo that completely and use current data with current bugs /me ignores it for now | 19:06 |
| fungi | mordred: i think when trusty comes around, then yes | 19:07 |
| mordred | fungi: yah - but I thnk ALL we need is the interpreter | 19:07 |
| fungi | mordred: but right now the py3k-precise nodes are preinstalled with ppa-provided python 3.3 and pypy interpreter backports | 19:07 |
| *** ominakov has quit IRC | 19:07 | |
| fungi | mordred: though yes, i agree, we drop py3k-precise as a node type and use bare-trusty to run those as soon as we can | 19:08 |
| mordred | fungi: sure - but I think there was something else we did on them because of the old tox bug that made doign them side by side with other nodes tricky | 19:08 |
| *** markmcclain has joined #openstack-infra | 19:08 | |
| *** mgagne1 is now known as mgagne | 19:09 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs https://review.openstack.org/83172 | 19:09 |
| fungi | mordred: right, we were doing puppet gyrations to get the pip provider to install tox and virtualenv into the module search path for python3 but since newer versions are bundling and our entry point for tox should work when called under python2.7 to run py33 (or py34) and pypy tox envs, shouldn't matter any longer | 19:09 |
| mordred | yah | 19:10 |
| fungi | mordred: something we certainly want to test to make sure, but we'll be doing that as part of teh trusty transition anyway | 19:10 |
| mordred | ++ just wanted to make sure I wasn't missing something | 19:11 |
| fungi | so makes the most sense to stop using a separate node type for those jobs at that point | 19:11 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Revert "Allow multiple invocations of the same job" https://review.openstack.org/83181 | 19:13 |
| jeblair | mtreinish, sdague, jogo: ^ | 19:14 |
| *** wchrisj has joined #openstack-infra | 19:14 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Take advantage of the new build_short_uuid field. https://review.openstack.org/83182 | 19:16 |
| *** malini has joined #openstack-infra | 19:17 | |
| clarkb | jogo: mtreinish sdague ^ I haven't actually tested that, but was hoping for feedback on the variable renames for consistency and more discussion about the test json data | 19:17 |
| marun | https://review.openstack.org/#/c/82226/ < ready for review ;) | 19:20 |
| marun | job is experimental only so I can test it with | 19:20 |
| marun | https://review.openstack.org/#/c/72585/ | 19:20 |
| *** jp_at_hp has quit IRC | 19:20 | |
| clarkb | jeblair: fungi fyi https://review.openstack.org/#/c/82959/ failed pep8 | 19:21 |
| jeblair | clarkb: do you inderstand what happened there? | 19:22 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/meetbot: Prevents meetbot from starting meetings in non-meeting channels https://review.openstack.org/82898 | 19:22 |
| clarkb | jeblair: no, looking at the log file it almost apepars like we used a different pep8 or something | 19:22 |
| fungi | clarkb: yeah, it's complaining about style differences in files which weren't touched by that change | 19:23 |
| jeblair | clarkb: a new pep8 was released today, but i thought it comes from pep8 | 19:23 |
| jeblair | er flake8 | 19:23 |
| jeblair | does flake8 not actually pin the pep8 version? | 19:23 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Revert "Allow multiple invocations of the same job" https://review.openstack.org/83181 | 19:23 |
| clarkb | jeblair: it may not, I am working to reproduce locally really quickly | 19:24 |
| clarkb | jeblair: we pin flake8 to 2.0 | 19:24 |
| clarkb | but if flake8 doesn't pin pep8 then that may be the issue | 19:24 |
| clarkb | yup reproduced with tox -repep8 | 19:25 |
| jeblair | "pep8 >= 1.4.3", | 19:25 |
| jeblair | is in setup.py of flake8 2.0 | 19:25 |
| clarkb | pep8==1.5.0 was used according to pip freeze | 19:26 |
| fungi | that's today's release | 19:26 |
| jeblair | so basically, once again, all of the style checks for all of the projects just now changed all at once because pep8 did a release? | 19:26 |
| *** krotscheck has quit IRC | 19:26 | |
| clarkb | jeblair: no, all of the other projects use hacking | 19:27 |
| anteaya | mordred: can you take a look at https://review.openstack.org/#/c/83116/ please? | 19:27 |
| *** markwash has quit IRC | 19:27 | |
| clarkb | and hacking pins flake8 and pep8 | 19:27 |
| *** thuc has joined #openstack-infra | 19:27 | |
| anteaya | mordred: seems gerrit is happier with an --empty-commit flag when creating empty repos | 19:27 |
| jeblair | clarkb: oh, it pins flake8 _and_ pep8 | 19:27 |
| jeblair | clarkb: i thought it was transitive | 19:27 |
| clarkb | nodepool does not use hacking and thus ran into this, I will add an explicit pep8 pin to nodepool | 19:27 |
| dstufft | you mean you don't want your style to change on someone elses whim? | 19:27 |
| fungi | looks that way... http://git.openstack.org/cgit/openstack-dev/hacking/tree/requirements.txt#n3 | 19:27 |
| *** thuc_ has joined #openstack-infra | 19:28 | |
| jeblair | (someone told me that and i believed them but i shouldn't have) | 19:28 |
| fungi | pins pyflakes too, presumably for similar reasons | 19:28 |
| clarkb | oh right, I should pin pyflakes in nodepool too | 19:28 |
| fungi | yep | 19:29 |
| *** pcrews has quit IRC | 19:29 | |
| sdague | clarkb: sorry, catching up on scrollback. The json files were just things I snagged over time. I don't know if I've got the generator for them (probably should) | 19:30 |
| *** SumitNaiksatam has quit IRC | 19:31 | |
| *** thuc has quit IRC | 19:32 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Pin flake8 deps so they don't change on us. https://review.openstack.org/83187 | 19:32 |
| clarkb | jeblair: fungi ^ | 19:32 |
| fungi | clarkb: any reason not to pin to the same revs used by hacking? | 19:33 |
| sdague | jeblair: so the n / 2 vs. n - 2 thing, I don't know that we have any data yet on 6 way tempest vs. failure results | 19:34 |
| *** andreaf has quit IRC | 19:34 | |
| fungi | clarkb: not saying we necessarily should, just wondering if pinning to the current latest working vs what hacking already settled on makes more/less sense | 19:34 |
| sdague | I actually think before we do another change like that we should try to get the "master queue" running so we can run some numbers against master regularly and see | 19:34 |
| jeblair | fungi, clarkb: i think pinning to hacking versions makes sense | 19:36 |
| clarkb | we don't use hacking | 19:36 |
| jeblair | clarkb: yes, but i think it's good to be in sync with the pep8 and pyflakes versions used in openstack | 19:36 |
| clarkb | ok, I can update the change | 19:36 |
| jeblair | clarkb: (that's why i'm also okay with depending on hacking but ignore H errors, but apparently that confuses people) | 19:37 |
| *** bhuvan_ has quit IRC | 19:37 | |
| *** pcrews has joined #openstack-infra | 19:38 | |
| *** bhuvan has joined #openstack-infra | 19:39 | |
| *** bhuvan has joined #openstack-infra | 19:39 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Depend on hacking for its dependencies. https://review.openstack.org/83187 | 19:40 |
| clarkb | fungi: jeblair ^ that includes a fix for the one line that hacking's deps failed on | 19:40 |
| jeblair | sdague: btw, heads up, when hacking updates to pep8 1.5.0, you and i are going to want to ignore E129 everywhere: https://github.com/jcrocholl/pep8/issues/126#issuecomment-38517943 | 19:40 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add specs-cookiecutter to openstack-dev https://review.openstack.org/83192 | 19:41 |
| jeblair | clarkb: what rule was that? | 19:41 |
| clarkb | jeblair: nodepool/nodedb.py:258:13: E122 continuation line missing indentation or outdented | 19:42 |
| jeblair | ok | 19:42 |
| *** dripton_afk is now known as dripton | 19:42 | |
| fungi | i do agree with that additional indentation, fwiw. it's inside deeper nested parens than the preceding lines so helps readability | 19:42 |
| jogo | jeblair: we pin flake8 | 19:42 |
| harlowja | hey guys, qq, there was talk about moving away from the blueprint model, but i forgot what the suggestions where to move to instead (correct me if i am wrong) | 19:42 |
| *** SumitNaiksatam has joined #openstack-infra | 19:42 | |
| *** sabari has quit IRC | 19:43 | |
| jeblair | jogo: yeah, and pep8 too. i'm caught up now | 19:43 |
| dstufft | anyone mind me bothering them openstack's CI system for a few? :] | 19:43 |
| jeblair | harlowja: not so much the model as implementation | 19:43 |
| jogo | jeblair: yeah just saw finished reading the backlog | 19:43 |
| *** yolanda has joined #openstack-infra | 19:43 | |
| harlowja | jeblair ah, a different impl? | 19:43 |
| *** freyes has quit IRC | 19:43 | |
| clarkb | dstufft: hrm? I am happy to answer questions if that is what you mean | 19:43 |
| jeblair | harlowja: some projects are trying out the idea of writing blueprints in a git repo and putting them through code review in gerrit (nova-specs and qa-specs) | 19:44 |
| fungi | harlowja: being more explicit about where and how to do the design documentation | 19:44 |
| jeblair | harlowja: but they still have blueprints in launchpad for reference and tracking purposes | 19:44 |
| harlowja | hmmm, right, thought i remembered some talk of moving to something entirely different | 19:44 |
| *** freyes has joined #openstack-infra | 19:44 | |
| jeblair | harlowja: take a look at the README's in those repos and you'll get a better idea | 19:44 |
| jeblair | harlowja: oh, you may be thinking of storyboard | 19:44 |
| dstufft | clarkb: I'm thinking about pip/virtualenv/setuptools, and making a more expansive CI and testing infra for it (Right now it just uses travis ci) I'm trying to figure out what zuul does ontop of jenkins? :/ | 19:45 |
| harlowja | hmmm, could be | 19:45 |
| jeblair | harlowja: it's a unified bug/feature tracker we're developing | 19:45 |
| harlowja | ah, ya, that could be it | 19:45 |
| jeblair | harlowja: super early days, but it's in CD at https://storyboard.openstack.org/ | 19:45 |
| harlowja | coolness | 19:45 |
| jeblair | harlowja: git repo is openstack-infra/storyboard | 19:46 |
| fungi | dstufft: the documentation at http://ci.openstack.org/zuul/ and presentation at http://docs.openstack.org/infra/publications/zuul/ are great introductions | 19:46 |
| jeblair | (and storyboard-webclient) | 19:46 |
| clarkb | dstufft: two major things. The first is straightforward, zuul schedules jobs on gearman queues so you can have arbitrary job runners, including jenkins (this allows us to do multimaster jenkins) | 19:46 |
| clarkb | the second is a bit more complicated and is it allows us to gate | 19:47 |
| dstufft | clarkb: would it be the thing that lets me run pip tests if setuptools makes a change? | 19:48 |
| jeblair | dstufft: also, i think http://legacy.python.org/dev/peps/pep-0462/ may be a topic at the language summit, and i'll be around pycon for chatting, etc | 19:48 |
| * dstufft won't be at PyCon | 19:48 | |
| sdague | jeblair: ah the E125 wars :) | 19:48 |
| clarkb | dstufft: yes, if you create a shared integration test in a gate pipeline for setuptools and pip you can have them test against each other when changes are made to them | 19:48 |
| jeblair | dstufft: that's terrible, i don't see the point of even having a pycon without you! | 19:49 |
| pabelanger | jeblair, how did you enable zmq events globally in jenkins? I don't see an option to do so in manage jenkins. | 19:49 |
| clarkb | pabelanger: its in your global jenkins config if you have the zmq event publisher plugin installed | 19:50 |
| clarkb | pabelanger: there is a setting for the port to publish on and a check box to do it for all jobs | 19:50 |
| pabelanger | clarkb, I see host, port and enable gearman. All are filled in a test connection works. Is that all that is needed? | 19:51 |
| pabelanger | wait | 19:51 |
| SpamapS | clarkb: FYI I have not seen ipv6 fail for git in a long time | 19:51 |
| pabelanger | that is gearman plugin | 19:51 |
| pabelanger | okay | 19:51 |
| pabelanger | I see it | 19:51 |
| dstufft | clarkb: more or less I want a few things 1) I want to run the test suites of dependent projects (pip depends on setuptools and distlib and wheel atm) 2) I want to do this whenever someoone makes a PR on Github and report status 3) I want to be able to test on "wierd" OSes (Windows, AIX, etc) | 19:51 |
| dstufft | jeblair: :] | 19:52 |
| *** dstanek_afk has joined #openstack-infra | 19:52 | |
| clarkb | dstufft: zuul can't do that today but only because it does't speak github. If it was made to udnerstand github it could do all of that | 19:52 |
| jeblair | i would accept a patch for that | 19:52 |
| *** hashar has joined #openstack-infra | 19:52 | |
| *** dstanek_afk is now known as dstanek | 19:53 | |
| dstufft | I could write a patch for that | 19:53 |
| dstufft | would zuul be a better thing to look at than just using jenkins or buildbot or something? | 19:54 |
| *** skraynev_afk is now known as skraynev | 19:54 | |
| dstufft | Not going to lie, my 'setup a CI" experience ths far is getting mad at jenkins and buildbot and using travis-ci | 19:54 |
| clarkb | dstufft: for that I think you need to decide if you need gating or if you need a better scaling story than jenkins or travis allow | 19:54 |
| clarkb | (I haven't used buildbot so am not super familiar with its ability to talk to a hunddred slaves) | 19:55 |
| dstufft | clarkb: gating just means checking a patch prior to it landing right? or is it more involved? | 19:56 |
| clarkb | dstufft: that is the key bit but it is a bit more involved. zuul does all of that checking in parallel by building the desired end state and testing that for each change instead of waiting for things to go through one at a time | 19:56 |
| clarkb | dstufft: so say setuptools and pip have 2 PR each. and the code reviews on github give the go ahead to gate at about the same time. Zuul will construct a tree for that scenario (in the actual order it received the events) and test each PR in that scenario | 19:57 |
| clarkb | dstufft: for example setuptoolsA <- pipB <- pipC <- setuptoolsD. That last PR (setuptoolsD) will be tested with A, B, C as well | 19:58 |
| fungi | mainly because one pr might contain changes which don't merge-conflict with the other but manage to break functionality of the program when combined | 19:58 |
| *** bhuvan has quit IRC | 19:58 | |
| clarkb | dstufft: and if B fails for some reason C and D will have their tests restarted to exclude B which means A, B get tested, and A, C, D get tested | 19:58 |
| *** jergerber has joined #openstack-infra | 19:58 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Depend on hacking for its dependencies. https://review.openstack.org/83187 | 19:59 |
| fungi | dstufft: if you look at http://status.openstack.org/zuul/ you can see it in action (currently integration-testing changes for python-keystoneclient and swift together) | 20:00 |
| *** khyati_ has quit IRC | 20:00 | |
| *** mrodden has quit IRC | 20:01 | |
| *** mrda_away is now known as mrda | 20:01 | |
| dstufft | clarkb: so if we pretend for a minute that pip/setuptools/etc don't get near as many PRs at one time as openstack does (having more than one at a time is rare tbh) probably that bit isn't super important I'd guess? | 20:01 |
| mordred | dstufft: if you wrote the github trigger for zuul, then we could do the "Run openstack tests on changes to setuptools" | 20:01 |
| * mordred has been meaning to do that for about 6 months | 20:02 | |
| *** sabari has joined #openstack-infra | 20:02 | |
| dstufft | mordred: oh that'd be cool | 20:03 |
| clarkb | dstufft: ya it probably isn't super important | 20:03 |
| *** sabari has left #openstack-infra | 20:03 | |
| *** julim_ has quit IRC | 20:03 | |
| dstufft | that's another thing i'd like to figure out how to do TBH, is run tests against popular projects to make sure things still install (to prevent stuff like setuptools deprecating Feature to be a big surprise that lots of people ended up using it) | 20:04 |
| *** bhuvan has joined #openstack-infra | 20:04 | |
| dstufft | ah, I have to run to pick up my daughter from the bus stop, be back in a bit :) Thanks y'all :) | 20:05 |
| clarkb | dstufft: the other thing that comes with gating is the actual tests must pass before code merges rules which you may not get out of other tools either without writing code | 20:06 |
| clarkb | gating would still be useful with less PR volume in that way | 20:06 |
| *** e0ne has quit IRC | 20:07 | |
| *** ildikov_ has quit IRC | 20:08 | |
| *** thedodd has joined #openstack-infra | 20:08 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1291677 https://review.openstack.org/83162 | 20:10 |
| *** bauzas has joined #openstack-infra | 20:10 | |
| uvirtbot | Launchpad bug 1291677 in nova "delete interface times out in check-tempest-dsvm-neutron" [Undecided,New] https://launchpad.net/bugs/1291677 | 20:10 |
| mordred | yup. and as I understand it, github has an api command for "merge this PR" - so you could almost certainly write drivers for most of the peces of zuul | 20:10 |
| clarkb | jogo: can you do a quick braindump for me on how you run e-r locally? I can do that locally then write docs for it | 20:11 |
| clarkb | jogo: mostly wondering if you run the bot with it joining a private channel? | 20:11 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add specs-cookiecutter to openstack-dev https://review.openstack.org/83192 | 20:11 |
| clarkb | jogo: or is there some way to have it report to the console and not do irc? | 20:11 |
| jogo | clarkb: I like how build_short_uuid cauased so many changes | 20:11 |
| *** saju_m has quit IRC | 20:11 | |
| jogo | https://review.openstack.org/#/c/83182/1/elastic_recheck/query_builder.py | 20:12 |
| clarkb | jogo: yup we had two different names across tools... | 20:12 |
| jogo | clarkb: so we have a mode to run it locally without commenting anywhere | 20:12 |
| clarkb | jogo: unfortunately that didn't get caught in review of the lgostash/es change | 20:12 |
| jogo | clarkb: tox -erun | 20:12 |
| clarkb | jogo: does that require I copy the sample config and point it at valid gerrit keys? | 20:12 |
| jogo | you just need to createa elasticRecheck.conf file | 20:12 |
| clarkb | roger thanks | 20:13 |
| jogo | clarkb: yup | 20:13 |
| mtreinish | mordred: https://review.openstack.org/83192 I'm not sure we want to add this yet, but I figured I'd throw a review up for it anyway | 20:13 |
| jogo | clarkb: also control-C doesn't kill it for some reason | 20:13 |
| jogo | so I usually just do a 'kill' | 20:13 |
| *** marktraceur is now known as ba | 20:13 | |
| clarkb | jogo: probably a non daemon thread hanging around | 20:14 |
| jeblair | jogo: ctrl-\ should do it in those situations | 20:14 |
| jogo | jeblair: oh neato didn't know that one | 20:14 |
| jeblair | (though making threads daemon is probably a good idea) | 20:14 |
| *** ba is now known as marktraceur | 20:14 | |
| jogo | clarkb: your testing out https://review.openstack.org/#/c/83182 ? | 20:15 |
| clarkb | jogo: yup | 20:15 |
| *** yassine_ has quit IRC | 20:18 | |
| *** krotscheck has joined #openstack-infra | 20:18 | |
| *** malini is now known as malini_afk | 20:20 | |
| clarkb | jogo: woot its running, now I wait | 20:22 |
| fungi | "HEAD is now at e2fff8c Set paramiko version > 1.9.0" http://logs.openstack.org/59/82959/1/check/gate-nodepool-pep8/418adf1/console.html#_2014-03-26_20_12_35_499 | 20:22 |
| *** yassine has quit IRC | 20:23 | |
| fungi | except HEAD was already something else a good 15 minutes prior to that | 20:23 |
| fungi | i wonder whether we've got replication lagging again? | 20:23 |
| pabelanger | clarkb, when do zmq events get published for a job? Once it is created in jenkins?\ | 20:24 |
| fungi | that run also doesn't seem to have used the pin change | 20:24 |
| pabelanger | or only when jenkins launches the job | 20:24 |
| fungi | even though in theory it shouls have been included in the zuul ref | 20:24 |
| clarkb | pabelanger: on job start, job completion, and job finalization | 20:24 |
| clarkb | pabelanger: the socket won't be opened until the first job runs so you may lose the onStarted event for the first job but all other events shoudl show up | 20:25 |
| *** ominakov has joined #openstack-infra | 20:26 | |
| clarkb | fungi: why would it have been included in the zuul ref? | 20:26 |
| fungi | clarkb: zuul should have based the ref on the state of the repo in gerrit at the time the ref was being built | 20:26 |
| clarkb | fungi: it looked like jeblkair reran the check job before the other change merged so the zuul ref would not have included it | 20:26 |
| *** lcostantino has quit IRC | 20:26 | |
| clarkb | I think zuul did what is expected in this case | 20:26 |
| jeblair | oh right, i wanted to enqueue it in gate but i can't do that anymore. :( | 20:27 |
| clarkb | if not what is desired | 20:27 |
| fungi | clarkb: ahh, yep | 20:27 |
| fungi | clarkb: i misread the report time from jenkins as the start time not the end time | 20:27 |
| pabelanger | clarkb, thanks | 20:27 |
| clarkb | pabelanger: no problem. if that bothers you, we can probably update the plugin to start the pub thread when jenkins starts | 20:28 |
| clarkb | pabelanger: but I went the cheap route and simply check the config each time an event is ready and deal with ti then | 20:28 |
| clarkb | there are only about 100k events per day across all of our masters on a busy day which isn't many | 20:29 |
| *** saju_m has joined #openstack-infra | 20:29 | |
| *** _nadya_ has quit IRC | 20:30 | |
| pabelanger | clarkb, Ya, trying to see if some event could be triggered when jenkins created the job, so zuul knows of it. | 20:30 |
| pabelanger | so, was trying to better understand what the plugin did | 20:30 |
| clarkb | pabelanger: you when the job is created, hrm. You could probably extend the zmq event publisher to do that | 20:32 |
| clarkb | pabelanger: you would have to set up a second jenkins event listener for config changes, then pass the same queue the publisher thread pops events off of to that listener | 20:32 |
| clarkb | but it doesn't do that today | 20:32 |
| pabelanger | okay | 20:33 |
| pabelanger | I'm just thinking about ways, but will wait to see how jeblair handles it | 20:33 |
| *** jeckersb is now known as jeckersb_gone | 20:34 | |
| *** david-lyle has quit IRC | 20:34 | |
| *** ildikov_ has joined #openstack-infra | 20:36 | |
| clarkb | jeblair: what was the problem with storyboard db tests using your fixture? | 20:36 |
| *** saju_m has quit IRC | 20:37 | |
| lifeless | clarkb: did you see jeblair's email about it ? | 20:38 |
| *** afazekas has joined #openstack-infra | 20:38 | |
| clarkb | lifeless: I did, but yesterday he said he found the reason the fixture wasn't workign and a patch was incoming, reading his change in that email the particular bug fix doesn't pop out at me | 20:39 |
| jeblair | clarkb: hrm, i may have said a lot of things yesterday, some of them might even have been correct... :) | 20:40 |
| jeblair | clarkb: at any rate, i believe the current state is that the patch works as intended as far as mysql isolation for parallel testr | 20:40 |
| lifeless | \o/ | 20:41 |
| clarkb | jeblair: awesome | 20:41 |
| *** _nadya_ has joined #openstack-infra | 20:41 | |
| clarkb | jeblair: re the miyiasm thing, can sqlalchemy enforce an engine for us or possibly the fixture when it creates the schema? | 20:41 |
| clarkb | I suppose for that I should respond to the thread | 20:42 |
| *** atiwari has quit IRC | 20:42 | |
| *** nkinder has quit IRC | 20:42 | |
| jeblair | clarkb: but the schema is created with myisam tables on the slaves because sqlalchemy is used and the engine is not specified; alembic specifies the engine but alembic is only used in production, not for creating the schema for the normal unit tests | 20:42 |
| clarkb | I see. according to mysql docs with mysql 5.5 and onward innodb is the default table type | 20:43 |
| jeblair | there are two obvious solutions for that; i prefer the one that mirrors production use | 20:43 |
| clarkb | jeblair: I think that means we are ok on our ubuntu hosts but not the centos hosts | 20:43 |
| jeblair | clarkb: we set it to myisam on the slaves specifically so that we catch errors like this. :) | 20:43 |
| clarkb | oh right | 20:43 |
| clarkb | in the pupept | 20:43 |
| jeblair | it worked | 20:43 |
| clarkb | jeblair: shouldn't storyboard nodepool et al do http://docs.sqlalchemy.org/en/rel_0_8/orm/extensions/declarative.html#table-configuration to force innodb? | 20:45 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: Add StackTach project https://review.openstack.org/83208 | 20:45 |
| *** julim has joined #openstack-infra | 20:45 | |
| clarkb | I don't like relying on alembic to enforce that, the orm should | 20:45 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Include provider names in timeout messages https://review.openstack.org/82959 | 20:45 |
| jeblair | clarkb: yes, they should do that; definitely nodepool. | 20:46 |
| openstackgerrit | A change was merged to openstack-infra/config: Add a neutron job for api testing https://review.openstack.org/82226 | 20:46 |
| jeblair | clarkb: however, i still think alembic is the more important bit in storyboard because that's what's _actually_ used to create the database in production. | 20:46 |
| jeblair | i think ruslan is suggesting that if you have a test that proves that the result of creating the db with alembic is the same as creating it with the model, then it's okay. i could buy that. | 20:48 |
| clarkb | ++ | 20:48 |
| jeblair | however, that does start to get into another project i'm interested in: removing foreign keys from the database | 20:48 |
| jeblair | one way to do that is to have the sqla model define the foreign keys, but not define them in alembic. that makes it easy to use sqla, but you don't actually end up with the constaints in the db. | 20:49 |
| *** ominakov has quit IRC | 20:49 | |
| jeblair | it's still possible to use sqla without declaring fk's in the model (nodepool does it, actually) but it's a little more verbose. | 20:49 |
| jeblair | mordred wants to make a patch to sqla that lets you specify fks but then tell sqlalchemy to not actually tell the db to create them | 20:49 |
| jeblair | the really short version of why that is desirable is: your app needs to do the enforcement anyway, so don't waste database cpu resources on it (which are more constrained than your app cpu resources). and besides, they usually cause more problems than they solve. | 20:50 |
| jeblair | but perhaps it's possible to write the sqla == alembic schema check to disregard foreign keys | 20:52 |
| ruhe | jeblair: re fks in storyboard. krotscheck rised a valid question in https://review.openstack.org/#/c/83058/ | 20:53 |
| *** signed8bit has quit IRC | 20:54 | |
| jeblair | ruhe: thanks, hadn't seen that yet | 20:55 |
| *** pdmars has quit IRC | 20:56 | |
| *** weshay has quit IRC | 20:56 | |
| *** e0ne has joined #openstack-infra | 20:57 | |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: Add StackTach project https://review.openstack.org/83208 | 20:57 |
| sdague | jeblair: for what's it's worth, in nova we found that if you create db in 2 different ways, there are often subtle differences | 20:58 |
| sdague | so I'd say use alembic | 20:58 |
| *** branen has joined #openstack-infra | 20:58 | |
| clarkb | sdague: is that re table engine or fks? | 20:58 |
| sdague | I thought there were even table engine thigns | 20:58 |
| sdague | it's been a while, there were real differences though | 20:59 |
| *** boris-42 has quit IRC | 20:59 | |
| ruhe | sdague: but using alembic means that every developer has to have local mysql instance. i don't mind to have mysql instance for unit-tests, but do others like this idea? | 20:59 |
| sdague | I'm not sure how developing a db driven app, having local mysql is an undo burden | 21:00 |
| *** signed8bit has joined #openstack-infra | 21:00 | |
| jeblair | i'm okay with that requirement; especially since we'll run tests for people if they push patches | 21:01 |
| sdague | jeblair: is there no escape valve for sqlite like we have in migrate? | 21:01 |
| sdague | where you can specify an alternate migration for a version | 21:01 |
| sdague | for sqlite | 21:02 |
| sdague | which is needed for all the alters that drop things | 21:02 |
| sdague | because sqlite doesn't do that | 21:02 |
| *** mfer has quit IRC | 21:02 | |
| jeblair | sdague: i don't know; i'm not very familiar with alembic (i've almost learned how to invoke it). but that seems like the sort of thing i'd rather not do (extra work to support a db that isn't really used) | 21:03 |
| sdague | jeblair: totally fair | 21:03 |
| sdague | I think the only issue is having a standard way to provide creds for tox to pick up | 21:03 |
| ruhe | sdague: in StoryBoard we now expect openstackci_test user:password | 21:04 |
| sdague | ruhe: sure, then you can do that | 21:04 |
| sdague | just like from nova | 21:04 |
| sdague | jeblair: presumbly you could also be nice and skip the unit tests that are db related if you detect they don't have a db | 21:05 |
| sdague | it would run the rest of them | 21:05 |
| ruhe | yes, we have the same opportunistic DB migration tests - if mysql or postgres are present and have this user/password, then run the tests, otherwise skip | 21:05 |
| *** aysyd has quit IRC | 21:05 | |
| sdague | ruhe: cool, I wasn't clear how much of that copied in | 21:05 |
| sdague | I'm glad that pattern remains useful :) | 21:06 |
| jeblair | sdague: yeah, as long as we have the "require that the db tests be run" env flag set in the gate | 21:06 |
| ruhe | but, if we really decide to run ALL the unit tests on real DB, then we'll have to enforce developers to have local mysql instance | 21:06 |
| jeblair | (but i'm also okay being mean and saying to run the tests you need mysql; i won't argue against that though) | 21:07 |
| sdague | jeblair: also, IIRC, we are explicitly setting the gate mysql instances to myisam to detect when nova fails to specify engine in table definition | 21:07 |
| jeblair | sdague: yes, that's something i'm clarifying in my response to SpamapS now | 21:07 |
| sdague | that was all part and parcel of the opportunitistic testing | 21:07 |
| *** e0ne has quit IRC | 21:07 | |
| sdague | when I put that together near the end of folsom | 21:08 |
| jeblair | sdague: because that's responsible for some of the behavior i described | 21:08 |
| sdague | yeh | 21:08 |
| sdague | specifying and engine is good though, if you depend on behavior of one | 21:08 |
| jeblair | sdague: i view it as successful -- it helped me realize that storyboard did not set the engine in its sqla model | 21:08 |
| sdague | agreed :) | 21:08 |
| sdague | double plus good | 21:08 |
| *** ameade has joined #openstack-infra | 21:08 | |
| *** ArxCruz has quit IRC | 21:09 | |
| *** wchrisj has quit IRC | 21:11 | |
| *** skraynev is now known as skraynev_afk | 21:14 | |
| *** signed8bit has quit IRC | 21:15 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Remove pip version specifier https://review.openstack.org/82356 | 21:15 |
| mattoliverau | Morning | 21:15 |
| *** signed8bit has joined #openstack-infra | 21:15 | |
| *** boris-42 has joined #openstack-infra | 21:16 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 21:17 |
| *** marun has left #openstack-infra | 21:18 | |
| *** lifeless has quit IRC | 21:19 | |
| *** dprince has quit IRC | 21:20 | |
| zaro | mattoliverau: afternoon | 21:21 |
| *** e0ne has joined #openstack-infra | 21:22 | |
| *** CaptTofu has quit IRC | 21:24 | |
| *** CaptTofu has joined #openstack-infra | 21:25 | |
| *** CaptTofu has quit IRC | 21:26 | |
| *** CaptTofu_ has joined #openstack-infra | 21:26 | |
| *** dstanek has quit IRC | 21:26 | |
| zaro | clarkb, jeblair: your feedback to https://review.openstack.org/#/c/78930 would be appreciated. | 21:27 |
| *** dstanek has joined #openstack-infra | 21:27 | |
| clarkb | jogo: sdague mtreinish https://review.openstack.org/#/c/83182/ has been tested locally and I am fairly ocnfident it is working | 21:29 |
| *** moted has joined #openstack-infra | 21:29 | |
| sdague | clarkb: let's do it | 21:29 |
| clarkb | woot thanks | 21:31 |
| *** mrodden has joined #openstack-infra | 21:31 | |
| clarkb | going to push a change to the contributing file spelling out how local fucntional testing works | 21:31 |
| *** markmc has quit IRC | 21:32 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Take advantage of the new build_short_uuid field. https://review.openstack.org/83182 | 21:32 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Prepare for deletion of stable/grizzly branches https://review.openstack.org/83225 | 21:33 |
| fungi | that's ^ the first step toward grizzly eol | 21:33 |
| jeblair | !!!!!!!!! | 21:33 |
| openstack | jeblair: Error: "!!!!!!!!" is not a valid command. | 21:33 |
| jeblair | i was so happy and now i am sad | 21:34 |
| fungi | openstack!!!!!!!!!!!!1!11eleven | 21:34 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: Add StackTach project https://review.openstack.org/83208 | 21:34 |
| *** whoops has joined #openstack-infra | 21:34 | |
| fungi | anyway, once it's merged, i'll tag/delete the corresponding branches on all projects and then un-wip the cleanup patches i'm preparing to push up | 21:35 |
| *** jamielennox|away is now known as jamielennox | 21:35 | |
| *** ociuhandu has quit IRC | 21:35 | |
| *** markmc has joined #openstack-infra | 21:35 | |
| * jeblair remembers how to use bzr | 21:35 | |
| anteaya | morning mattoliverau | 21:36 |
| anteaya | zaro so workflow label | 21:36 |
| anteaya | zaro: since workflow label +1 == Approved | 21:37 |
| anteaya | zaro: does this mean the workflow label can only be used by core? | 21:37 |
| clarkb | jeblair: hacking on python jenkins? | 21:37 |
| clarkb | fungi: will review shortly | 21:37 |
| jeblair | clarkb: yeah, reviewing zaro's change | 21:38 |
| openstackgerrit | Sandy Walsh proposed a change to openstack-infra/config: Add StackTach project https://review.openstack.org/83208 | 21:38 |
| fungi | anteaya: why would it mean that? we're able to set arbitrary vote ranges in acls (remember core reviewers can review vote -2..+2 but registered users only -1..+1 for example) | 21:38 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Document process to run elastic-recheck locally https://review.openstack.org/83226 | 21:38 |
| anteaya | fungi: just wondering | 21:39 |
| fungi | anteaya: so we'd set the workflow label via acl allowing change owner to set -1..0 and core reviewers -1..+1 | 21:39 |
| *** mrodden has quit IRC | 21:39 | |
| anteaya | if we can set the voting range in the acl so that everyone can -1 workflow and only core can +1 workflow that answers my question | 21:39 |
| anteaya | okay, thanks | 21:39 |
| fungi | that's exactly the plan, yes | 21:39 |
| jeblair | not everyone can -1, only change owner and core | 21:39 |
| fungi | right | 21:39 |
| fungi | "everyone" can -1 or 0 their own change (e.g. each change owner) | 21:40 |
| jeblair | indeed | 21:41 |
| *** exz has quit IRC | 21:41 | |
| clarkb | fungi: reviewed | 21:41 |
| anteaya | okay thanks, getting a clearer picture now | 21:42 |
| SpamapS | jeblair: I so agree on ORMs being a horrible abstraction layer. :) | 21:42 |
| jeblair | SpamapS: yeah, they're great at it as long as you take care to specify all the things that could possibly be different. :) | 21:42 |
| *** freyes has quit IRC | 21:43 | |
| SpamapS | jeblair: right, like a concierge that speaks two languages, neither of which are the one you speak. | 21:43 |
| jeblair | SpamapS: Kiel mi vin helpos? | 21:44 |
| clarkb | zaro: for 78930 I think you captured the issues | 21:45 |
| fungi | clarkb: responded | 21:45 |
| clarkb | zaro: we should see if we can get plugin version detection going before we break backward compat with that plugin | 21:45 |
| *** hashar has quit IRC | 21:45 | |
| jeblair | i want this counted as code review in stackalytics! | 21:46 |
| jeblair | this is hard! | 21:46 |
| SpamapS | jeblair: ich bin ein berliner | 21:46 |
| clarkb | fungi: thanks, responded | 21:47 |
| clarkb | fungi: I guess there isn't much we can do about the grenade thing, but I still thik that change should remove the unneeded jobs at the same time | 21:47 |
| clarkb | jeblair: ha | 21:48 |
| *** sandywalsh has quit IRC | 21:48 | |
| *** mriedem has quit IRC | 21:48 | |
| fungi | clarkb: i'd rather not have to figure out which jobs are safe to remove along with that change, rather than just remove them all as soon as the last grizzly changes are abandoned | 21:49 |
| jeblair | zaro: why did you choose to key on longName instead of shortName? | 21:50 |
| clarkb | fungi: so I have the opposite idea, by removing them jobs from JJB you get tests that make sure you didn't remove too much from zuul and vice versa | 21:50 |
| jgriffith | fungi: jeblair anything change today in how unit tests jobs are checked in the gate for cinder? | 21:50 |
| clarkb | fungi: basically by removing on both sides you get gate tests that make sure you didn't imbalance them | 21:50 |
| clarkb | jgriffith: no | 21:50 |
| jgriffith | fungi: jeblair we're getting check config errors | 21:50 |
| jgriffith | but I can't figure out where the diff are coming from | 21:50 |
| clarkb | jgriffith: can you link to a failure? | 21:50 |
| jgriffith | https://jenkins01.openstack.org/job/gate-cinder-python27/5762/console | 21:51 |
| fungi | jgriffith: it's just the same old "keystoneclient changed options again, regenerate your sample configs" issue, right? | 21:51 |
| jgriffith | fungi: yeah, but that trick didn't work it seems | 21:51 |
| clarkb | yup thats the one | 21:51 |
| jgriffith | fungi: OH | 21:51 |
| jgriffith | did that push a new client? | 21:51 |
| jgriffith | they | 21:51 |
| clarkb | I wish that check would diaf | 21:51 |
| jgriffith | I'm considering taking it away | 21:51 |
| fungi | jgriffith: there's a whole mailing list thread going about ripping those sample configs out of everyone's git repos. i'm personally a fan of generating them when api docs are built | 21:52 |
| * jgriffith sheepishly walks away | 21:52 | |
| fungi | or at least in a similar way to how api docs are built | 21:52 |
| jgriffith | fungi: I would agree | 21:52 |
| clarkb | fungi: going back to the grizzly thing, it should be safe to remove all grizzly jobs | 21:52 |
| jgriffith | alright... so update the keystone client and regenerate should do the trick I'm guessing | 21:53 |
| jgriffith | but now everybody's hosed behind the keystone change | 21:53 |
| clarkb | fungi: and that is desired to avoid the wedged state you describe as a possibility | 21:53 |
| jgriffith | I think I'll just disable the thing now and merge it | 21:53 |
| clarkb | jgriffith: well I am not sure you can straight up disable it | 21:53 |
| jgriffith | clarkb: sure I can :) | 21:53 |
| clarkb | jgriffith: because things depend on that conf file (devstack) | 21:53 |
| fungi | clarkb: removing all grizzly-specific jobs doesn't prevent zuul from testing grizzly changes though, right? am i just overthinking it? | 21:54 |
| clarkb | jgriffith: and if you disable ti you will see failures elsewhere in harder to debug places | 21:54 |
| jgriffith | #/bin/bash exit | 21:54 |
| jgriffith | not saying remove the conf file, remove the check | 21:54 |
| clarkb | jgriffith: I think that is inviting more trouble than its worth | 21:54 |
| clarkb | jgriffith: right, | 21:54 |
| jgriffith | ok | 21:54 |
| clarkb | if that conf file doesn't have the correct data in it it will break stuff | 21:54 |
| jgriffith | You're my voice of reason for the day :) | 21:54 |
| clarkb | which is why the check exists, its just poorly implemented imo | 21:54 |
| jgriffith | clarkb: oh yeah I get that for sure | 21:54 |
| fungi | it's checking in the wrong place at the wrong time, basically | 21:55 |
| clarkb | fungi: I think that is overthinking it, changes to grizzly will use the non special grizzly jobs | 21:55 |
| jgriffith | clarkb: no +2 for patches that change config, but you're right, you're right ;) | 21:55 |
| zaro | anteaya: no, who can vote on that label depends on how we configure gerrit permissions | 21:55 |
| *** dcramer_ has quit IRC | 21:55 | |
| jgriffith | ok... well I know how to reset at least thanks to you all | 21:55 |
| jgriffith | appreciate it | 21:55 |
| clarkb | fungi: basically changes to grizzly use gate-python27 and friends and normal devstack for the most part | 21:55 |
| anteaya | zaro: great thank you | 21:55 |
| zaro | jeblair: are you referring to my patch for python-jenkins? | 21:56 |
| anteaya | fungi outlined some scenarios for me about how permissions might be set | 21:56 |
| jeblair | zaro: yes | 21:56 |
| jeblair | https://code.launchpad.net/~zaro0508/python-jenkins/plugin-info-method/+merge/196653 | 21:56 |
| zaro | looking | 21:56 |
| fungi | clarkb: meh. i'm still unconvinced that changing the way grizzly changes are tested before deleting teh grizzly branches is the technically correct way to do this (thinking in terms of future eol cycles which might be more tightly timed), but am willing to do so if that will get this approved faster | 21:57 |
| clarkb | fungi: I think we are maybe talking about two different things | 21:57 |
| clarkb | fungi: you are talking about how grizzly changes are tested and I am saying remove the unneeded grizzly specific jobs | 21:57 |
| clarkb | grizzly specific jobs aren't used to test grizzly changes iirc | 21:57 |
| jeblair | zaro: left a comment in lp; you might want to reload | 21:58 |
| fungi | clarkb: but i still then need a separate cleanup change to remove the grizzly branch filters from the layout.yaml anyway | 21:58 |
| *** b3nt_pin has joined #openstack-infra | 21:58 | |
| fungi | doing so before the branches are deleted will change the way they're tested | 21:58 |
| clarkb | fungi: hrm, good point. so you are suggesting to do both job removal and job branch filter updates in one change? | 21:59 |
| fungi | since we have filters saying "only run this job on grizzly changes" or others saying "never run this job on grizzly changes" | 21:59 |
| clarkb | fungi: I was not suggesting you updated the job branch filters fwiw | 21:59 |
| *** nkinder has joined #openstack-infra | 22:00 | |
| fungi | and there are still plenty of grizzly-specific jobs (particularly old-docs jobs) which shouldn't be deleted before the branches are deleted | 22:00 |
| *** b3nt_pin is now known as beagles | 22:00 | |
| jgriffith | fungi: so I'm curious, could we just automate the whole conf thing as part of the merge process? | 22:01 |
| jgriffith | fungi: ie generate the conf file on succesful verify rigth before the merge? | 22:01 |
| fungi | jgriffith: no, because that would change the commit being merged | 22:01 |
| jgriffith | oh | 22:01 |
| jgriffith | never mind | 22:01 |
| jgriffith | got it | 22:01 |
| jgriffith | back under my rock I go | 22:01 |
| *** _nadya_ has quit IRC | 22:02 | |
| clarkb | fungi: ok, I will +2 this change and we can do all of them at once | 22:02 |
| fungi | jgriffith: ideally we do something to stop including those files in the repo at all, and have devstack generate the sample configs it needs, and have a publish job that generates and sticks a copy in a useful place for deployers/operators to consult and docs to link to | 22:02 |
| clarkb | fungi: its just weird to me that we would stop running tests permanently as part of a change and not delete those tests too :) | 22:02 |
| zaro | jeblair: i see now. will update that. just need to figure out how to work with lp/bzr again :) | 22:03 |
| *** dhellmann is now known as dhellmann_ | 22:03 | |
| *** blamar has quit IRC | 22:03 | |
| *** bhuvan_ has joined #openstack-infra | 22:03 | |
| fungi | clarkb: it's part of a series... overall plan it stop running things which will cause negative impact if the branches are deleted first, then tag/delete the branches, then remove jobs and related job filters for everything grizzly-specific in multiple repos (config, d-g, et cetera) | 22:04 |
| clarkb | fungi: right | 22:04 |
| zaro | anteaya: my chage earlier shows the permissions required for workflow. we basically want to allow 'change owner' and 'admin' set workflow label for all projects. | 22:05 |
| fungi | i'm stacking the config cleanup change on top of that one, so we can still see whether things are heinous | 22:05 |
| anteaya | zaro: I must have missed that, which patch has that? | 22:05 |
| fungi | but will wip/-2 it until remaining changes are abandoned and branches are deleted | 22:06 |
| anteaya | zaro: looked at a few or yours | 22:06 |
| clarkb | fungi: its more the lack of 1:1 removal in that change that botehrs me than the planned stepping | 22:06 |
| zaro | anteaya: https://review.openstack.org/#/c/70818 | 22:06 |
| anteaya | k, thanks | 22:06 |
| *** bhuvan has quit IRC | 22:06 | |
| *** bknudson has quit IRC | 22:09 | |
| zaro | anteaya: refere that change. i've answered your questions | 22:09 |
| *** rcleere has quit IRC | 22:09 | |
| *** ameade has quit IRC | 22:09 | |
| anteaya | zaro I think I have it now | 22:11 |
| clarkb | fungi: but given your plan it wfm | 22:11 |
| anteaya | patch owner gets -1 .. 0 voting on workflow label, core gets -1 .. +1 on workflow label | 22:12 |
| *** jooools has quit IRC | 22:12 | |
| zaro | anteaya: exactly! | 22:13 |
| anteaya | \o/ | 22:13 |
| jeblair | fungi: are you planning on removing grizzly branch specifiers in the zuul layout in a subsequent change? | 22:13 |
| jeblair | fungi: i think i'll interpret your comment as yes | 22:13 |
| jeblair | grizzlyectomy part1 approved | 22:15 |
| fungi | jeblair: yep, about to push that change now | 22:15 |
| anteaya | this was approved 10 hours ago by fungi is not on the zuul status page and never got merged: https://review.openstack.org/#/c/80858/ | 22:16 |
| anteaya | I don't see why so I have no bug number for reverify | 22:16 |
| *** moted has quit IRC | 22:16 | |
| clarkb | jeblair: ya that is part of what fungi and I were just discussing, basically massive change post branch deletion to remove most things | 22:16 |
| jeblair | anteaya: that was far more than 10 hours ago. it was likely during the time when gerrit was stuck. | 22:17 |
| *** moted has joined #openstack-infra | 22:17 | |
| *** eharney has quit IRC | 22:17 | |
| openstackgerrit | A change was merged to openstack-infra/config: Prepare for deletion of stable/grizzly branches https://review.openstack.org/83225 | 22:17 |
| *** mwagner_lap has quit IRC | 22:17 | |
| fungi | anteaya: reapproved, as jeblair noted that was when gerrit streaming was having issues yesterday | 22:18 |
| anteaya | it said March 25 8:07am eastern time | 22:18 |
| anteaya | are we not still March 25? | 22:18 |
| anteaya | I know I get a lot of stuff wrong | 22:18 |
| fungi | not on this planet anyway ;) | 22:18 |
| fungi | it's about to turn march 27th utc in a couple hours | 22:19 |
| *** dstanek has quit IRC | 22:19 | |
| anteaya | thanks for letting me know | 22:19 |
| anteaya | I need to do something on the 27th | 22:19 |
| *** whoops has quit IRC | 22:20 | |
| *** signed8bit has quit IRC | 22:21 | |
| *** jhesketh__ has joined #openstack-infra | 22:21 | |
| jhesketh_ | Morning | 22:21 |
| *** signed8bit has joined #openstack-infra | 22:22 | |
| anteaya | morning jhesketh_ | 22:22 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove grizzly jobs and job filters https://review.openstack.org/83232 | 22:22 |
| fungi | wip for now ^ but also i think there may be more jobs i can remove which were hanging around only to support grizzly changes so i'm hunting them down and adding to that shortly | 22:23 |
| jeblair | jhesketh_: good morning! have you started looking into the streaming decompress data from swift problem? | 22:24 |
| jhesketh_ | jeblair: yeah, I made some progress on that: https://review.openstack.org/#/c/76796/ | 22:25 |
| *** beagles has quit IRC | 22:25 | |
| *** ArxCruz has joined #openstack-infra | 22:25 | |
| jeblair | jhesketh_: oh cool, i guess i got mixed up since that discussion didn't happen on the review | 22:26 |
| jeblair | i will try to look at that soon; sdague ^ | 22:26 |
| jhesketh_ | jeblair, not sure what discussion you're referring to? I may have missed it | 22:26 |
| jhesketh_ | jeblair: The zuul swift stuff and client stuff have been rebased and should be good to remerge | 22:27 |
| jhesketh_ | well re-review | 22:27 |
| jeblair | jhesketh_: the one where sdague elucidated the problem to you and i on irc; i thought it happened after the most recent patchset, but it must have been before | 22:27 |
| sdague | jhesketh_: sure, though that doesn't solve it for the compressed case | 22:27 |
| *** lcheng has quit IRC | 22:27 | |
| *** bauzas has quit IRC | 22:27 | |
| sdague | which I thought was required to go this route | 22:27 |
| jhesketh_ | sdague: I have compressed loading gzip in that patchset | 22:27 |
| jhesketh_ | eh, bad speaking | 22:28 |
| jhesketh_ | s/compressed/addressed | 22:28 |
| *** jergerber has quit IRC | 22:28 | |
| sdague | oh, ok, I see it now | 22:28 |
| *** ameade has joined #openstack-infra | 22:28 | |
| jhesketh_ | admittedly only works for gzip | 22:28 |
| *** ameade has quit IRC | 22:29 | |
| sdague | that should be fine | 22:29 |
| *** ameade has joined #openstack-infra | 22:29 | |
| jeblair | jhesketh_: zuul change aprvd | 22:29 |
| jgriffith | Ok, so I have a bit of a problem with this config file stuff | 22:30 |
| jgriffith | I can't seem to update it localy? | 22:30 |
| *** dhellmann_ is now known as dhellmann | 22:31 | |
| jgriffith | I run check and it fails, I run generate and no change? | 22:31 |
| *** lcheng has joined #openstack-infra | 22:31 | |
| jgriffith | so somwhere/somehow check and generate got out of sync | 22:31 |
| jhesketh_ | thanks jeblair | 22:32 |
| sdague | jhesketh_: that seems like a reasonable first pass | 22:32 |
| sdague | jeblair: do we have a livish env to be able to play with this to make sure it works end to end? | 22:33 |
| jhesketh_ | jeblair, clarkb: Also if you have time today it'd be great to get the javascript changes in, including mordred's, so future changes can work off the tail of jshint | 22:33 |
| jeblair | sdague: not yet, but once the zuul changes go in, we can start shoving real logs into a container | 22:36 |
| sdague | cool | 22:37 |
| openstackgerrit | A change was merged to openstack-infra/gitdm: Add alias email for james.polley@hp.com https://review.openstack.org/80858 | 22:37 |
| *** mrodden has joined #openstack-infra | 22:38 | |
| *** jgrimm has quit IRC | 22:39 | |
| *** yolanda has quit IRC | 22:40 | |
| sdague | couple of minor things | 22:41 |
| *** ociuhandu has joined #openstack-infra | 22:41 | |
| *** dhellmann is now known as dhellmann_ | 22:41 | |
| sdague | there was a reason why I was doing the file read and failing on it to throw the 404 instead of doing a file exists check | 22:41 |
| sdague | but, honestly, I can't remember what that was now | 22:42 |
| *** blamar has joined #openstack-infra | 22:42 | |
| sdague | so if it's still an issue we'll probably see it pretty quickly once it lands | 22:42 |
| jhesketh_ | hmm okay | 22:44 |
| *** thedodd has quit IRC | 22:45 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 22:46 |
| *** rfolco has quit IRC | 22:46 | |
| *** e0ne has quit IRC | 22:49 | |
| *** dstanek has joined #openstack-infra | 22:49 | |
| *** lcheng has quit IRC | 22:49 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove grizzly jobs and job filters https://review.openstack.org/83232 | 22:50 |
| *** loquacities is now known as starving | 22:52 | |
| *** blamar has quit IRC | 22:53 | |
| jhesketh_ | jeblair: yay, swift is merged | 22:54 |
| fungi | adam_g: if you're around, any objection to me (ab)using gerrit admin privs to abandon the one remaining open grizzly change? https://review.openstack.org/56619 | 22:54 |
| jhesketh_ | jeblair: so how do we set up a swift account.. I'm guessing you might need to do something? | 22:54 |
| adam_g | fungi, yeah--definitely | 22:56 |
| fungi | about to delete the branch out from under it, so just wanted to be sure it's dead, dead, deadski | 22:57 |
| *** alex-gone is now known as Alexandra | 22:58 | |
| *** harlowja has quit IRC | 22:58 | |
| adam_g | is there some way for jenkins to blacklist a given cloud provider for specific jobs? wondering if workaround https://review.openstack.org/#/c/82191/ can be avoided if check-tempest-dsvm-virtual-ironic never ended up on rackspace | 23:00 |
| *** mrodden has quit IRC | 23:02 | |
| fungi | adam_g: not trivially--we basically need special node types defined which are only on some providers and not others | 23:03 |
| fungi | adam_g: and that makes it hard for us to run jobs if the wrong provider is down, so we'd really rather not | 23:03 |
| *** harlowja has joined #openstack-infra | 23:03 | |
| fungi | also makes it harder to balance workloads across providers | 23:03 |
| adam_g | ah okay | 23:03 |
| adam_g | someone had pitched the idea as a better work around, didn't know if its actually doable | 23:04 |
| fungi | adam_g: more likely that problem could vaporize soon if newer libvirt starts getting used? | 23:04 |
| fungi | oh, nevermind the libvirt thing--it's a host setup issue i guess | 23:05 |
| *** bknudson has joined #openstack-infra | 23:05 | |
| fungi | i was looking at the comment above it | 23:05 |
| adam_g | fungi, well, a newer libvirt would help--theres an update in the queue but appears to be slow moving | 23:05 |
| *** starving is now known as Loquacities | 23:05 | |
| *** julim has quit IRC | 23:05 | |
| sdague | adam_g: why does that get in the way? | 23:05 |
| *** markmc has quit IRC | 23:06 | |
| adam_g | sdague, libvirt bug in saucy/UCA-icehouse - https://bugs.launchpad.net/bugs/1248025 | 23:06 |
| uvirtbot | Launchpad bug 1248025 in libvirt "[SRU] libvirt-bin fails to start inside Xen" [High,In progress] | 23:06 |
| sdague | ah | 23:06 |
| adam_g | wonder if we can just use UCA icehouse for that instead | 23:06 |
| fungi | oh, that craziness again | 23:06 |
| sdague | adam_g: is there a reason you guys aren't running on precise? | 23:07 |
| adam_g | sdague, we are, the devstack setup relies on newer OVS features in newer libvirt to get the fake baremetal networking environment setup | 23:07 |
| *** signed8bit has quit IRC | 23:08 | |
| adam_g | so libvirt comes from UCA | 23:08 |
| sdague | ah | 23:08 |
| sdague | and no way to do that with old ovs? | 23:08 |
| fungi | adam_g: in theory this also goes away (and we presumably replace it with a whole new set of troubles) when we move to testing on trusty for juno? | 23:09 |
| *** juice has quit IRC | 23:09 | |
| *** Trozz has joined #openstack-infra | 23:10 | |
| adam_g | sdague, AFAIK its libvirt features specific to OVS, not OVS itself. | 23:11 |
| adam_g | sdague, <virtualport='openvswitch'> in domain xml definination, for example | 23:12 |
| *** dims_ has quit IRC | 23:12 | |
| adam_g | fungi, yeah, in theory. at this point if the ironic job needs to install anything newer, it should probably be what we'll be using in juno anyway | 23:12 |
| *** mrodden has joined #openstack-infra | 23:13 | |
| *** mburned is now known as mburned_out | 23:14 | |
| *** bhuvan_ has quit IRC | 23:15 | |
| adam_g | hmm | 23:16 |
| adam_g | i wonder if its still okay for me to upload to UCA. | 23:16 |
| fungi | adam_g: there's one way to find out... if they don't revoke your account for it (or if they do for that matter!) you'll have your answer ;) | 23:17 |
| adam_g | :) | 23:18 |
| *** bhuvan has joined #openstack-infra | 23:18 | |
| *** flaper87 is now known as flaper87|afk | 23:18 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Remove stable/grizzly provisions and workarounds https://review.openstack.org/83245 | 23:18 |
| *** jepoy has joined #openstack-infra | 23:19 | |
| *** jepoy has quit IRC | 23:19 | |
| *** mkoderer has quit IRC | 23:21 | |
| zaro | jeblair: just update plugin info patch for python-jenkins, hope i've done it correctly :) | 23:23 |
| *** nati_ueno has quit IRC | 23:23 | |
| *** nati_ueno has joined #openstack-infra | 23:23 | |
| zaro | jeblair: update will key on both shortName and longName | 23:23 |
| *** mrodden has quit IRC | 23:24 | |
| *** mrodden has joined #openstack-infra | 23:25 | |
| *** juice has joined #openstack-infra | 23:26 | |
| clarkb | fungi: can you check my comment on the config chagne to rip stuff out? | 23:27 |
| *** arosen-laptop has joined #openstack-infra | 23:28 | |
| clarkb | fungi: pretty sure its related to the grenade branch stuff of before and am looking for clarification | 23:28 |
| *** alexpilotti has quit IRC | 23:28 | |
| jeblair | zaro: cool, thx | 23:28 |
| *** mburned_out is now known as mburned | 23:29 | |
| *** dims_ has joined #openstack-infra | 23:30 | |
| fungi | clarkb: good point... i guess that was only needed because grenade's stable/grizzly branch ran no voting gate jobs? | 23:31 |
| fungi | clarkb: i guess we can just remove that? | 23:31 |
| clarkb | fungi: yeah I think we can just rip it out | 23:31 |
| clarkb | fungi: basically grenade didn't work with grizzly or something | 23:31 |
| fungi | but had a grizzly branch anyway? | 23:32 |
| clarkb | fungi: let me do some git history detectiving to track that down | 23:32 |
| *** mriedem has joined #openstack-infra | 23:32 | |
| clarkb | fungi: aha! | 23:33 |
| clarkb | fungi: 94b80e00e334917f2b2dd1a7223ce2fd580ec726 it can be removed | 23:33 |
| fungi | perfect | 23:34 |
| jhesketh_ | sdague: for making swiftclient option in loganalyze, should I check if swift is set up in the config and import it at that point | 23:36 |
| jhesketh_ | that will mean importing it at each application request | 23:36 |
| *** yamahata has quit IRC | 23:37 | |
| *** thuc_ has quit IRC | 23:37 | |
| *** mwagner_lap has joined #openstack-infra | 23:38 | |
| *** thuc has joined #openstack-infra | 23:38 | |
| sdague | jhesketh_: honestly, I'd be fine with a try at the top of the imports | 23:39 |
| *** Trozz has quit IRC | 23:40 | |
| sdague | mostly I want to be able to run locally without swiftclient when I don't need swiftclient | 23:40 |
| jhesketh_ | works for me | 23:41 |
| *** emagana has quit IRC | 23:41 | |
| *** thuc has quit IRC | 23:42 | |
| *** mrodden has quit IRC | 23:43 | |
| *** mrodden has joined #openstack-infra | 23:45 | |
| *** mrodden has quit IRC | 23:47 | |
| clarkb | fungi: d-g change for grizzly reviewed | 23:58 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Fetch tag refs from zuul in gerrit-git-prep https://review.openstack.org/82583 | 23:58 |
| clarkb | jhesketh_: sdague: is loganalyze going to fetch from swift, do modifications, then serve to the client? | 23:58 |
| fungi | clarkb: is that ^ what you had in mind for your previous comment on that change? | 23:58 |
| clarkb | fungi: looking | 23:59 |
| sdague | clarkb: yeh | 23:59 |
| jhesketh_ | clarkb:yes | 23:59 |
| sdague | jhesketh_: just implemented a generator stack to do that | 23:59 |
| sdague | so the top layers look exactly the same as before | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!