| *** UtahDave has joined #openstack-infra | 00:05 | |
| *** boris-42 has joined #openstack-infra | 00:13 | |
| *** boris-42 has quit IRC | 00:17 | |
| *** fifieldt has joined #openstack-infra | 00:36 | |
| *** dcramer_ has quit IRC | 00:39 | |
| *** fifieldt has quit IRC | 00:45 | |
| *** blamar has quit IRC | 01:08 | |
| *** CaptTofu has quit IRC | 01:11 | |
| *** CaptTofu has joined #openstack-infra | 01:12 | |
| *** Alex_Gaynor has quit IRC | 01:35 | |
| *** Alex_Gaynor has joined #openstack-infra | 01:35 | |
| *** talluri has joined #openstack-infra | 01:40 | |
| *** adarazs has quit IRC | 01:41 | |
| *** UtahDave has quit IRC | 01:42 | |
| *** talluri has quit IRC | 01:45 | |
| *** markwash has joined #openstack-infra | 01:46 | |
| *** CaptTofu has quit IRC | 01:54 | |
| *** CaptTofu has joined #openstack-infra | 01:54 | |
| *** pcrews has quit IRC | 02:00 | |
| *** CaptTofu has quit IRC | 02:01 | |
| *** CaptTofu has joined #openstack-infra | 02:01 | |
| *** nati_ueno has joined #openstack-infra | 02:20 | |
| *** nati_ueno has quit IRC | 02:20 | |
| *** nati_ueno has joined #openstack-infra | 02:43 | |
| *** senk has joined #openstack-infra | 02:44 | |
| *** wenlock has joined #openstack-infra | 02:48 | |
| *** nati_ueno has quit IRC | 02:48 | |
| *** senk has quit IRC | 02:53 | |
| *** jamesmcarthur has joined #openstack-infra | 02:55 | |
| *** senk has joined #openstack-infra | 02:56 | |
| *** senk has quit IRC | 03:00 | |
| *** senk has joined #openstack-infra | 03:01 | |
| *** senk has quit IRC | 03:11 | |
| *** mriedem has quit IRC | 03:20 | |
| *** reed has joined #openstack-infra | 03:33 | |
| *** reed has quit IRC | 03:39 | |
| *** UtahDave has joined #openstack-infra | 03:56 | |
| *** CaptTofu has quit IRC | 04:07 | |
| *** CaptTofu has joined #openstack-infra | 04:08 | |
| *** marun has joined #openstack-infra | 04:14 | |
| openstackgerrit | Noorul Islam K M proposed a change to openstack-infra/reviewstats: Add new core reviewers https://review.openstack.org/56806 | 04:22 |
|---|---|---|
| *** UtahDave has quit IRC | 04:36 | |
| *** blamar has joined #openstack-infra | 04:45 | |
| *** sdake_ has quit IRC | 04:48 | |
| openstackgerrit | Noorul Islam K M proposed a change to openstack-infra/reviewstats: Add rubick project tracking https://review.openstack.org/56808 | 04:52 |
| openstackgerrit | Noorul Islam K M proposed a change to openstack-infra/reviewstats: Add python-solumclient subproject https://review.openstack.org/56810 | 04:57 |
| *** jamesmcarthur has quit IRC | 04:58 | |
| *** WarrenUsui has joined #openstack-infra | 05:06 | |
| *** aardvark has quit IRC | 05:09 | |
| *** sdake_ has joined #openstack-infra | 05:14 | |
| *** boris-42 has joined #openstack-infra | 05:25 | |
| *** che-arne has quit IRC | 05:34 | |
| *** che-arne has joined #openstack-infra | 05:36 | |
| *** wenlock has quit IRC | 05:36 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:57 | |
| *** SergeyLukjanov has quit IRC | 05:59 | |
| *** boris-42 has quit IRC | 06:31 | |
| *** boris-42 has joined #openstack-infra | 06:34 | |
| *** davidhadas has joined #openstack-infra | 06:40 | |
| *** UtahDave has joined #openstack-infra | 06:49 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Serve local mirror using apache https://review.openstack.org/56752 | 07:08 |
| *** alexpilotti has joined #openstack-infra | 07:10 | |
| *** rwsu-pto has quit IRC | 07:17 | |
| *** rwsu-pto has joined #openstack-infra | 07:17 | |
| *** yolanda has joined #openstack-infra | 07:32 | |
| *** talluri has joined #openstack-infra | 07:40 | |
| *** boris-42 has quit IRC | 07:41 | |
| clarkb | mordred: jetlag must really be getting to you :) | 07:48 |
| *** talluri has quit IRC | 07:50 | |
| *** talluri has joined #openstack-infra | 07:50 | |
| *** UtahDave has quit IRC | 07:56 | |
| mordred | clarkb: o m g | 08:04 |
| mordred | clarkb: I can't seem to stop myself from going to sleep at 5pm | 08:05 |
| *** julim has joined #openstack-infra | 08:26 | |
| *** julim has quit IRC | 08:34 | |
| *** dkliban has joined #openstack-infra | 08:36 | |
| *** markwash has quit IRC | 08:37 | |
| *** noorul has joined #openstack-infra | 08:58 | |
| *** noorul has left #openstack-infra | 08:58 | |
| *** rwsu-pto has quit IRC | 09:50 | |
| *** rwsu-pto has joined #openstack-infra | 09:50 | |
| *** adarazs has joined #openstack-infra | 09:52 | |
| *** harlowja has quit IRC | 09:54 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Serve local mirror using apache https://review.openstack.org/56752 | 10:25 |
| openstackgerrit | A change was merged to openstack/requirements: Update stevedore https://review.openstack.org/56240 | 10:26 |
| *** Ryan_Lane has quit IRC | 10:28 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/pypi-mirror: Refactor run-mirror to make it readable https://review.openstack.org/56784 | 10:44 |
| *** boris-42 has joined #openstack-infra | 11:04 | |
| *** boris-42 has quit IRC | 11:06 | |
| *** SergeyLukjanov has joined #openstack-infra | 11:16 | |
| *** branen_ has quit IRC | 11:18 | |
| *** boris-42 has joined #openstack-infra | 11:18 | |
| *** SergeyLukjanov has quit IRC | 11:20 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Serve local mirror using apache https://review.openstack.org/56752 | 11:26 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run the mirror if we land changes to pypi-mirror https://review.openstack.org/56777 | 11:29 |
| *** salv-orlando has quit IRC | 11:30 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add support for per-distro wheel mirrors https://review.openstack.org/41268 | 11:46 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Use select-mirror in devstack-gate https://review.openstack.org/48064 | 11:48 |
| *** talluri has quit IRC | 11:55 | |
| *** Loquacity has quit IRC | 11:58 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Clean up integration script https://review.openstack.org/56816 | 12:19 |
| *** SergeyLukjanov has joined #openstack-infra | 12:20 | |
| *** jeblair has quit IRC | 12:21 | |
| *** jeblair has joined #openstack-infra | 12:21 | |
| *** marun has quit IRC | 12:26 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Enable wheel processing in the tests https://review.openstack.org/56817 | 12:28 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation https://review.openstack.org/48803 | 12:28 |
| mordred | for those following along at home, I'm making an easy to understand set of changes around wheels | 12:30 |
| mordred | https://review.openstack.org/#/q/status:open+topic:wheels-in-mirror,n,z | 12:30 |
| *** wayneeseguin has quit IRC | 12:32 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Enable the use of wheels in the gate https://review.openstack.org/56660 | 12:38 |
| *** wayneeseguin has joined #openstack-infra | 12:38 | |
| *** jamesmcarthur has joined #openstack-infra | 12:46 | |
| *** yolanda has quit IRC | 12:56 | |
| *** yolanda has joined #openstack-infra | 12:59 | |
| *** nati_ueno has joined #openstack-infra | 13:04 | |
| *** SergeyLukjanov has quit IRC | 13:04 | |
| *** adarazs has quit IRC | 13:18 | |
| *** jamesmcarthur has quit IRC | 13:23 | |
| *** jamesmcarthur has joined #openstack-infra | 13:24 | |
| *** yolanda has quit IRC | 13:27 | |
| *** boris-42 has quit IRC | 13:33 | |
| *** boris-42 has joined #openstack-infra | 13:41 | |
| *** CaptTofu has quit IRC | 13:41 | |
| *** CaptTofu has joined #openstack-infra | 13:42 | |
| *** ace05_ has joined #openstack-infra | 13:42 | |
| *** nati_uen_ has joined #openstack-infra | 13:48 | |
| *** nati_ueno has quit IRC | 13:52 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:57 | |
| *** boris-42 has quit IRC | 13:57 | |
| *** boris-42 has joined #openstack-infra | 13:59 | |
| *** nati_uen_ has quit IRC | 14:00 | |
| *** boris-42 has quit IRC | 14:07 | |
| *** boris-42 has joined #openstack-infra | 14:08 | |
| *** yolanda has joined #openstack-infra | 14:09 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Enable wheel processing in the tests https://review.openstack.org/56817 | 14:24 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Clean up integration script https://review.openstack.org/56816 | 14:24 |
| *** yolanda has quit IRC | 14:24 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation https://review.openstack.org/48803 | 14:24 |
| *** mriedem has joined #openstack-infra | 14:35 | |
| *** SergeyLukjanov has quit IRC | 14:39 | |
| *** nati_ueno has joined #openstack-infra | 14:42 | |
| *** dstanek has joined #openstack-infra | 14:43 | |
| *** alexpilotti has quit IRC | 14:51 | |
| anteaya | mordred I keep going to sleep at 7pm | 14:55 |
| mordred | anteaya: I'm jealous - I keep going to sleep at 5 | 14:56 |
| anteaya | I went for an extra long walk yesterday and then watched some movie | 14:56 |
| anteaya | was enought to keep me up to 7pm | 14:56 |
| anteaya | Aliens is one of my favourite movies, it was on at 8pm - couldn't stay up for it though | 14:57 |
| anteaya | mordred: I am rechecking 56752 | 14:57 |
| anteaya | let's hope it passes grenade this time | 14:58 |
| mordred | awesome | 14:58 |
| *** jamesmcarthur has quit IRC | 14:59 | |
| anteaya | 56816 is failing on integration | 14:59 |
| anteaya | so is 56817 | 15:00 |
| mordred | yes. I've uploaded new versions of those | 15:01 |
| mordred | oh. crap. still failing | 15:01 |
| anteaya | yeah | 15:01 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Enable wheel processing in the tests https://review.openstack.org/56817 | 15:04 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Clean up integration script https://review.openstack.org/56816 | 15:04 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation https://review.openstack.org/48803 | 15:04 |
| mordred | that shoudl do it | 15:04 |
| anteaya | let's find out what jenkins says | 15:05 |
| anteaya | *crosses fingers* | 15:05 |
| anteaya | where's that rubber chicken? | 15:05 |
| *** SergeyLukjanov has joined #openstack-infra | 15:21 | |
| *** dstanek has quit IRC | 15:24 | |
| anteaya | 56752 passed grenade this time | 15:25 |
| anteaya | and failed on devstack vm full | 15:39 |
| *** jamesmcarthur has joined #openstack-infra | 15:39 | |
| mordred | wee | 15:39 |
| * mordred cries | 15:39 | |
| anteaya | :( | 15:41 |
| anteaya | well it is a Sunday so not a large check queue | 15:42 |
| anteaya | mordred: do you have a minute to join us in -neutron | 15:49 |
| anteaya | a question about increasing the number of lines for the backtrace in a failed test, run locally | 15:50 |
| *** jamesmcarthur has quit IRC | 15:51 | |
| *** jamesmcarthur has joined #openstack-infra | 15:54 | |
| *** pcrews has joined #openstack-infra | 16:00 | |
| *** CaptTofu has quit IRC | 16:08 | |
| *** CaptTofu has joined #openstack-infra | 16:09 | |
| *** mriedem has quit IRC | 16:27 | |
| *** CaptTofu has quit IRC | 16:33 | |
| *** CaptTofu has joined #openstack-infra | 16:33 | |
| *** blamar has quit IRC | 16:41 | |
| *** mestery_ has joined #openstack-infra | 16:42 | |
| *** mestery has quit IRC | 16:45 | |
| *** jamesmcarthur has quit IRC | 16:46 | |
| *** salv-orlando has joined #openstack-infra | 16:48 | |
| *** boris-42 has quit IRC | 16:48 | |
| *** mestery_ is now known as mestery | 16:48 | |
| *** nati_ueno has quit IRC | 16:52 | |
| *** nati_ueno has joined #openstack-infra | 16:53 | |
| *** blamar has joined #openstack-infra | 16:56 | |
| *** ilyashakhat has joined #openstack-infra | 17:04 | |
| *** ogelbukh has quit IRC | 17:07 | |
| *** wenlock has joined #openstack-infra | 17:07 | |
| *** ilyashakhat_ has quit IRC | 17:08 | |
| *** metabro has quit IRC | 17:14 | |
| *** metabro has joined #openstack-infra | 17:16 | |
| *** dcramer_ has joined #openstack-infra | 17:17 | |
| *** plomakin has quit IRC | 17:19 | |
| *** plomakin has joined #openstack-infra | 17:21 | |
| *** ogelbukh has joined #openstack-infra | 17:23 | |
| *** dstanek has joined #openstack-infra | 17:23 | |
| *** dcramer_ has quit IRC | 17:23 | |
| *** yolanda has joined #openstack-infra | 17:23 | |
| *** wenlock has quit IRC | 17:25 | |
| *** yolanda has quit IRC | 17:28 | |
| *** crank has quit IRC | 17:30 | |
| *** crank has joined #openstack-infra | 17:34 | |
| *** dcramer_ has joined #openstack-infra | 17:35 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: proposal to add new jjb release managers group https://review.openstack.org/56823 | 17:37 |
| *** talluri has joined #openstack-infra | 17:38 | |
| *** wenlock has joined #openstack-infra | 17:40 | |
| *** dstanek has quit IRC | 17:44 | |
| *** dkliban has quit IRC | 17:49 | |
| *** wenlock has quit IRC | 17:55 | |
| *** DennyZhang has joined #openstack-infra | 18:08 | |
| *** dstanek has joined #openstack-infra | 18:08 | |
| *** metabro has quit IRC | 18:20 | |
| *** ilyashakhat has quit IRC | 18:23 | |
| lifeless | mordred: have you eyeballed http://code.google.com/p/vitess/ ? | 18:27 |
| mordred | nope | 18:27 |
| lifeless | I ratholed down into that following a language thingy | 18:27 |
| lifeless | but thought 'Oh, that might interest mordred' | 18:27 |
| mordred | looks at first light like mysql-proxy | 18:28 |
| mordred | yup. looks pretty much like they re-implemented jan's mysql-proxy | 18:29 |
| mordred | which is super helpful to lot of people in a lot of usecases | 18:29 |
| mordred | so vitess is proably also similarly really useful | 18:29 |
| lifeless | I thought mysql proxy took in mysql ? | 18:30 |
| mordred | it does - but no reason it has to - the other features on the list are where teh shared feature list is | 18:31 |
| mordred | and where the interesting bits are | 18:31 |
| lifeless | ack | 18:31 |
| lifeless | thanks | 18:31 |
| mordred | the architectural core concepts in vitess seem to be sound and battle-tested | 18:32 |
| mordred | https://github.com/youtube/vitess has more info | 18:34 |
| mordred | another zookeeper user - should I get over my aversion and learn to like zookeeper at some point? | 18:34 |
| *** zul has quit IRC | 18:37 | |
| *** SergeyLukjanov has quit IRC | 18:39 | |
| lifeless | mordred: I haven't yet. | 18:40 |
| lifeless | mordred: I may if it gets built in SSL | 18:40 |
| *** salv-orlando has quit IRC | 18:41 | |
| *** dstanek has quit IRC | 18:44 | |
| *** ogelbukh has quit IRC | 18:46 | |
| *** SergeyLukjanov has joined #openstack-infra | 18:48 | |
| *** plomakin has quit IRC | 18:50 | |
| *** dstanek has joined #openstack-infra | 18:56 | |
| *** DennyZha` has joined #openstack-infra | 18:59 | |
| *** EmilienM has quit IRC | 19:01 | |
| *** aardvark has joined #openstack-infra | 19:02 | |
| *** wenlock has joined #openstack-infra | 19:02 | |
| *** EmilienM has joined #openstack-infra | 19:02 | |
| *** DennyZhang has quit IRC | 19:04 | |
| *** WarrenUsui has quit IRC | 19:05 | |
| *** metabro has joined #openstack-infra | 19:07 | |
| *** plomakin has joined #openstack-infra | 19:07 | |
| *** DennyZha` has quit IRC | 19:07 | |
| *** ogelbukh has joined #openstack-infra | 19:07 | |
| *** thedodd has joined #openstack-infra | 19:14 | |
| *** thedodd has quit IRC | 19:17 | |
| *** thedodd has joined #openstack-infra | 19:18 | |
| *** thedodd has quit IRC | 19:21 | |
| *** thedodd has joined #openstack-infra | 19:21 | |
| clarkb | mordred: I think the gate trouble for havana is related to jog0's smoke-serial change | 19:24 |
| clarkb | mordred: I have just proposed a backport to stable/havana and am trying to sort out how to make it work with stable/grizzly though with grizzly it should already work I think | 19:25 |
| mordred | clarkb: awesome | 19:29 |
| clarkb | I think grizzly needed a change too | 19:30 |
| clarkb | so I pushed that | 19:30 |
| clarkb | mordred: https://review.openstack.org/#/c/56825/ and https://review.openstack.org/#/c/56826/ if those pass tests or at least change the failure then we are on the right track | 19:31 |
| *** dcramer_ has quit IRC | 19:33 | |
| *** yolanda has joined #openstack-infra | 19:33 | |
| *** SergeyLukjanov has quit IRC | 19:33 | |
| *** thedodd has quit IRC | 19:33 | |
| *** thedodd has joined #openstack-infra | 19:34 | |
| *** thedodd has quit IRC | 19:34 | |
| *** thedodd has joined #openstack-infra | 19:35 | |
| *** CaptTofu has quit IRC | 19:36 | |
| *** thedodd has quit IRC | 19:36 | |
| *** thedodd has joined #openstack-infra | 19:37 | |
| *** CaptTofu has joined #openstack-infra | 19:37 | |
| zaro | clarkb: I noticed that our gerrit project are not in heirarchy. | 19:37 |
| zaro | clarkb: do we NOT want to apply acls in heirarchy order? | 19:37 |
| clarkb | zaro: I am not parsing, you are talking about our fork of gerrit? and what hierarchy? | 19:38 |
| zaro | clarkb: no i'm talking about how our gerrit projects are setup in config. | 19:39 |
| zaro | clarkb: their's only 1 container project, 'All-Projects' | 19:39 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Add version override support from nova https://review.openstack.org/36511 | 19:39 |
| clarkb | we don't manage All-Projects in config iirc | 19:40 |
| clarkb | just the individual projects | 19:40 |
| zaro | clarkb: i'm thinking we would want to setup containers for 'openstack-infra', 'openstack-dev', etc.. | 19:40 |
| zaro | clarkb: then apply global acls to each container | 19:40 |
| clarkb | zaro: we could do that, not sure how much it helps beyond openstack-infra though. All of theo ther projects are pretty different | 19:40 |
| *** dstanek has quit IRC | 19:41 | |
| zaro | clarkb: would help to segregate global acls for each project type. | 19:41 |
| clarkb | zaro: but we don't need them to be segregated do we? | 19:41 |
| clarkb | my concern is that there are almost as many project types as there are projects | 19:41 |
| clarkb | so this won't simplify much | 19:41 |
| mordred | it could help some of them | 19:42 |
| zaro | clarkb: it looks like 'openstack-dev' and 'openstack-infra' have different global acl settings. | 19:42 |
| mordred | oslo, savana, fuel, manila all have multiple repos | 19:42 |
| zaro | clarkb: would it not make sense to setup global settings for each one? | 19:43 |
| clarkb | mordred: I think for the wrong reasons :) | 19:43 |
| mordred | clarkb: :) | 19:43 |
| mordred | otoh - we don't change them much either | 19:43 |
| mordred | I find the lack of templating in layout.yaml more upsetting regularly | 19:43 |
| clarkb | zaro: I definitely think openstack-infra is one place where it makes senese | 19:43 |
| mordred | yah. openstack-infra could use it | 19:43 |
| mordred | openstack-dev is all different | 19:44 |
| zaro | well it can't happen unless theere is some heirarchy. | 19:44 |
| zaro | what about the others? stackforge, and openstack | 19:45 |
| *** senk has joined #openstack-infra | 19:45 | |
| clarkb | stackforge and openstack are pretty different (particularly stackforge) | 19:45 |
| clarkb | since they are lots of disparate projects under a common prefix | 19:46 |
| clarkb | Hmm I should've put the bug number in my tempest changes too but I didn't want to change the commit message too much. | 19:46 |
| zaro | ok. i see. so i assume it doesn't make sense to introduce heirarchy for just openstack-infra? | 19:46 |
| clarkb | zaro: that is what I am thinking, but it might (I am not sure what sort of changes need to be made to have that happen) | 19:47 |
| *** talluri has quit IRC | 19:47 | |
| zaro | i think that would involve moving projects, so i'm not sure that's easily or safely done in gerrit. | 19:47 |
| mordred | nah | 19:48 |
| mordred | you just need an inheritfrom line | 19:48 |
| mordred | look at | 19:48 |
| mordred | for instance | 19:48 |
| mordred | ./modules/openstack_project/files/gerrit/acls/openstack/identity-api.config | 19:49 |
| clarkb | thats simple | 19:49 |
| zaro | i guess worth considering then. i think it makes a lot of sense. i just proposed a patch that sets up a new group. it would be good if setup with inheritance. | 19:50 |
| zaro | also i think patch security review gerrit would need changes to global acls, and i'm not sure it that would apply accross all proejcts. | 19:51 |
| clarkb | zaro: I think that is correct, global ACLs would need to be much more restricted by default (eg no read to anonymous users) | 19:51 |
| *** Alex_Gaynor has quit IRC | 19:55 | |
| *** Alex_Gaynor has joined #openstack-infra | 19:56 | |
| *** mriedem has joined #openstack-infra | 19:58 | |
| *** markwash has joined #openstack-infra | 20:04 | |
| mordred | O M G Gate Races | 20:06 |
| mordred | the gate races on this are kiling me: https://review.openstack.org/#/c/56752/ | 20:06 |
| clarkb | mordred: we need a support group | 20:12 |
| mordred | clarkb: we need for things to be less broken | 20:12 |
| clarkb | where we get together, flip all of the tables in the room, then hack on fixing the problems | 20:12 |
| mordred | yah | 20:12 |
| mordred | clarkb, sdague: so - more thoughts on removing tests | 20:13 |
| mordred | one of the resaons we clean warnings and turn them into errors and whatnot | 20:13 |
| *** wenlock has quit IRC | 20:13 | |
| mordred | is so that people don'tignore it when the system tells them something | 20:13 |
| *** yolanda has quit IRC | 20:14 | |
| mordred | I know the tests are pointing out real problems - but I'm concerned that continual recheck hell is going to undermine (or already has) people's reflex to see a negative response from the gate as aproblem to be fixed | 20:15 |
| mordred | so I know we talk about it all the time | 20:15 |
| mordred | but I want to make sure that when we frustratedly consider removing individual tests, that we consider that as one of the factors | 20:16 |
| clarkb | mordred: I definitely think the ability to force things through compounds the problem | 20:16 |
| clarkb | are you suggesting that we unforce thigns by removing particular tests? | 20:16 |
| mordred | I am | 20:16 |
| mordred | and then figure out a mechanism to be more strict on adding additoinal things in to make sure that they work | 20:17 |
| mordred | when we talk about 3rd party testing rigs, one of the design points that we bring up is ability to respond consistently without a high number of false negatives | 20:18 |
| clarkb | educating people on how to do some debugging themselves is what I would like to see | 20:18 |
| mordred | and we think that's a super important feature of the system | 20:18 |
| clarkb | the submit bug aginst ci then recheck issue is a symptom of folks not knowing what to do | 20:18 |
| clarkb | these are not false negatives though | 20:18 |
| mordred | I understand that - but they are false negatives on the patch | 20:19 |
| mordred | part of the point of the gate is to protect devs against other devs breaking the tree | 20:20 |
| mordred | but we managed to let tree breakages slip in | 20:20 |
| clarkb | right, and I think the minimal debug then recheck loop is a large contributor to that | 20:21 |
| mordred | so we're no longer sitting in the original design state, which is that errors from a test run should be errors caused by the developer's patch | 20:21 |
| clarkb | if your patch introduces a thing that passes 80% of the time, recheck will allow you to eventaully get it in | 20:21 |
| mordred | yup. I agree | 20:21 |
| mordred | problem is - you have to recheck EVERY patch | 20:22 |
| mordred | it's so frequent | 20:22 |
| mordred | and it's so frequently someone else's fault | 20:22 |
| mordred | that we're actually training people to not debug their own patches | 20:22 |
| clarkb | right, so I am proposing that we teach everyone how to properly deal with rechecks (or maybe just remove them all together) | 20:22 |
| clarkb | initially that will be super painful because of the state we are in now | 20:22 |
| clarkb | but after a couple weeks we should be in a better place assumign we actually fix the bugs | 20:23 |
| mordred | I think that's extremely unfair to most of our devs | 20:23 |
| clarkb | I don't think so | 20:23 |
| clarkb | this is a project we are all working together on | 20:23 |
| mordred | well, neutron already knows that they need to do a complete rewrite of their agent code | 20:23 |
| mordred | and that's apparently something that only 1 or 2 people can work on | 20:24 |
| mordred | is what we're saying that we need to do is hold everyone else up from being able to do work on other pieces while we wait on 2 guys to do that? | 20:24 |
| mordred | that seems baby-with-bathwater | 20:24 |
| clarkb | no, there are many many many other things that need fixing | 20:24 |
| clarkb | mordred: your change for example has nothing to do with neutron failures | 20:25 |
| mordred | my change has triggered like, 5 different failures at least | 20:25 |
| jesusaurus | are we tracking rechecks? do we have a per-project break-down of frequence or percentage of rechecks? im curious how wide-spread the problem is | 20:25 |
| mordred | we are | 20:25 |
| mordred | http://status.openstack.org/rechecks/ | 20:25 |
| mordred | and then there is elastic recheck | 20:26 |
| clarkb | we also have http://status.openstack.org/elastic-recheck/ tracking what we think are actual numbers for thigns that have had resources put into diagnosing them | 20:26 |
| mordred | but I'm growing more worried that what clarkb said above is a greater problem | 20:26 |
| mordred | whichis that, in a world where recheck is the norm | 20:26 |
| mikal | Can one of yo guys rescue me from IBMers who don't know how the internet works? | 20:26 |
| mordred | people will recheck their own buggy race condition code into the tree | 20:26 |
| clarkb | mikal: you assume we know how the internet works :) | 20:27 |
| clarkb | mordred: exactly | 20:27 |
| mikal | I replied to the thread "[OpenStack-Infra] Request a a dedicated user account." from an IBMer asking for a gerrit account | 20:27 |
| mikal | And now they think I'm the guy setting it up | 20:27 |
| mikal | And are emailing me personally | 20:27 |
| clarkb | mikal: oh that one, I was so happy you responded to that :) sure point them back at the list and I can respond to what ends up there | 20:27 |
| mordred | and that will happen with higher velocity than we can fix the current outstaning bugs | 20:27 |
| mikal | My sin was replying to their request saying I thought "root" was a bad account name to be request | 20:27 |
| mikal | ing | 20:27 |
| mordred | mikal: thanks for that, btw | 20:27 |
| clarkb | mikal: it was a bad account name :) | 20:27 |
| mikal | Oh, I know I was right | 20:28 |
| mikal | I'm now just being punished | 20:28 |
| mikal | I've told them to go back to the list | 20:28 |
| mikal | I'd ask an IBMer with clue to lean in and help, but I think they've all quit (except cyeoh, but I think he's a different team) | 20:29 |
| clarkb | mtreinish is on vacation | 20:29 |
| jesusaurus | mordred: oh, i see, you're talking about preventing future intermittent bugs | 20:29 |
| mordred | jesusaurus: and dealing with the current ones too | 20:29 |
| jesusaurus | clarkb: is this why you mentioned running tests multiple times the other day? | 20:29 |
| mordred | yah | 20:29 |
| clarkb | jesusaurus: yup | 20:29 |
| clarkb | after feature freeze a bunch of stuff was fixed | 20:30 |
| clarkb | and we got the rates of failure to be pretty low, but since then we are back up to crazy amounts of hurt | 20:30 |
| clarkb | and I think it is because we have a mechanism to get half passing code in | 20:30 |
| clarkb | mordred: what if we get rid of reverify | 20:31 |
| clarkb | mordred: force cores to look at the failures and decide if a second +A is appropriate | 20:31 |
| clarkb | then you can recheck all you want and build up data on a change | 20:31 |
| mordred | hrm | 20:31 |
| clarkb | but code doesn't go into gate without someone actually thinking about it | 20:31 |
| mordred | I don't think that will have the result you think it will | 20:32 |
| mordred | there are over 100 cores | 20:32 |
| clarkb | ya good point | 20:32 |
| mordred | I'm honestly stumped. I hate all of the options | 20:33 |
| mordred | I do not think us taking a hard-line is working | 20:33 |
| *** dstanek has joined #openstack-infra | 20:33 | |
| mordred | I do not think cancelling the gate will work or be a good thing | 20:33 |
| lifeless | oh hai | 20:33 |
| mordred | I concede that disabling tests is a bad idea | 20:34 |
| lifeless | etoomucscrollback | 20:34 |
| lifeless | how reliable do we want the gate to be ? | 20:34 |
| mordred | I think us 'educating' people is not just not working, but i also a dangerous position to sit in | 20:34 |
| mordred | lifeless: we want it to be 100% reliable | 20:34 |
| mordred | ultimate goal, no failures that are not caused by the dev's patch | 20:35 |
| clarkb | and that number depends on perspective right | 20:35 |
| mordred | that may be a nirvana of course | 20:35 |
| clarkb | these are not really false negatives so it is reliable in that case | 20:35 |
| lifeless | ideally sure, but pragmatically? How many reverifies a day would we accept ? 10? 20 ? 5? | 20:35 |
| clarkb | but there are a lot of unrelated patches being hurt by the pain | 20:35 |
| lifeless | diminishing returns and all that | 20:35 |
| lifeless | Second question, how long does it take to detect a fragile change has landed ? | 20:35 |
| clarkb | lifeless: a bit of time as humans need to go back and identify the problem, find when it started and fix it | 20:36 |
| clarkb | its usually a few days | 20:36 |
| clarkb | mikal found one that started midweek yesterday | 20:36 |
| * mordred cries | 20:36 | |
| mordred | I do like the ideaof ratcheting up iterations | 20:36 |
| mordred | to increase surface area possibility of catching them | 20:36 |
| mordred | but that is, of course, expensive | 20:37 |
| mikal | clarkb: I think I need a login on a box which is experiencing that failure to debug further | 20:37 |
| mikal | I'm not sure if its a libvirt problem at this point | 20:37 |
| mordred | and still sits us inthe same place that removing rechecks do | 20:37 |
| clarkb | mikal: ok | 20:37 |
| lifeless | mordred: probability of a problem occuring that isn't detected in N runs is N/3 | 20:37 |
| mikal | Or I can add more debugging code to nova I suppose | 20:38 |
| mikal | The basic problem seems to be very slow or empty console logs | 20:38 |
| clarkb | mikal: that might be an easy first step | 20:38 |
| mikal | Sometimes tempest times out with just have of the cirros boot done | 20:38 |
| mordred | whichis that we are currently in a world of broken, and if we all of a sudden make everything run 10x, we will trigger failure more often, efectively stopping everyone | 20:38 |
| mikal | Sometimes nothing has hit the console log at all | 20:38 |
| clarkb | mikal: interesting | 20:38 |
| lifeless | mordred: that result will be needed if you want to ratchet up runs to reduce flakiness: unless we ratchet it up enough to force failures under the current failure rate, we'll gain nothing | 20:38 |
| mordred | yup | 20:39 |
| mikal | clarkb: so I do wonder if there is something environmental which is causing libvirt to be slow on some test nodes | 20:39 |
| mordred | so my suggestion would be | 20:39 |
| clarkb | mikal: could be something related to the underlying cloud | 20:39 |
| lifeless | e.g. 10 runs does not get you a 10% failure rtte | 20:39 |
| lifeless | its gets you 30% | 20:39 |
| mordred | ratched it up to trigger point | 20:39 |
| lifeless | 30 runs will get you 10% | 20:39 |
| lifeless | 60 runs 5% | 20:39 |
| *** markwash has quit IRC | 20:40 | |
| mordred | then disable the blergy tests, then work on getting them and fixes for them back in under the higher ratchet | 20:40 |
| mordred | but - its a terrible idea | 20:40 |
| lifeless | I think we have to expect this to happen and engineer to manage it | 20:40 |
| lifeless | e.g. drive the TTD down very low | 20:40 |
| mordred | TTD=? | 20:40 |
| clarkb | time to detection? | 20:43 |
| clarkb | I think we help time to detection if we kill easy rechecks | 20:43 |
| clarkb | because it will force people to debug | 20:43 |
| jesusaurus | how do you define easy? | 20:43 |
| mordred | I just don't think it will do that | 20:44 |
| mordred | I think it will cause people to file more bugs against us | 20:44 |
| mordred | and I think it will cause more people to block on 'central' people | 20:44 |
| clarkb | jesusaurus: anyone being able to do it | 20:44 |
| jesusaurus | that would definitely put a lot of pressure on those able to recheck | 20:46 |
| clarkb | I agree and like I said earlier it will suck short term | 20:46 |
| lifeless | once we have it detected, then perhaps locking the trees into release mode and fixing it, then resuming normal process is sane | 20:47 |
| clarkb | but should give us breathing room to shake the current problems out without adding heaps more | 20:47 |
| lifeless | or | 20:47 |
| lifeless | revert the commit | 20:47 |
| lifeless | if we get the TTD low enough | 20:47 |
| mordred | but what do we do about the current set of fails? | 20:47 |
| clarkb | mordred: basically what we did around havana feature freeze | 20:48 |
| lifeless | if we can track down the commit, proposing a revert and seeing if it passes would be a good start | 20:49 |
| mordred | clarkb: the problem is, that clearly is not a maintainable state, because in a week where a couple of core people were on vacation, the entire project regressed | 20:49 |
| lifeless | we capture enough data | 20:49 |
| lifeless | that getting a low TTD should be possible | 20:49 |
| *** salv-orlando has joined #openstack-infra | 20:50 | |
| clarkb | mordred: it isn't sustainable, but hopefulyl you get enough changes in place that now you can breath and prevent future breakage of the same sort | 20:50 |
| clarkb | either by running more tests or no more recheks or removing tests that are particularl;y problematic | 20:50 |
| lifeless | clarkb: whats the current aggregate failure rate? | 20:50 |
| clarkb | I just think this is a race we can't get ahead in without drastic measures | 20:50 |
| mordred | or some combo of all of the above | 20:50 |
| mordred | lifeless: right now it's LUDICROUS | 20:50 |
| clarkb | lifeless: it is ~50% according to jog0 iirc | 20:51 |
| mikal | clarkb: it occurs to me one question I haven't asked is does that failure happen for anyone else's change? | 20:52 |
| lifeless | ok, so running 6 jobs (which might mean just 3 times the current tempest jobs?) should put an effective barrier stopping that getting worse [95% of the time 6 runs will detect anything with an incident rate higher than 50%] | 20:52 |
| mikal | Perhaps my change very subtly breaks libvirt booting in a way I don't expect | 20:52 |
| lifeless | but again, I think the stats are against us | 20:52 |
| mikal | (Unlikely, but possible) | 20:52 |
| mordred | lifeless: well, it's not a problem with a 50% failure rate - it's mutliple problems with an average effective 50% failure rate | 20:53 |
| lifeless | It seems to me that leveraging the vast amount of data we already gather to detect failures fast + revert is better | 20:53 |
| clarkb | mikal: did you find ~90 failures over the last week | 20:53 |
| lifeless | mordred: yes, I know that | 20:53 |
| clarkb | mikal: I thought you had a query for it | 20:53 |
| lifeless | mordred: but you can invert the thing and look for successful runs that shouldn't have succeeded and it becomes easier to think about | 20:54 |
| mordred | lifeless: well, and we have plans to work on all of this | 20:54 |
| lifeless | anyhow | 20:54 |
| lifeless | what can I do to help? | 20:54 |
| fungi | i'm wondering if elastic-recheck can't get to the point where we simply don't recheck a failed change unless it's already known to and tracked within e-r | 20:54 |
| clarkb | fungi: OOOHHHHH | 20:55 |
| mordred | both faster detectio, and offline analysis of the corpus | 20:55 |
| clarkb | quick someone give that man a beer | 20:55 |
| fungi | and to recheck anything else it has to get researched and added to e-r first | 20:55 |
| jesusaurus | +1 | 20:55 |
| fungi | please note i haven't thought through the implications there, and it is the weekend still here, so it might be a terrible, terrible idea after all | 20:55 |
| clarkb | fungi: I think it is the best one I have heard so far | 20:56 |
| mordred | starting with patchset 6 this morning: https://review.openstack.org/#/c/56752/ | 20:56 |
| mordred | 5 failures, all unknown, all unrelated to the patch in question | 20:56 |
| fungi | i've been heads down hacking on arm hardware all weekend with hardly a break, so need to reset my brain to the topic anyway | 20:56 |
| mordred | fungi: nice! | 20:56 |
| mordred | one of them had two different failures even | 20:56 |
| mordred | and I'm not saying we have to fix it for me - just merely that I've got a data point we can look at | 20:57 |
| mikal | clarkb: ther's a query in the bug I filed | 20:58 |
| mikal | So yeah, perhaps I'm innocent | 20:58 |
| mikal | Which would be nice | 20:58 |
| mikal | clarkb: I'm on the road tomorrow. I'll spend some more time digging into the failure I found when I get home tonight. | 21:01 |
| mikal | Unless someone beats me to it | 21:02 |
| clarkb | I probably won't beat you to it, today is my day of being lazy on a couch | 21:03 |
| mikal | Fair enough | 21:03 |
| mikal | One can hope that lifeles will fix it though | 21:03 |
| mordred | lifeless: yeah - if you could just take two days off and fix all the bugs, that would be great | 21:04 |
| mikal | GET ON THAT | 21:04 |
| lifeless | mordred: let me just cellotape C to the ceiling and put on noise cancelling headphones. | 21:06 |
| mordred | lifeless: I'm fine with both of those things | 21:06 |
| mordred | especially use of the very cute word cellotape | 21:06 |
| lifeless | fungi: I likes. | 21:06 |
| * mikal goes to the doctor | 21:09 | |
| * fungi goes to find post-hacking beer^Wdinner | 21:21 | |
| *** loq_mac has joined #openstack-infra | 21:22 | |
| *** Loquacity has joined #openstack-infra | 21:24 | |
| *** markwash has joined #openstack-infra | 21:28 | |
| *** mkoderer has quit IRC | 21:31 | |
| *** sdake has quit IRC | 21:33 | |
| * mordred goes ... | 21:34 | |
| * mordred goes nowhere. he's in a plane seat. there is nowhere to go. | 21:34 | |
| *** mkoderer_ has joined #openstack-infra | 21:34 | |
| *** mkoderer_ is now known as mkoderer | 21:34 | |
| openstackgerrit | Masayuki Igawa proposed a change to openstack-infra/elastic-recheck: Add query for bug 1251999 https://review.openstack.org/56839 | 21:34 |
| *** sdake__ has joined #openstack-infra | 21:34 | |
| *** dstanek has quit IRC | 21:43 | |
| *** dstanek has joined #openstack-infra | 21:50 | |
| *** dstanek has quit IRC | 21:54 | |
| *** DennyZhang has joined #openstack-infra | 21:58 | |
| *** slong has joined #openstack-infra | 22:01 | |
| *** jhesketh has joined #openstack-infra | 22:02 | |
| *** dcramer_ has joined #openstack-infra | 22:06 | |
| *** mriedem has quit IRC | 22:06 | |
| *** cody-somerville has quit IRC | 22:09 | |
| *** markwash has quit IRC | 22:09 | |
| *** thedodd has quit IRC | 22:23 | |
| *** zul has joined #openstack-infra | 22:23 | |
| *** jamesmcarthur has joined #openstack-infra | 22:24 | |
| *** dcramer_ has quit IRC | 22:25 | |
| *** DennyZhang has quit IRC | 22:27 | |
| *** DennyZhang has joined #openstack-infra | 22:28 | |
| *** nati_ueno has quit IRC | 22:37 | |
| *** senk has quit IRC | 22:38 | |
| *** dcramer_ has joined #openstack-infra | 22:39 | |
| *** CaptTofu has quit IRC | 22:42 | |
| *** CaptTofu has joined #openstack-infra | 22:43 | |
| *** dstanek has joined #openstack-infra | 22:52 | |
| *** Ryan_Lane has joined #openstack-infra | 22:53 | |
| *** Ryan_Lane has joined #openstack-infra | 22:53 | |
| *** DennyZhang has quit IRC | 23:02 | |
| *** paul-- has joined #openstack-infra | 23:03 | |
| *** herndon has joined #openstack-infra | 23:06 | |
| *** loq_mac has quit IRC | 23:06 | |
| *** loq_mac has joined #openstack-infra | 23:07 | |
| *** loq_mac has quit IRC | 23:07 | |
| *** salv-orlando has quit IRC | 23:07 | |
| *** salv-orlando has joined #openstack-infra | 23:07 | |
| *** jamesmcarthur has quit IRC | 23:11 | |
| *** wenlock has joined #openstack-infra | 23:14 | |
| *** dripton has quit IRC | 23:14 | |
| *** dripton has joined #openstack-infra | 23:15 | |
| *** michchap has quit IRC | 23:18 | |
| *** dkliban has joined #openstack-infra | 23:19 | |
| *** michchap has joined #openstack-infra | 23:21 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add new core reviewers https://review.openstack.org/56806 | 23:21 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add rubick project tracking https://review.openstack.org/56808 | 23:22 |
| *** bknudson has quit IRC | 23:26 | |
| *** michchap has quit IRC | 23:34 | |
| *** michchap has joined #openstack-infra | 23:35 | |
| *** thomasem has joined #openstack-infra | 23:36 | |
| *** dstanek has quit IRC | 23:41 | |
| openstackgerrit | Noorul Islam K M proposed a change to openstack-infra/reviewstats: Add python-solumclient subproject https://review.openstack.org/56810 | 23:43 |
| *** jamesmcarthur has joined #openstack-infra | 23:44 | |
| *** michchap has quit IRC | 23:48 | |
| *** dstanek has joined #openstack-infra | 23:53 | |
| *** michchap has joined #openstack-infra | 23:54 | |
| *** talluri has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!