| *** dcramer_ has joined #openstack-infra | 00:00 | |
| *** mrodden has joined #openstack-infra | 00:01 | |
| *** jroovers has quit IRC | 00:02 | |
| *** pabelanger has quit IRC | 00:03 | |
| *** herndon_ has quit IRC | 00:07 | |
| *** rfolco has quit IRC | 00:13 | |
| *** pabelanger has joined #openstack-infra | 00:24 | |
| *** nati_ueno has quit IRC | 00:24 | |
| *** nati_ueno has joined #openstack-infra | 00:25 | |
| *** saper has joined #openstack-infra | 00:25 | |
| *** pabelanger has quit IRC | 00:25 | |
| *** rfolco has joined #openstack-infra | 00:26 | |
| *** rnirmal has quit IRC | 00:27 | |
| *** pabelanger has joined #openstack-infra | 00:27 | |
| *** pabelanger has quit IRC | 00:27 | |
| *** pabelanger has joined #openstack-infra | 00:28 | |
| fungi | clarkb: looking | 00:28 |
|---|---|---|
| *** toddmorey has joined #openstack-infra | 00:29 | |
| *** matsuhashi has quit IRC | 00:29 | |
| *** nati_ueno has quit IRC | 00:30 | |
| *** matsuhashi has joined #openstack-infra | 00:30 | |
| fungi | clarkb: lgtm. i guess we weren't actually re-exec'ing d-g | 00:31 |
| clarkb | fungi: we were but the rexec was happening on the old code | 00:32 |
| fungi | got it | 00:33 |
| *** matsuhashi has quit IRC | 00:34 | |
| *** ^demon|away has quit IRC | 00:38 | |
| *** fifieldt has joined #openstack-infra | 00:40 | |
| *** toddmorey has quit IRC | 00:43 | |
| *** rfolco has quit IRC | 00:44 | |
| *** scairbus has joined #openstack-infra | 00:48 | |
| *** toddmorey has joined #openstack-infra | 00:48 | |
| *** sdake has quit IRC | 00:57 | |
| *** CaptTofu has quit IRC | 00:58 | |
| *** CaptTofu has joined #openstack-infra | 00:58 | |
| *** sdake has joined #openstack-infra | 01:00 | |
| *** sdake has quit IRC | 01:00 | |
| *** sdake has joined #openstack-infra | 01:00 | |
| *** nati_ueno has joined #openstack-infra | 01:02 | |
| *** nosnos has joined #openstack-infra | 01:05 | |
| *** nosnos has quit IRC | 01:05 | |
| *** nosnos has joined #openstack-infra | 01:06 | |
| *** herndon_ has joined #openstack-infra | 01:07 | |
| *** matsuhashi has joined #openstack-infra | 01:07 | |
| *** ^demon|away has joined #openstack-infra | 01:07 | |
| jog0 | any easy way to find the failure rate for tempest? | 01:09 |
| jog0 | or how many revisions it takes before merging something | 01:11 |
| jog0 | sdague: ^, I am playing with gate math | 01:12 |
| jog0 | and trying to get numbers that are accurate | 01:12 |
| openstackgerrit | DennyZhang proposed a change to openstack-infra/config: defensive coding for docbook-properties.sh https://review.openstack.org/54642 | 01:14 |
| sdague | jog0: my brain power is expended, but I was going to dig in the morning | 01:19 |
| sdague | to do just that | 01:19 |
| *** toddmorey has quit IRC | 01:19 | |
| jog0 | any graphite gurus here? | 01:19 |
| sdague | jog0: you need the answer tonight? or will late tomorrow suffice/ | 01:20 |
| *** mrodden has quit IRC | 01:20 | |
| openstackgerrit | Hunter Haugen proposed a change to openstack-infra/config: Pass $mysql_password through to gerrit class https://review.openstack.org/56306 | 01:21 |
| jog0 | i'll keep dinking a round | 01:21 |
| *** dims has quit IRC | 01:21 | |
| *** UtahDave has quit IRC | 01:22 | |
| reed | lots of thoughts on the proposal raised by nijaba :) | 01:23 |
| *** wenlock has quit IRC | 01:23 | |
| *** ljjjustin has joined #openstack-infra | 01:25 | |
| *** guohliu has quit IRC | 01:30 | |
| *** rhsu has quit IRC | 01:33 | |
| *** jamesmcarthur has joined #openstack-infra | 01:35 | |
| *** dims has joined #openstack-infra | 01:36 | |
| pleia2 | reed: I can only really talk about infra/qa stuff (so having more folks there as well would be good), but if you saw my blog I'm in a good position to do so and my schedule is wide open next week | 01:38 |
| *** saper has quit IRC | 01:39 | |
| reed | pleia2, great! | 01:39 |
| *** markwash has quit IRC | 01:40 | |
| *** saper has joined #openstack-infra | 01:40 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** jamesmcarthur has joined #openstack-infra | 01:50 | |
| *** herndon_ has quit IRC | 01:51 | |
| *** ^demon|away has quit IRC | 02:00 | |
| *** mriedem has joined #openstack-infra | 02:03 | |
| *** melwitt has quit IRC | 02:07 | |
| *** yaguang has joined #openstack-infra | 02:10 | |
| *** markwash has joined #openstack-infra | 02:11 | |
| *** guohliu has joined #openstack-infra | 02:20 | |
| *** toddmorey has joined #openstack-infra | 02:20 | |
| *** rhsu has joined #openstack-infra | 02:26 | |
| *** jerryz has quit IRC | 02:28 | |
| *** giulivo has left #openstack-infra | 02:29 | |
| *** wenlock has joined #openstack-infra | 02:30 | |
| jog0 | http://graphite.openstack.org/graphlot/?lineMode=staircase&from=-14days&connectedLimit=&height=308&width=586&_salt=1384396291.739&target=movingAverage(asPercent(stats.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.FAILURE%2Csum(stats.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.%7BSUCCESS%2CFAILURE%7D))%2C2000)&target=movingAverage(asPercent(stats.zuul.pipeline.check.job.check-tempest-devstack-vm-neutron.FAILURE%2Csum(stats.zuul.pipel | 02:32 |
| jog0 | purty | 02:32 |
| clarkb | jog0: the irc byte limit cut you off | 02:33 |
| clarkb | any chance you can tinyurl it? | 02:33 |
| jog0 | http://tinyurl.com/ly32azd | 02:33 |
| *** nati_ueno has quit IRC | 02:34 | |
| clarkb | is that showing a ratio | 02:34 |
| clarkb | oh it is a percent | 02:34 |
| jog0 | y axis is percentege | 02:34 |
| jog0 | I haven't figured out how to set the label yet | 02:34 |
| *** nati_ueno has joined #openstack-infra | 02:35 | |
| *** odyi has quit IRC | 02:35 | |
| *** vipul is now known as vipul-away | 02:35 | |
| clarkb | its interesting that the check percentage is climbing steadily (I suppose that indicates that people are pushing bad code?) while the gate is relatively flat | 02:36 |
| *** oubiwann has quit IRC | 02:36 | |
| jog0 | yeah | 02:36 |
| jog0 | hold on more sexy coming soon | 02:37 |
| *** krtaylor has joined #openstack-infra | 02:37 | |
| *** nati_uen_ has joined #openstack-infra | 02:37 | |
| jog0 | http://tinyurl.com/mqju53r | 02:37 |
| *** nati_ueno has quit IRC | 02:38 | |
| *** esker has joined #openstack-infra | 02:38 | |
| *** odyi has joined #openstack-infra | 02:38 | |
| jog0 | gate and check neutron and full | 02:39 |
| jog0 | gate and check * neutron and full | 02:39 |
| jog0 | this is pretty slick | 02:40 |
| *** markwash has quit IRC | 02:41 | |
| *** dcramer_ has quit IRC | 02:41 | |
| jog0 | devstack-full-check has a 10* failure rate wow | 02:41 |
| jog0 | 10% | 02:41 |
| clarkb | thats the swift stuff | 02:42 |
| jog0 | clarkb: yeah hopefully we fixed that | 02:42 |
| clarkb | though that particular bug seems to maybe be falling off after the bump to 6GB | 02:42 |
| clarkb | ya | 02:42 |
| jog0 | to the Ml | 02:42 |
| jog0 | ML, going to send out a gate math email | 02:42 |
| *** vipul-away is now known as vipul | 02:43 | |
| *** che-arne has quit IRC | 02:43 | |
| jog0 | clarkb: recheck because its not your fault per patch = failure_rate * tempest_jobs * patch_revisions | 02:43 |
| *** nati_uen_ has quit IRC | 02:44 | |
| jog0 | and with failure as high as 10% | 02:44 |
| jog0 | and 4 tempest jobs | 02:44 |
| jog0 | and a few revisions | 02:44 |
| jog0 | it gets bad | 02:44 |
| *** jamesmcarthur has quit IRC | 02:44 | |
| *** nati_ueno has joined #openstack-infra | 02:44 | |
| clarkb | ya | 02:45 |
| clarkb | the probability of a change failing is the sum of the probabilities that any given test fails because the tests are independent (sort of because bad changes skew the numbers as they aren't actually independent) | 02:46 |
| jog0 | yup | 02:47 |
| clarkb | but if we are just looking at the gate we should get fairly reasonable probabilities out of the gate since in theory those are all good changes | 02:47 |
| jog0 | so in the example its 0.1*4*5 | 02:47 |
| jog0 | for 5 revisions | 02:47 |
| jog0 | or two recehcks | 02:47 |
| jog0 | I like now neutron gate got much better this week | 02:48 |
| jog0 | actually both gates did | 02:48 |
| *** nati_ueno has quit IRC | 02:48 | |
| *** Ryan_Lane has quit IRC | 02:48 | |
| *** Ryan_Lane has joined #openstack-infra | 02:48 | |
| *** scairbus has quit IRC | 02:51 | |
| *** Ryan_Lane has quit IRC | 02:53 | |
| *** jamesmcarthur has joined #openstack-infra | 02:53 | |
| *** dcramer_ has joined #openstack-infra | 02:54 | |
| jog0 | http://lists.openstack.org/pipermail/openstack-dev/2013-November/019042.html | 03:16 |
| jog0 | lifeless: hows that instead of a syntax thread | 03:16 |
| clarkb | jog0: its ok lifeless has his own synatx threads ;) | 03:17 |
| jog0 | I noticed :) | 03:17 |
| *** michchap has quit IRC | 03:19 | |
| *** michchap has joined #openstack-infra | 03:19 | |
| dims | jog0, added tests in https://review.openstack.org/#/c/56075/ - thanks for the feedback! | 03:20 |
| lifeless | jog0: aaaaa | 03:23 |
| lifeless | jog0: bah, failnet. | 03:23 |
| lifeless | jog0: nice. | 03:23 |
| lifeless | jog0: I'd still like to remove the period check from hacking; think there is enough projects ignoring it now? | 03:23 |
| lifeless | jog0: since you brought up syntax :) | 03:24 |
| *** toddmorey has quit IRC | 03:24 | |
| * jog0 walks away from the discussion to eat dinner | 03:25 | |
| *** toddmorey has joined #openstack-infra | 03:26 | |
| *** mriedem has quit IRC | 03:29 | |
| *** dims has quit IRC | 03:36 | |
| jog0 | lifeless: I agree with your two comments by the way. but to clarify the 5% isn't one bug but a collection of bugs | 03:37 |
| *** dims has joined #openstack-infra | 03:37 | |
| openstackgerrit | Noorul Islam K M proposed a change to openstack-dev/cookiecutter: Adjust _TRUE_VALUES to be consistent with OpenStack projects. https://review.openstack.org/56318 | 03:37 |
| *** slong has joined #openstack-infra | 03:38 | |
| *** matsuhashi has quit IRC | 03:39 | |
| *** slong_ has joined #openstack-infra | 03:39 | |
| *** matsuhashi has joined #openstack-infra | 03:40 | |
| *** slong has quit IRC | 03:43 | |
| *** nati_ueno has joined #openstack-infra | 03:44 | |
| *** matsuhashi has quit IRC | 03:44 | |
| *** DennyZhang has joined #openstack-infra | 03:45 | |
| *** ^d has joined #openstack-infra | 03:47 | |
| *** ^d has joined #openstack-infra | 03:47 | |
| *** nati_ueno has quit IRC | 03:48 | |
| clarkb | jhesketh: are you going to stackforgify turbo-hipster? | 03:51 |
| *** Ryan_Lane has joined #openstack-infra | 03:52 | |
| jhesketh__ | clarkb: already done | 03:52 |
| clarkb | jhesketh: if you do that the process will create a new group for you and we will add you to the group at that time | 03:52 |
| clarkb | oh | 03:52 |
| * clarkb looks better | 03:52 | |
| jhesketh__ | I'm not sure if gerrit has been updated though as I can't see the group | 03:52 |
| *** sandywalsh has quit IRC | 03:52 | |
| clarkb | ah | 03:52 |
| clarkb | yeah ok I don't see the project yet either | 03:52 |
| jhesketh__ | clarkb: https://review.openstack.org/#/c/56180/ | 03:53 |
| *** nati_ueno has joined #openstack-infra | 03:53 | |
| *** michchap_ has joined #openstack-infra | 03:53 | |
| clarkb | jhesketh__: I think manage projects is still having trouble | 03:53 |
| clarkb | it didn't even create the project on github | 03:54 |
| *** DennyZhang has quit IRC | 03:54 | |
| *** DennyZhang has joined #openstack-infra | 03:54 | |
| clarkb | jhesketh__: I will poke at it in the morning. Feel free to bug me tomorrow if it isn't working by then | 03:54 |
| jhesketh__ | clarkb: no worries, Thanks :-) | 03:54 |
| jhesketh__ | clarkb: Just be careful of this one when you update things: https://review.openstack.org/#/c/56179/2 | 03:55 |
| *** HenryG has quit IRC | 03:55 | |
| clarkb | ok, I will probably kick it on a just turbohipster until it works | 03:55 |
| *** michchap has quit IRC | 03:55 | |
| jhesketh__ | sorry? | 03:57 |
| clarkb | oh we can run the automagic stuff on specific projects | 03:57 |
| clarkb | so I will ignore the reordering for now :) | 03:57 |
| clarkb | sorry english it fails me after dinner | 03:58 |
| jhesketh__ | ah, no worries, sounds good | 03:58 |
| jhesketh__ | english fails me at all times | 03:58 |
| clarkb | though maybe I should bug mordred do look at it since he approved the cahnges :) | 03:58 |
| *** nati_ueno has quit IRC | 03:59 | |
| *** nati_ueno has joined #openstack-infra | 04:00 | |
| clarkb | I thought this was going to be an easy bug to close :) | 04:01 |
| *** michchap has joined #openstack-infra | 04:02 | |
| *** CaptTofu has quit IRC | 04:02 | |
| *** CaptTofu has joined #openstack-infra | 04:03 | |
| *** nati_uen_ has joined #openstack-infra | 04:04 | |
| *** nati_ueno has quit IRC | 04:04 | |
| *** nati_uen_ has quit IRC | 04:04 | |
| *** michchap_ has quit IRC | 04:04 | |
| *** nati_ueno has joined #openstack-infra | 04:05 | |
| jhesketh__ | sorry! :-( | 04:05 |
| *** sandywalsh has joined #openstack-infra | 04:06 | |
| *** nati_ueno has quit IRC | 04:09 | |
| *** vipul has quit IRC | 04:09 | |
| *** nati_ueno has joined #openstack-infra | 04:09 | |
| *** vipul has joined #openstack-infra | 04:10 | |
| *** pabelanger has quit IRC | 04:15 | |
| *** matsuhashi has joined #openstack-infra | 04:17 | |
| *** talluri has joined #openstack-infra | 04:19 | |
| *** masayukig has quit IRC | 04:21 | |
| *** balar has quit IRC | 04:21 | |
| *** balar has joined #openstack-infra | 04:22 | |
| *** dims has quit IRC | 04:23 | |
| *** aardvark has quit IRC | 04:24 | |
| *** aardvark has joined #openstack-infra | 04:24 | |
| *** boris-42 has joined #openstack-infra | 04:25 | |
| *** chandankumar has quit IRC | 04:25 | |
| *** chandankumar has joined #openstack-infra | 04:26 | |
| *** talluri has quit IRC | 04:26 | |
| *** ^d has quit IRC | 04:27 | |
| *** marun has joined #openstack-infra | 04:40 | |
| *** ljjjustin has quit IRC | 04:42 | |
| *** talluri has joined #openstack-infra | 04:44 | |
| *** davidhadas_ has joined #openstack-infra | 04:47 | |
| *** nati_ueno has quit IRC | 04:49 | |
| *** davidhadas has quit IRC | 04:49 | |
| *** nati_ueno has joined #openstack-infra | 04:49 | |
| *** nati_ueno has quit IRC | 04:53 | |
| *** romcheg has joined #openstack-infra | 04:58 | |
| *** talluri has quit IRC | 04:59 | |
| *** matsuhashi has quit IRC | 05:01 | |
| *** matsuhashi has joined #openstack-infra | 05:02 | |
| *** matsuhashi has quit IRC | 05:02 | |
| *** matsuhashi has joined #openstack-infra | 05:07 | |
| *** jamesmcarthur has quit IRC | 05:14 | |
| *** esker has quit IRC | 05:16 | |
| *** esker has joined #openstack-infra | 05:16 | |
| *** nati_ueno has joined #openstack-infra | 05:18 | |
| *** talluri has joined #openstack-infra | 05:20 | |
| *** vipul is now known as vipul-away | 05:20 | |
| *** UtahDave has joined #openstack-infra | 05:23 | |
| *** nati_ueno has quit IRC | 05:24 | |
| *** sarob has joined #openstack-infra | 05:24 | |
| *** nati_ueno has joined #openstack-infra | 05:24 | |
| *** sarob has quit IRC | 05:26 | |
| *** sarob has joined #openstack-infra | 05:27 | |
| *** scairbus has joined #openstack-infra | 05:28 | |
| *** sarob has joined #openstack-infra | 05:28 | |
| *** nati_ueno has quit IRC | 05:29 | |
| *** sarob has quit IRC | 05:32 | |
| *** DennyZha` has joined #openstack-infra | 05:33 | |
| *** DennyZhang has quit IRC | 05:36 | |
| *** vipul-away is now known as vipul | 05:37 | |
| *** nicedice_ has quit IRC | 05:37 | |
| *** reed has quit IRC | 05:43 | |
| *** toddmorey has quit IRC | 05:44 | |
| *** vipul is now known as vipul-away | 05:52 | |
| *** Ryan_Lane has quit IRC | 05:55 | |
| *** Ryan_Lane has joined #openstack-infra | 05:56 | |
| *** vipul-away is now known as vipul | 05:56 | |
| *** matsuhashi has quit IRC | 06:02 | |
| *** toddmorey has joined #openstack-infra | 06:02 | |
| *** matsuhashi has joined #openstack-infra | 06:02 | |
| *** afazekas has joined #openstack-infra | 06:03 | |
| *** matsuhashi has quit IRC | 06:07 | |
| *** matsuhashi has joined #openstack-infra | 06:07 | |
| *** resker has joined #openstack-infra | 06:09 | |
| *** bingbu has joined #openstack-infra | 06:09 | |
| *** scairbus has quit IRC | 06:11 | |
| *** matsuhashi has quit IRC | 06:11 | |
| *** esker has quit IRC | 06:12 | |
| *** matsuhashi has joined #openstack-infra | 06:12 | |
| *** matsuhashi has quit IRC | 06:16 | |
| *** matsuhashi has joined #openstack-infra | 06:17 | |
| *** mrmartin has joined #openstack-infra | 06:23 | |
| *** xchu has joined #openstack-infra | 06:23 | |
| *** romcheg has quit IRC | 06:24 | |
| *** ogelbukh1 has quit IRC | 06:27 | |
| *** dcramer_ has quit IRC | 06:28 | |
| *** yaguang has quit IRC | 06:28 | |
| *** jhesketh__ has quit IRC | 06:28 | |
| *** salv-orlando has quit IRC | 06:28 | |
| *** freyes has quit IRC | 06:28 | |
| *** zoresvit has quit IRC | 06:28 | |
| *** Shrews has quit IRC | 06:28 | |
| *** danger_fo_away has quit IRC | 06:28 | |
| *** swifterdarrell has quit IRC | 06:28 | |
| *** echohead has quit IRC | 06:28 | |
| *** matsuhashi has quit IRC | 06:30 | |
| *** matsuhashi has joined #openstack-infra | 06:30 | |
| *** matsuhashi has quit IRC | 06:30 | |
| *** matsuhashi has joined #openstack-infra | 06:30 | |
| *** toddmorey has quit IRC | 06:33 | |
| *** jhesketh__ has joined #openstack-infra | 06:34 | |
| *** markwash has joined #openstack-infra | 06:34 | |
| *** D30 has joined #openstack-infra | 06:38 | |
| *** sarob has joined #openstack-infra | 06:38 | |
| *** bingbu has quit IRC | 06:40 | |
| *** bingbu has joined #openstack-infra | 06:40 | |
| *** dcramer_ has joined #openstack-infra | 06:42 | |
| *** yfried has joined #openstack-infra | 06:42 | |
| *** bingbu is now known as bing_bu | 06:43 | |
| *** sarob has quit IRC | 06:43 | |
| *** bing_bu is now known as bingbu | 06:43 | |
| *** pblaho has joined #openstack-infra | 06:51 | |
| *** mrmartin has quit IRC | 06:53 | |
| lifeless | russellb: I think a secondary sort by received reviews might be interesting | 06:59 |
| *** nati_ueno has joined #openstack-infra | 07:00 | |
| *** denis_makogon has joined #openstack-infra | 07:02 | |
| *** toddmorey has joined #openstack-infra | 07:04 | |
| *** romcheg has joined #openstack-infra | 07:04 | |
| *** dkliban has joined #openstack-infra | 07:07 | |
| lifeless | I have a friend trying to sign up with the foundation | 07:07 |
| lifeless | and they are getting an error | 07:07 |
| lifeless | '"You must add at least one Affiliation."' | 07:07 |
| lifeless | this seems very strange to me | 07:07 |
| *** senk has joined #openstack-infra | 07:08 | |
| *** nati_ueno has quit IRC | 07:09 | |
| *** mrmartin has joined #openstack-infra | 07:09 | |
| *** alexpilotti has joined #openstack-infra | 07:12 | |
| *** nati_ueno has joined #openstack-infra | 07:13 | |
| *** nati_uen_ has joined #openstack-infra | 07:17 | |
| *** nati_ueno has quit IRC | 07:20 | |
| *** yolanda has joined #openstack-infra | 07:23 | |
| *** toddmorey has quit IRC | 07:25 | |
| *** alexpilotti has quit IRC | 07:26 | |
| nati_uen_ | Hi I'm writing docs about "How to build 3rd Party OpenStack Gating System" https://docs.google.com/a/ntti3.com/presentation/d/1WJInaSt_H2kVkjnhtPmiATP1F-0BVbuk1eefQalL5Q0/edit#slide=id.p | 07:27 |
| nati_uen_ | I wanna know how to use nodepool | 07:27 |
| nati_uen_ | do you have any docs on this? | 07:27 |
| *** davidhadas_ has quit IRC | 07:29 | |
| *** DennyZha` has quit IRC | 07:29 | |
| *** alexpilotti has joined #openstack-infra | 07:30 | |
| lifeless | nati_uen_: http://ci.openstack.org/running-your-own.html - docs you might want to contribute to | 07:31 |
| lifeless | nati_uen_: and http://ci.openstack.org/nodepool.html | 07:32 |
| lifeless | nati_uen_: plus nodepool has it's own docs, of course | 07:32 |
| nati_uen_ | lifeless: Thanks! | 07:32 |
| nati_uen_ | lifeless: where can I get it? | 07:32 |
| lifeless | nati_uen_: the nodepool source tree, I don't think it's published anywhere yet | 07:33 |
| nati_uen_ | lifeless: it looks like README is blank | 07:33 |
| nati_uen_ | Ah sorry | 07:34 |
| nati_uen_ | README.rst is blank but it has README | 07:34 |
| nati_uen_ | Thanks | 07:34 |
| *** slong_ has quit IRC | 07:39 | |
| *** jcoufal has joined #openstack-infra | 07:41 | |
| *** senk has quit IRC | 07:42 | |
| *** senk has joined #openstack-infra | 07:46 | |
| *** flaper87|afk is now known as flaper87 | 07:51 | |
| *** senk1 has joined #openstack-infra | 07:52 | |
| *** senk has quit IRC | 07:53 | |
| *** Krast has joined #openstack-infra | 07:58 | |
| *** alexpilotti has quit IRC | 07:58 | |
| *** nati_uen_ has quit IRC | 08:03 | |
| *** nati_ueno has joined #openstack-infra | 08:08 | |
| *** osanchez has joined #openstack-infra | 08:11 | |
| *** cody-somerville has quit IRC | 08:13 | |
| *** wenlock has quit IRC | 08:16 | |
| *** jpich has joined #openstack-infra | 08:22 | |
| *** toddmorey has joined #openstack-infra | 08:23 | |
| *** toddmorey has quit IRC | 08:27 | |
| *** matsuhashi has quit IRC | 08:30 | |
| *** UtahDave has quit IRC | 08:33 | |
| *** matsuhashi has joined #openstack-infra | 08:35 | |
| *** locke105 has joined #openstack-infra | 08:37 | |
| openstackgerrit | Marcus Nilsson proposed a change to openstack-infra/jenkins-job-builder: Added support for Stash Notifier https://review.openstack.org/56337 | 08:39 |
| *** markwash has quit IRC | 08:40 | |
| *** hub_cap_ has joined #openstack-infra | 08:40 | |
| *** denis_makogon has quit IRC | 08:40 | |
| *** mikap has joined #openstack-infra | 08:41 | |
| *** mikap has joined #openstack-infra | 08:41 | |
| *** mrmartin has quit IRC | 08:44 | |
| *** michchap is now known as 45PAALEL3 | 08:44 | |
| *** 20WAAKLSA has joined #openstack-infra | 08:44 | |
| *** michchap has joined #openstack-infra | 08:44 | |
| *** yaguang has joined #openstack-infra | 08:44 | |
| *** salv-orlando has joined #openstack-infra | 08:44 | |
| *** freyes has joined #openstack-infra | 08:44 | |
| *** zoresvit has joined #openstack-infra | 08:44 | |
| *** Shrews has joined #openstack-infra | 08:44 | |
| *** danger_fo_away has joined #openstack-infra | 08:44 | |
| *** swifterdarrell has joined #openstack-infra | 08:44 | |
| *** echohead has joined #openstack-infra | 08:44 | |
| *** flaper87 has quit IRC | 08:44 | |
| *** flaper87 has joined #openstack-infra | 08:44 | |
| *** yfried has quit IRC | 08:45 | |
| *** yfried has joined #openstack-infra | 08:45 | |
| *** pblaho has quit IRC | 08:45 | |
| *** pblaho has joined #openstack-infra | 08:45 | |
| *** jcoufal has quit IRC | 08:45 | |
| *** jcoufal has joined #openstack-infra | 08:45 | |
| *** osanchez has quit IRC | 08:45 | |
| *** osanchez has joined #openstack-infra | 08:45 | |
| *** jpich has quit IRC | 08:45 | |
| *** Krast has quit IRC | 08:45 | |
| *** 45PAALEL3 has quit IRC | 08:45 | |
| *** saper has quit IRC | 08:45 | |
| *** locke1051 has quit IRC | 08:45 | |
| *** mugsie has quit IRC | 08:45 | |
| *** fbo_away has quit IRC | 08:45 | |
| *** jesusaurus has quit IRC | 08:45 | |
| *** hub_cap has quit IRC | 08:45 | |
| *** GheRivero has quit IRC | 08:45 | |
| *** mika has quit IRC | 08:45 | |
| *** mikap is now known as mika | 08:46 | |
| *** che-arne has joined #openstack-infra | 08:46 | |
| *** jpich has joined #openstack-infra | 08:46 | |
| *** Krast has joined #openstack-infra | 08:46 | |
| *** saper has joined #openstack-infra | 08:46 | |
| *** mugsie has joined #openstack-infra | 08:46 | |
| *** fbo_away has joined #openstack-infra | 08:46 | |
| *** jesusaurus has joined #openstack-infra | 08:46 | |
| *** GheRivero has joined #openstack-infra | 08:46 | |
| *** nosnos has quit IRC | 08:46 | |
| *** boris-42 has quit IRC | 08:47 | |
| *** boris-42 has joined #openstack-infra | 08:49 | |
| *** hashar has joined #openstack-infra | 08:49 | |
| *** markwash has joined #openstack-infra | 08:54 | |
| *** ogelbukh1 has joined #openstack-infra | 08:54 | |
| *** markwash has quit IRC | 09:00 | |
| *** mrmartin has joined #openstack-infra | 09:01 | |
| *** nati_ueno has quit IRC | 09:06 | |
| *** nati_ueno has joined #openstack-infra | 09:09 | |
| *** davidhadas has joined #openstack-infra | 09:10 | |
| *** yassine has joined #openstack-infra | 09:11 | |
| nati_ueno | Hi! Where can I find how to setup gearman? | 09:12 |
| *** mrmartin has quit IRC | 09:12 | |
| openstackgerrit | Roman Prykhodchenko proposed a change to openstack-infra/devstack-gate: Support of Ironic https://review.openstack.org/53899 | 09:14 |
| *** derekh has joined #openstack-infra | 09:19 | |
| *** boris-42 has quit IRC | 09:21 | |
| *** bingbu has quit IRC | 09:21 | |
| *** mrmartin has joined #openstack-infra | 09:22 | |
| *** toddmorey has joined #openstack-infra | 09:24 | |
| *** locke105 has quit IRC | 09:25 | |
| *** senk1 has quit IRC | 09:26 | |
| *** salv-orlando has quit IRC | 09:28 | |
| *** toddmorey has quit IRC | 09:28 | |
| *** osanchez has quit IRC | 09:31 | |
| *** osanchez has joined #openstack-infra | 09:33 | |
| *** jorisroovers has joined #openstack-infra | 09:34 | |
| chmouel | if someone from the ci has a bit of spare time adding this group https://bugs.launchpad.net/openstack-ci/+bug/1251041 | 09:41 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added task ordering https://review.openstack.org/56026 | 09:44 |
| *** davidhadas has quit IRC | 09:51 | |
| *** Ryan_Lane has quit IRC | 10:01 | |
| *** xchu has quit IRC | 10:06 | |
| *** nati_ueno has quit IRC | 10:06 | |
| *** nati_ueno has joined #openstack-infra | 10:07 | |
| *** mrmartin has quit IRC | 10:08 | |
| *** D30 has quit IRC | 10:09 | |
| *** mrmartin has joined #openstack-infra | 10:11 | |
| *** nati_ueno has quit IRC | 10:11 | |
| *** Krast has quit IRC | 10:13 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Fix import grouping https://review.openstack.org/52220 | 10:14 |
| *** matsuhashi has quit IRC | 10:20 | |
| *** matsuhashi has joined #openstack-infra | 10:21 | |
| openstackgerrit | Roman Prykhodchenko proposed a change to openstack-infra/devstack-gate: Support of Ironic https://review.openstack.org/53899 | 10:23 |
| *** SergeyLukjanov has joined #openstack-infra | 10:24 | |
| *** toddmorey has joined #openstack-infra | 10:25 | |
| *** matsuhashi has quit IRC | 10:25 | |
| *** senk1 has joined #openstack-infra | 10:29 | |
| mika | is there an ETA for next release of https://github.com/openstack-infra/jenkins-job-builder available? there are quite some nice improvements available since 0.5.0 release :) | 10:32 |
| *** 20WAAKLSA has quit IRC | 10:32 | |
| *** nosnos has joined #openstack-infra | 10:33 | |
| *** senk1 has quit IRC | 10:33 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Revert "Revert "Use Ubuntu Cloud Archive on precise nodes"" https://review.openstack.org/48226 | 10:35 |
| openstackgerrit | Noorul Islam K M proposed a change to openstack-dev/cookiecutter: Adjust _TRUE_VALUES to be consistent with OpenStack projects. https://review.openstack.org/56318 | 10:36 |
| *** nosnos has quit IRC | 10:37 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/jenkins-job-builder: Add build-publisher to jenkins-job-builder https://review.openstack.org/56356 | 10:39 |
| *** OlivierSanchez has joined #openstack-infra | 10:42 | |
| *** osanchez has quit IRC | 10:42 | |
| *** wenlock has joined #openstack-infra | 10:44 | |
| *** dachary has quit IRC | 10:52 | |
| *** dachary has joined #openstack-infra | 10:53 | |
| *** wenlock has quit IRC | 10:55 | |
| *** toddmorey has quit IRC | 10:58 | |
| *** metabro has joined #openstack-infra | 11:01 | |
| *** salv-orlando has joined #openstack-infra | 11:01 | |
| hashar | mika: I am not sure when releases are cut | 11:03 |
| mika | hashar: who might know? :) | 11:04 |
| hashar | I can't even find the launchpad project | 11:04 |
| hashar | a bug filled in https://bugs.launchpad.net/openstack-ci/ would certainly raise awareness | 11:05 |
| *** fifieldt has quit IRC | 11:06 | |
| mika | hm | 11:09 |
| mika | will file a bugreport there, let's see :) | 11:11 |
| *** jorisroovers has quit IRC | 11:11 | |
| *** afrittoli has quit IRC | 11:15 | |
| *** lcestari has joined #openstack-infra | 11:16 | |
| *** adalbas has joined #openstack-infra | 11:18 | |
| mika | hashar: filed it as https://bugs.launchpad.net/openstack-ci/+bug/1251201 | 11:18 |
| *** yfried has left #openstack-infra | 11:18 | |
| hashar | great | 11:19 |
| hashar | most of them are in US | 11:19 |
| hashar | so you might want to ping here later in the evening | 11:19 |
| mika | ok, thx :) | 11:19 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Make requirements jobs voting for savanna repos https://review.openstack.org/54788 | 11:21 |
| *** ilyashakhat has joined #openstack-infra | 11:21 | |
| *** mdenny has quit IRC | 11:21 | |
| *** guohliu has quit IRC | 11:21 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Communicate to user when we skip a requirement https://review.openstack.org/54628 | 11:22 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Base python 2.7 skip list on parsed names https://review.openstack.org/54152 | 11:22 |
| *** branen has quit IRC | 11:23 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add display-name job property. https://review.openstack.org/41828 | 11:24 |
| *** jorisroovers has joined #openstack-infra | 11:24 | |
| *** jorisroovers has quit IRC | 11:24 | |
| *** boris-42 has joined #openstack-infra | 11:24 | |
| *** jorisroovers has joined #openstack-infra | 11:25 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add build-publisher to jenkins-job-builder https://review.openstack.org/56356 | 11:25 |
| hashar | mordred: you should enjoy your breakfast instead of voting in Gerrit :D | 11:28 |
| mordred | hah | 11:28 |
| mordred | hashar: my new rule for myself is that I do my review queue first thing | 11:28 |
| mordred | that way it doesn't get out of hand | 11:28 |
| hashar | I tend to do that as well | 11:28 |
| hashar | but eventually ends up spending most of my day only doing reviews :( | 11:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Make requirements jobs voting for savanna repos https://review.openstack.org/54788 | 11:29 |
| mordred | hashar: I spent a full day doing them the other day - but now it's actually not terrible | 11:30 |
| mordred | I can do the morning's reviews in less than an hour | 11:30 |
| openstackgerrit | Ondra Machacek proposed a change to openstack-infra/jenkins-job-builder: Fix format of debug message to be compatible with py2.6 https://review.openstack.org/56361 | 11:34 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Fix savanna repos mappings https://review.openstack.org/56362 | 11:35 |
| SergeyLukjanov | hey guys | 11:37 |
| SergeyLukjanov | it looks like requirements push script doesn't work correctly | 11:38 |
| SergeyLukjanov | after updating hacking | 11:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Add the tomograph project to stackforge. https://review.openstack.org/56115 | 11:38 |
| SergeyLukjanov | it pushes update version only to the savanna-dashboard and python-savannaclient repos | 11:38 |
| SergeyLukjanov | but not to the savanna | 11:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Add rubick project to stackforge https://review.openstack.org/53858 | 11:38 |
| SergeyLukjanov | trying to find corresponding logs | 11:39 |
| *** salv-orlando has quit IRC | 11:39 | |
| ogelbukh | mordred: many thanks! | 11:39 |
| *** masayukig has joined #openstack-infra | 11:45 | |
| *** yaguang has quit IRC | 11:48 | |
| *** vladan has joined #openstack-infra | 11:48 | |
| *** mrmartin has quit IRC | 11:49 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Make -o option required if using 'test' command https://review.openstack.org/56250 | 11:49 |
| *** zhiyan has joined #openstack-infra | 11:50 | |
| *** rfolco has joined #openstack-infra | 11:52 | |
| zhiyan | hello any body can help take a look a grenade/upgrade test failure? seems glance-api pid has no permission to access /tmp, this caused code can't get lock successful. thanks | 11:53 |
| zhiyan | http://logs.openstack.org/79/46479/12/check/check-grenade-devstack-vm/87c1400/logs/new/screen-n-cpu.txt.gz#_2013-11-12_16_21_21_098 | 11:53 |
| zhiyan | http://logs.openstack.org/79/46479/12/check/check-grenade-devstack-vm/87c1400/logs/new/screen-g-api.txt.gz#_2013-11-12_16_21_19_931 | 11:53 |
| *** toddmorey has joined #openstack-infra | 11:55 | |
| openstackgerrit | A change was merged to openstack-infra/config: Stop running stable/folsom jobs https://review.openstack.org/56277 | 11:57 |
| *** rfolco has quit IRC | 11:58 | |
| *** toddmorey has quit IRC | 11:59 | |
| *** michchap has quit IRC | 12:00 | |
| *** apevec has joined #openstack-infra | 12:00 | |
| mordred | ^^ woot | 12:00 |
| mordred | fungi: morning | 12:00 |
| *** rfolco has joined #openstack-infra | 12:00 | |
| mordred | clarkb, fungi: I just thought - we have an opt-in list for people to subscribe to requirements changes | 12:00 |
| mordred | perhaps we should use that same list to tell if they should have the mirror enforced | 12:01 |
| mordred | instead of basing it on their repo prefix name | 12:01 |
| *** michchap has joined #openstack-infra | 12:02 | |
| *** weshay has joined #openstack-infra | 12:05 | |
| *** mrmartin has joined #openstack-infra | 12:08 | |
| *** pblaho has quit IRC | 12:08 | |
| *** senk has joined #openstack-infra | 12:12 | |
| *** senk has quit IRC | 12:17 | |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Map all fuel stackforge repos to fuel at launchpad https://review.openstack.org/56204 | 12:21 |
| *** dims has joined #openstack-infra | 12:21 | |
| *** senk has joined #openstack-infra | 12:22 | |
| *** ArxCruz has quit IRC | 12:26 | |
| *** ArxCruz has joined #openstack-infra | 12:26 | |
| *** dkranz has quit IRC | 12:31 | |
| *** dkranz has joined #openstack-infra | 12:32 | |
| *** wenlock has joined #openstack-infra | 12:35 | |
| *** michchap has quit IRC | 12:37 | |
| *** michchap has joined #openstack-infra | 12:37 | |
| *** jcoufal has quit IRC | 12:37 | |
| *** senk has quit IRC | 12:37 | |
| mordred | fungi: also, I could really use a second review on https://review.openstack.org/#/c/48888/ | 12:42 |
| *** dkranz has quit IRC | 12:43 | |
| *** wenlock has quit IRC | 12:44 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: New base class to declare test cases https://review.openstack.org/49775 | 12:45 |
| openstackgerrit | Marcus Nilsson proposed a change to openstack-infra/jenkins-job-builder: Added support for Stash Notifier https://review.openstack.org/56337 | 12:46 |
| *** pcm_ has joined #openstack-infra | 12:46 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: tests for the node option https://review.openstack.org/56378 | 12:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add new HP Cloud regions https://review.openstack.org/56260 | 12:48 |
| *** HenryG has joined #openstack-infra | 12:49 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add new HP Cloud regions https://review.openstack.org/56260 | 12:50 |
| *** thomasem has joined #openstack-infra | 12:51 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: New base class to declare test cases and tests to the node option https://review.openstack.org/49775 | 12:53 |
| *** weshay has quit IRC | 12:54 | |
| *** weshay has joined #openstack-infra | 12:55 | |
| *** toddmorey has joined #openstack-infra | 12:56 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Gate current clients on stable branches https://review.openstack.org/41931 | 12:56 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Gate current clients on stable branches https://review.openstack.org/41931 | 12:58 |
| *** dachary has quit IRC | 12:59 | |
| *** ryanpetrello has quit IRC | 12:59 | |
| *** dachary has joined #openstack-infra | 12:59 | |
| hashar | mordred: would you mind restoring https://review.openstack.org/#/c/49268 for me please ? | 13:03 |
| hashar | for some reason I am lacking Gerrit right to restore changes abandoned for which I am not the author :/ | 13:03 |
| mordred | hashar: done | 13:03 |
| mordred | that's an admin role I believe | 13:03 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Added support for Git shallow clone parameter https://review.openstack.org/49268 | 13:03 |
| hashar | thank you! | 13:03 |
| *** dims has quit IRC | 13:04 | |
| *** dims has joined #openstack-infra | 13:06 | |
| *** CaptTofu has quit IRC | 13:10 | |
| *** CaptTofu has joined #openstack-infra | 13:11 | |
| *** alexpilotti has joined #openstack-infra | 13:14 | |
| *** mrmartin has quit IRC | 13:15 | |
| *** hashar has quit IRC | 13:16 | |
| *** toddmorey has quit IRC | 13:17 | |
| *** masayukig has quit IRC | 13:17 | |
| openstackgerrit | Christopher Yeoh proposed a change to openstack-infra/reviewstats: Adds new/merge rate to reviewers information https://review.openstack.org/56380 | 13:18 |
| *** DinaBelova has joined #openstack-infra | 13:19 | |
| *** resker has quit IRC | 13:20 | |
| *** dims has quit IRC | 13:24 | |
| *** jhesketh has quit IRC | 13:24 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 13:25 | |
| *** _SergeyLukjanov has quit IRC | 13:26 | |
| *** dims has joined #openstack-infra | 13:28 | |
| mordred | russellb: http://russellbryant.net/openstack-stats/infra-reviewers-30.txt | 13:30 |
| mordred | russellb: check the krtaylor line | 13:31 |
| mordred | 0% doesn't seem happily rendered :) | 13:31 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added task ordering https://review.openstack.org/56026 | 13:31 |
| freyes | hi, does somebody know Antoine Musso's IRC nickname? | 13:32 |
| *** sandywalsh has quit IRC | 13:33 | |
| mordred | freyes: yes. it's hashar | 13:33 |
| mordred | freyes: and he dropped off about 15 minutes ago | 13:33 |
| freyes | mordred: OK, thanks ;) .. so he's not online | 13:33 |
| *** yaguang has joined #openstack-infra | 13:35 | |
| *** mrmartin has joined #openstack-infra | 13:36 | |
| *** jhesketh has joined #openstack-infra | 13:38 | |
| *** CaptTofu has quit IRC | 13:41 | |
| *** CaptTofu has joined #openstack-infra | 13:41 | |
| *** toddmorey has joined #openstack-infra | 13:43 | |
| *** dkranz has joined #openstack-infra | 13:45 | |
| *** sandywalsh has joined #openstack-infra | 13:46 | |
| *** CaptTofu has quit IRC | 13:49 | |
| *** CaptTofu has joined #openstack-infra | 13:49 | |
| *** dprince has joined #openstack-infra | 13:50 | |
| *** xchu has joined #openstack-infra | 13:50 | |
| *** dcramer_ has quit IRC | 13:51 | |
| ilyashakhat | hi, a small question on gerrit / review.openstack.org - is there a way to retrieve group members? rest api or ssh? | 13:53 |
| *** senk has joined #openstack-infra | 13:54 | |
| *** herndon_ has joined #openstack-infra | 13:54 | |
| *** mriedem has joined #openstack-infra | 13:54 | |
| mattymo | ilyashakhat, you mean who has access to gerrit groups? | 13:55 |
| *** mrmartin has quit IRC | 13:56 | |
| ilyashakhat | i'd like to have a list of users belonging to a group | 13:56 |
| ilyashakhat | for example to know who belongs to nova-core | 13:57 |
| *** romcheg has left #openstack-infra | 13:57 | |
| ilyashakhat | it is possible to get that data via web ui, but i'm looking for more automated api | 13:57 |
| krtaylor | mordred, looks like I am causing problems again, I guess I just need to do more reviews | 13:58 |
| *** xeyed4good has joined #openstack-infra | 13:59 | |
| *** senk has quit IRC | 13:59 | |
| *** xeyed4good has left #openstack-infra | 13:59 | |
| *** sc68cal has left #openstack-infra | 13:59 | |
| *** DinaBelova has quit IRC | 13:59 | |
| krtaylor | nm, that's Received | 14:00 |
| krtaylor | but I still need to do more reviews | 14:00 |
| ilyashakhat | it means that you ought to post more patches :) | 14:01 |
| *** noorul has joined #openstack-infra | 14:05 | |
| krtaylor | ilyashakhat, +1 | 14:05 |
| *** mrmartin has joined #openstack-infra | 14:06 | |
| krtaylor | ArxCruz, is doing that for the team at the moment | 14:06 |
| ArxCruz | :) | 14:10 |
| russellb | mordred: yeah, that was supposed to be infinity :) | 14:11 |
| *** talluri has quit IRC | 14:14 | |
| *** hashar has joined #openstack-infra | 14:16 | |
| *** toddmorey has quit IRC | 14:17 | |
| *** julim has joined #openstack-infra | 14:19 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: cloverphp publisher https://review.openstack.org/52940 | 14:20 |
| *** boris-42 has quit IRC | 14:20 | |
| *** jpeeler has joined #openstack-infra | 14:21 | |
| *** jpeeler has quit IRC | 14:21 | |
| *** jpeeler has joined #openstack-infra | 14:21 | |
| *** senk has joined #openstack-infra | 14:22 | |
| *** toddmorey has joined #openstack-infra | 14:22 | |
| *** johnthetubaguy has joined #openstack-infra | 14:24 | |
| *** nsaje has joined #openstack-infra | 14:25 | |
| *** zhiyan has quit IRC | 14:25 | |
| *** zhiyan has joined #openstack-infra | 14:25 | |
| *** esker has joined #openstack-infra | 14:26 | |
| *** senk has quit IRC | 14:27 | |
| *** zoresvit has quit IRC | 14:27 | |
| fungi | mordred: since the folsom job removals merged, i've opened the d-g one up for review (just needs one more +2) https://review.openstack.org/56284 | 14:30 |
| *** mrmartin has quit IRC | 14:31 | |
| fungi | mordred: as far as requirements being enforced on arbitrary projects, i agree it may ease the transition for projects pre-incubation | 14:32 |
| *** zoresvit has joined #openstack-infra | 14:32 | |
| *** jcoufal has joined #openstack-infra | 14:32 | |
| *** yassine has quit IRC | 14:33 | |
| *** senk has joined #openstack-infra | 14:33 | |
| *** yassine has joined #openstack-infra | 14:33 | |
| *** senk has quit IRC | 14:33 | |
| *** herndon_ has quit IRC | 14:33 | |
| mordred | russellb: haha | 14:35 |
| *** johnthetubaguy1 has joined #openstack-infra | 14:36 | |
| mordred | fungi: +A | 14:36 |
| *** johnthetubaguy has quit IRC | 14:37 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix format of debug message to be compatible with py2.6 https://review.openstack.org/56361 | 14:37 |
| *** markmcclain has quit IRC | 14:38 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:38 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Pagination added to task lists https://review.openstack.org/55813 | 14:39 |
| *** wenlock has joined #openstack-infra | 14:39 | |
| *** yamahata has joined #openstack-infra | 14:39 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: cloverphp publisher https://review.openstack.org/52940 | 14:40 |
| *** dizquierdo has joined #openstack-infra | 14:41 | |
| *** nsaje has quit IRC | 14:41 | |
| *** nsaje has joined #openstack-infra | 14:41 | |
| zhiyan | fungi: morning. can you help take a look a grenade/upgrade test failure? seems glance-api pid has no permission to access /tmp, this caused code can't get lock successful. thanks | 14:43 |
| *** nsaje has quit IRC | 14:43 | |
| zhiyan | http://logs.openstack.org/79/46479/12/check/check-grenade-devstack-vm/87c1400/logs/new/screen-n-cpu.txt.gz#_2013-11-12_16_21_21_098 | 14:43 |
| zhiyan | http://logs.openstack.org/79/46479/12/check/check-grenade-devstack-vm/87c1400/logs/new/screen-g-api.txt.gz#_2013-11-12_16_21_19_931 | 14:43 |
| *** nsaje has joined #openstack-infra | 14:43 | |
| fungi | zhiyan: having a look real quick, but you may also have more luck raising attention of the grenade developers on #openstack-qa | 14:45 |
| zhiyan | fungi: sure | 14:45 |
| *** ryanpetrello has joined #openstack-infra | 14:45 | |
| *** jpeeler1 has joined #openstack-infra | 14:46 | |
| *** jamesmcarthur has joined #openstack-infra | 14:47 | |
| *** jpeeler has quit IRC | 14:48 | |
| *** mrmartin has joined #openstack-infra | 14:50 | |
| fungi | zhiyan: i'm not seeing where it's mentioning a permissions error on /tmp. did you find that in a different log? | 14:51 |
| fungi | this almost looks more like the issue we've been seeing where swift has insufficient space for glance to write images and causes a 500 error | 14:51 |
| * fungi wanders off in search of that bug number | 14:52 | |
| zhiyan | fungi: from the log i can see glance-api can not get file lock forever, you can see it from above log which i ping you. | 14:53 |
| fungi | ahh, got it. i was searching after the line you linked, not before | 14:54 |
| *** talluri has joined #openstack-infra | 14:54 | |
| *** dizquierdo has quit IRC | 14:54 | |
| *** jpeeler1 is now known as jpeeler | 14:55 | |
| *** jpeeler has quit IRC | 14:55 | |
| *** jpeeler has joined #openstack-infra | 14:55 | |
| fungi | looks like glance was erroring before that in the log too. i see other similar backtraces prior to that one | 14:56 |
| zhiyan | fungi: very strange to me, on the local i can run the code normal, the file lock can be allocated, but on the upgrade tests, from log i can see glance-api can't get lock, and code be blocked. | 14:56 |
| fungi | the earliest instance i see in that log is at http://logs.openstack.org/79/46479/12/check/check-grenade-devstack-vm/87c1400/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2013-11-12_16_15_40_789 | 14:58 |
| *** mfer has joined #openstack-infra | 14:58 | |
| *** yassine has quit IRC | 15:01 | |
| *** yassine has joined #openstack-infra | 15:02 | |
| *** wenlock has quit IRC | 15:03 | |
| zhiyan | fungi: yes, so you can see the issue on the glance-api side: http://logs.openstack.org/79/46479/12/check/check-grenade-devstack-vm/87c1400/logs/new/screen-g-api.txt.gz#_2013-11-12_16_15_40_785 | 15:03 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Fix savanna repos mappings https://review.openstack.org/56362 | 15:04 |
| zhiyan | glance-api cache middle code attempting to grab file lock, but never got it | 15:04 |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Run tox tests for pecan to gate against WSME, Ceilometer, and Ironic. https://review.openstack.org/54333 | 15:04 |
| *** johnthetubaguy1 has quit IRC | 15:05 | |
| *** johnthetubaguy has joined #openstack-infra | 15:05 | |
| fungi | zhiyan: keep in mind that /tmp is (as you would expect) entirely world-writable on our devstack test slave nodes, so presumably if there's a permissions issue, it has to do with a file (or subdirectory) created in /tmp as one user and then trying to get a write lock on that file with a different user | 15:07 |
| *** pblaho has joined #openstack-infra | 15:07 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: raise JenkinsJobsException instead of Exception https://review.openstack.org/56397 | 15:07 |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Add distribute to the pre-install list https://review.openstack.org/48888 | 15:08 |
| *** senk has joined #openstack-infra | 15:11 | |
| mordred | woot | 15:11 |
| *** mrmartin has quit IRC | 15:12 | |
| mordred | fungi: thank you | 15:12 |
| fungi | zhiyan: what is the default lock_path for glance-api? looks like that patch is setting it to /tmp so presumably it was putting lockfiles somewhere else previously (or does having that option commented out in the config stop it from making lockfiles at all?) | 15:14 |
| zhiyan | default is /tmp | 15:15 |
| zhiyan | fungi: in glance-api.conf template file | 15:15 |
| *** nsaje has quit IRC | 15:15 | |
| fungi | okay, so i guess that addition to cache.py is introducing a new lock named for the image id | 15:15 |
| *** nsaje has joined #openstack-infra | 15:16 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Remove folsom-specific logic https://review.openstack.org/56284 | 15:16 |
| *** yamahata has quit IRC | 15:16 | |
| *** herndon_ has joined #openstack-infra | 15:16 | |
| *** ArxCruz has quit IRC | 15:16 | |
| *** ArxCruz has joined #openstack-infra | 15:17 | |
| *** BobBallAway is now known as BobBall | 15:17 | |
| *** guohliu has joined #openstack-infra | 15:18 | |
| *** markmcclain has joined #openstack-infra | 15:19 | |
| fungi | zhiyan: is this normal in glance-api? "Stderr: '/bin/sh: 1: collie: not found\n'" | 15:19 |
| fungi | http://logs.openstack.org/79/46479/12/check/check-grenade-devstack-vm/87c1400/logs/new/screen-g-api.txt.gz#_2013-11-12_16_10_57_800 | 15:19 |
| zhiyan | fungi: yes, it's normal | 15:20 |
| zhiyan | i'm sure | 15:20 |
| fungi | okay | 15:20 |
| *** ruhe has joined #openstack-infra | 15:22 | |
| mordred | that's a normal thing? | 15:22 |
| fungi | it's pretty crazypants, but maybe that's normal in glance | 15:23 |
| mordred | glance/tests/unit/base.py: | 15:23 |
| mordred | """Create known stores. Mock out sheepdog's subprocess dependency | 15:23 |
| mordred | on collie. | 15:23 |
| mordred | something tells me that the mocking out is not working | 15:23 |
| *** mfer has quit IRC | 15:24 | |
| fungi | though doesn't seem to be introduced in the change in question at least | 15:24 |
| mordred | hah | 15:24 |
| mordred | that's because the mock is mocking out a method that is not used for running collie | 15:24 |
| *** herndon_ has quit IRC | 15:25 | |
| mordred | sigh | 15:25 |
| * mordred gives up looking at a random chunk of code that doesn't matter | 15:25 | |
| *** nsaje has quit IRC | 15:26 | |
| *** nsaje has joined #openstack-infra | 15:26 | |
| *** nsaje has quit IRC | 15:29 | |
| *** mfer has joined #openstack-infra | 15:29 | |
| *** Dafna has joined #openstack-infra | 15:30 | |
| *** senk has quit IRC | 15:31 | |
| anteaya | Dafna: here you are | 15:32 |
| Dafna | anteaya: yes :) | 15:32 |
| anteaya | :D | 15:32 |
| *** ben_duyujie has joined #openstack-infra | 15:33 | |
| *** wenlock has joined #openstack-infra | 15:34 | |
| fungi | zhiyan: so, i've poured over the logs from that run and i'm still not seeing an actual permissions error for /tmp (and i wouldn't expect one since it's definitely world-writable on those systems). was that just speculation on your part or can you show me where it actually mentions something like "insufficient permissions on /tmp/whatever"? | 15:35 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Fix rendering issue https://review.openstack.org/56403 | 15:35 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Fix rendering issue https://review.openstack.org/56403 | 15:36 |
| zhiyan | fungi: sorry i can't give you that also. i just conjecture that from the log | 15:37 |
| fungi | zhiyan: i retrieved, uncompressed and grepped through all the logs from that test run | 15:37 |
| zhiyan | fungi: glance-api try to get file lock, but never get it. thank you. | 15:38 |
| fungi | zhiyan: it's also hitting that change consistently and i can't find the same pattern of failure on any other tested changes, so my guess is it's either a problem in that change or an issue in either glance or tempest which is being uncovered by it | 15:38 |
| fungi | rather uncovered by that change in the context of a grenade post-upgrade tempest run | 15:39 |
| zhiyan | fungi: may be, but i personally don't think so tbh. | 15:39 |
| fungi | so probably your best bet is to get glance, grenade or tempest devs looking closer at it | 15:39 |
| fungi | from an infra sysadmin perspective, i don't see anything awry with the systems that job runs on which would explain the problem | 15:40 |
| *** nsaje has joined #openstack-infra | 15:40 | |
| fungi | i certainly can't make permissions on /tmp any more open than world-writable | 15:40 |
| zhiyan | fungi: other changes all can pass all Jeninks test, except this one, since i know this change using a file lock. (other one is scrubber code) | 15:40 |
| fungi | so my guess is it's trying to open a file already owned by another user, or in a subdirectory created by another user, or somewhere else you're not expecting (maybe the lock dir option is being ignored/modiefied?) | 15:41 |
| zhiyan | fungi: ok, thanks for you support anyway. | 15:41 |
| fungi | zhiyan: here's one possibility... | 15:41 |
| fungi | keep in mind that grenade insists that the upgrade from the previous version to the one with your change applied *must* work with the old config files (your config file changes are not being applied in the new environment) | 15:42 |
| fungi | so if this change depends on config changes, that's possibly your issue | 15:42 |
| fungi | if those config changes are really just reflecting the existing default values for those options like you say, then it probably isn't the cause. but it's something to double check | 15:43 |
| *** nsaje has quit IRC | 15:43 | |
| *** che-arne has quit IRC | 15:44 | |
| *** nsaje has joined #openstack-infra | 15:44 | |
| *** dcramer_ has joined #openstack-infra | 15:44 | |
| *** noorul has left #openstack-infra | 15:45 | |
| *** rcleere has joined #openstack-infra | 15:47 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-dev/pbr: Ignore jenkins@openstack.org in authors building https://review.openstack.org/56407 | 15:48 |
| anteaya | in this code: http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/l3_db.py#n601 | 15:49 |
| anteaya | how do I find out who wrote line 615 and 636? | 15:49 |
| *** SergeyLukjanov has quit IRC | 15:49 | |
| anteaya | must be some git magic I can use to find the commit for it, would there be? | 15:49 |
| mordred | fungi, clarkb:ok. my last patch to pypi-mirror did not fix it | 15:50 |
| mordred | fungi, clarkb: so I'm resorting to actually testing | 15:50 |
| mordred | the next patch will work :) | 15:50 |
| fungi | anteaya: you want to read up on the 'git blame' command | 15:51 |
| fungi | it will show you the most recent commit sha, author and date for every line in a file | 15:52 |
| anteaya | fungi I will do that, thank you | 15:52 |
| fungi | in the simplest case, just run 'git blame neutron/db/l3_db.py' and then look for the line in question | 15:53 |
| anteaya | fungi I will start there | 15:53 |
| pleia2 | reminder, I have name change stuff to do this morning (court + social security) so I'll be out for a couple hours | 15:55 |
| *** datsun180b has joined #openstack-infra | 15:55 | |
| *** che-arne has joined #openstack-infra | 15:55 | |
| anteaya | pleia2: happy name changing | 15:55 |
| *** julim has quit IRC | 15:55 | |
| anteaya | in the personal schedule category, I am driving with my parents on their annual migration south this year - it is time - week of Nov 25 | 15:56 |
| anteaya | then in Paris the next week, so hit and miss for email and irc | 15:56 |
| pleia2 | anteaya: that's thanksgiving week here, so it may be a bit quiet anyway :) | 15:56 |
| anteaya | not that I am in here much right now, but for info | 15:56 |
| anteaya | w00t! | 15:57 |
| anteaya | yay for timing | 15:57 |
| *** julim has joined #openstack-infra | 15:57 | |
| anteaya | fungi: look at all that lovely information | 15:59 |
| anteaya | thanks | 15:59 |
| *** nsaje has quit IRC | 15:59 | |
| anteaya | fungi: I see this -> f639788c quantum/db/l3_db.py (openstack | 16:00 |
| anteaya | is there an openstack user that makes edits to files? | 16:00 |
| anteaya | that is confusing | 16:01 |
| fungi | possibly... git show f639788c | 16:01 |
| fungi | anteaya: http://git.openstack.org/cgit/openstack/neutron/commit/neutron/db/l3_db.py?id=f639788c | 16:02 |
| fungi | you can also do it that way if you like web browsers | 16:02 |
| anteaya | fungi I know Sylvain | 16:02 |
| anteaya | I'll ask him to change his username | 16:03 |
| *** datsun180b has quit IRC | 16:03 | |
| fungi | apparently he has his git author real-name set to "openstack" but committer real-name is still "Sylvain Afchain" | 16:04 |
| fungi | chances are he wrote the patch and committed it to his repository on one machine, then pushed it to gerrit for review from another machine | 16:04 |
| fungi | possibly with a git format-patch and git am step in between | 16:05 |
| *** datsun180b has joined #openstack-infra | 16:05 | |
| *** datsun180b has quit IRC | 16:05 | |
| fungi | at least that's one of the easiest ways i can imagine it ending up that way | 16:05 |
| *** herndon has joined #openstack-infra | 16:05 | |
| *** datsun180b has joined #openstack-infra | 16:06 | |
| anteaya | fungi: I asked one of his co-workers to ask him to change it | 16:06 |
| anteaya | since I can't see he is on irc atm | 16:06 |
| anteaya | and your theory may be correct | 16:07 |
| fungi | yeah, i'm betting he was hacking the patch on a devstack environment or something and then recommitted it while preserving the original author name, so probably not a misconfiguration per se, just a confusing workflow | 16:08 |
| *** mfer has quit IRC | 16:08 | |
| *** rnirmal has joined #openstack-infra | 16:08 | |
| *** mfer has joined #openstack-infra | 16:08 | |
| *** afazekas has quit IRC | 16:09 | |
| *** dkranz has quit IRC | 16:11 | |
| anteaya | fungi you are probably right, the username was already changed to the commiter name | 16:12 |
| anteaya | not sure how many patches landed as openstack | 16:12 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/pypi-mirror: Don't upgrade setuptools or distribute https://review.openstack.org/56411 | 16:12 |
| mordred | fungi, clarkb: ^^ | 16:12 |
| mordred | also, dstufft ^^ | 16:12 |
| fungi | mordred: what specifically was the reason? does mysql-python insist on an earlier version or something? | 16:15 |
| *** pblaho has quit IRC | 16:15 | |
| mgagne | or maybe openstack is self-aware and writes its own code | 16:15 |
| fungi | just curious why having setuptools/distribute already installed causes that | 16:15 |
| fungi | mgagne: it definitely does, in fact | 16:15 |
| *** pblaho has joined #openstack-infra | 16:15 | |
| fungi | we have more than one change-generating ci job | 16:16 |
| *** pblaho has quit IRC | 16:18 | |
| mordred | fungi: I honestly have no idea | 16:18 |
| fungi | fair enough. giant ball of confuse | 16:18 |
| openstackgerrit | Michael Prokop proposed a change to openstack-infra/jenkins-job-builder: Deprecate projectName setting of CopyArtifact https://review.openstack.org/56412 | 16:19 |
| *** mrmartin has joined #openstack-infra | 16:19 | |
| fungi | should just replace some of that script with a comment which says "sacrifice more chickens" | 16:19 |
| *** markmcclain has quit IRC | 16:19 | |
| mordred | fungi, dstufft: http://paste.openstack.org/show/52851 | 16:21 |
| mordred | there is the smallest reproducible test case | 16:21 |
| *** xchu has quit IRC | 16:22 | |
| mordred | actually | 16:23 |
| mordred | http://paste.openstack.org/show/52855 | 16:23 |
| *** mrmartin has quit IRC | 16:23 | |
| mordred | is slightly smaller | 16:23 |
| fungi | so we first install/upgrade pip, wheel and virtualenv; then build wheels of pip, setuptools, distribute and virtualenv; then after the freeze install/upgrade pip and wheel inside the second virtualenv? | 16:23 |
| mordred | fungi: yes. that's right | 16:23 |
| mordred | we do the first round so that pip, setuptools and friends have wheels available in our mirror | 16:23 |
| fungi | so we're building wheels of whatever version of setuptools and distribute are installed by default by virtualenv? | 16:24 |
| *** danger_fo_away is now known as danger_fo | 16:25 | |
| fungi | no, i guess that actually downloads the latest ones before building wheels of them | 16:25 |
| fungi | since that's being done from the context of the upgrade_format pattern | 16:25 |
| mordred | yup | 16:26 |
| anteaya | fungi I bought a rubber chicken in HK for test running purposes | 16:26 |
| *** oubiwann has joined #openstack-infra | 16:26 | |
| mordred | excellent | 16:26 |
| anteaya | :D | 16:26 |
| *** danger_fo is now known as dangers | 16:27 | |
| fungi | so the intent is really: get latest pip, wheel and virtualenv; use those to retrieve and build wheels of latest pip, setuptools, distribute and virtualenv; then install latest pip and wheel inside the second virtualenv | 16:27 |
| *** dkranz has joined #openstack-infra | 16:27 | |
| *** ^d has joined #openstack-infra | 16:30 | |
| mordred | yes | 16:32 |
| dripton | Does anyone know if we control the credentials for sqlalchemy-migrate on PyPI? I need to do a new release, and being able to change the info there is part of it. | 16:33 |
| *** branen has joined #openstack-infra | 16:34 | |
| *** yaguang has quit IRC | 16:36 | |
| *** mrmartin has joined #openstack-infra | 16:37 | |
| fungi | dripton: looks like it... https://pypi.python.org/pypi/sqlalchemy-migrate/0.7.2 | 16:39 |
| fungi | "Package Index Owner: ches, evyaro, jandd, openstackci, sdirector " | 16:39 |
| fungi | those last two are our release automation and (i believe) mordred | 16:40 |
| dripton | fungi: thanks, I didn't see openstackci when I looked before | 16:40 |
| *** mrodden has joined #openstack-infra | 16:41 | |
| dripton | fungi: How do I check to see if our CI tools are setup to automatically update PyPI for a particular project? (openstackci in that list means they *could* be, but there might be a flag to turn on somewhere.) | 16:41 |
| *** dkranz has quit IRC | 16:42 | |
| *** yassine has quit IRC | 16:43 | |
| *** afazekas has joined #openstack-infra | 16:43 | |
| fungi | dripton: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n2615 | 16:45 |
| russellb | we've averaged 478.7 reviews a day for the last 90 days, neat. http://russellbryant.net/openstack-stats/all-reviewers-90.txt | 16:45 |
| *** ruhe has quit IRC | 16:45 | |
| fungi | that means our atcs are averaging roughly 0.3 reviews a day | 16:46 |
| russellb | 318.3/day by core reviewers as a group | 16:46 |
| *** Dafna has left #openstack-infra | 16:47 | |
| fungi | that's more like 2-3 reviews a day on average for cores | 16:47 |
| * fungi can't remember the exact core reviewer count | 16:47 | |
| russellb | not sure how many cores we have | 16:47 |
| russellb | it's supposed to be in the output here, but it's wrong | 16:47 |
| *** nicedice_ has joined #openstack-infra | 16:48 | |
| *** rhsu has quit IRC | 16:48 | |
| fungi | yeah, i've got a mostly working script which runs multiple queries and recursively solves the included groups (because in gerrit groups can include groups) but it's nontrivial | 16:48 |
| russellb | nice | 16:49 |
| fungi | parsing permissions out of all acls and then listing the projects on which a given reviewer has +1aprv | 16:49 |
| dripton | Thanks for your help fungi. It appears everything I need is set up; I just need to exercise it. | 16:49 |
| fungi | i should finish polishing that and publish it somewhere, but it needs gerrit admin access to run since it has to do direct db queries | 16:50 |
| russellb | some way to get a dump from it would be one less thing reviewstats has to track in its repo | 16:51 |
| russellb | we have json files mapping repos and core teams to programs ... | 16:51 |
| fungi | i wish gerrit provided finer-grained access to some of that data. maybe 2.8 does a better job of that than 2.4 | 16:51 |
| fungi | part of the problem is that it's a moving target though, so i can at best generate a snapshot in time, but not really a history of when someone was or wasn't core on particular projects | 16:53 |
| anteaya | russellb: is there a way to tell in one file how many cores each project has? | 16:54 |
| fungi | if only gerrit maintained group membership using files in a git repo | 16:54 |
| *** MarkAtwood has joined #openstack-infra | 16:55 | |
| *** dkranz has joined #openstack-infra | 16:55 | |
| fungi | anteaya: for the most part, you can go to admin->groups in gerrit and click on projectname-core, then members and count the list (but also look below the members list to see if other groups are included and look at those too if so) | 16:55 |
| *** rcleere has quit IRC | 16:56 | |
| anteaya | fungi: I'll do that | 16:56 |
| fungi | (and then de-dup common members if you're dealing with one or more additional included groups) | 16:56 |
| russellb | yep that's best | 16:56 |
| russellb | i have this in files ... perpetually out of date though ... https://git.openstack.org/cgit/openstack-infra/reviewstats/tree/projects | 16:56 |
| *** rcleere has joined #openstack-infra | 16:57 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Fix core member count https://review.openstack.org/56421 | 16:57 |
| *** johnthetubaguy has quit IRC | 16:58 | |
| russellb | so my thing for all projects over the last 90 days should have said 184 core team reviewers | 16:58 |
| russellb | so roughly 1.5/day per core reviewer | 16:58 |
| *** HenryG has quit IRC | 17:01 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Show average reviews per day https://review.openstack.org/56425 | 17:08 |
| zaro | ilyashakhat: did you figure out how to get gerrit group info? | 17:13 |
| anteaya | russellb: thx | 17:14 |
| *** datsun180b_ has joined #openstack-infra | 17:14 | |
| *** melwitt has joined #openstack-infra | 17:16 | |
| *** rhsu has joined #openstack-infra | 17:16 | |
| *** alexpilotti has quit IRC | 17:17 | |
| *** chandankumar has quit IRC | 17:17 | |
| *** datsun180b has quit IRC | 17:17 | |
| *** datsun180b_ is now known as datsun180b | 17:17 | |
| *** alexpilotti has joined #openstack-infra | 17:19 | |
| *** senk has joined #openstack-infra | 17:20 | |
| *** senk has quit IRC | 17:24 | |
| *** hub_cap_ is now known as hub_cap | 17:24 | |
| *** hashar has quit IRC | 17:37 | |
| *** mrmartin has quit IRC | 17:47 | |
| *** reed has joined #openstack-infra | 17:48 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:48 | |
| *** rhsu has quit IRC | 17:49 | |
| *** OlivierSanchez has quit IRC | 17:51 | |
| *** rhsu has joined #openstack-infra | 17:52 | |
| *** harlowja has joined #openstack-infra | 17:53 | |
| *** changbl has quit IRC | 17:56 | |
| *** hogepodge has joined #openstack-infra | 17:57 | |
| *** vipul is now known as vipul-away | 17:57 | |
| jog0 | looks like we fixed bug 1225664 (swift 50x errors) http://status.openstack.org/elastic-recheck/ | 17:58 |
| *** rnirmal has quit IRC | 17:58 | |
| *** derekh has quit IRC | 17:59 | |
| *** rcleere has quit IRC | 17:59 | |
| *** markmcclain has joined #openstack-infra | 18:00 | |
| *** rcleere has joined #openstack-infra | 18:03 | |
| *** andreaf has joined #openstack-infra | 18:04 | |
| *** krtaylor has quit IRC | 18:04 | |
| *** b3nt_pin has joined #openstack-infra | 18:05 | |
| *** beagles has quit IRC | 18:05 | |
| *** gyee has joined #openstack-infra | 18:05 | |
| *** b3nt_pin is now known as beagles | 18:05 | |
| mordred | jog0: w00t | 18:06 |
| anteaya | jog0: way to go, knocking those bugs down | 18:07 |
| jog0 | I think there we have a QA effort misalignment, in that the integrated projects should be the ones fixing there own bugs | 18:07 |
| *** dims has quit IRC | 18:07 | |
| mrodden | question for -infra: we have some db2 and qpid chef cookbooks that we us to do deployments of openstack and set it up with db2/qpid support. they are currently internal only but we want to put them out in the open for people to work on and use. would it be acceptable/possible to put these in stackforge? | 18:08 |
| jog0 | in this case (and not trying to pick on anyone) swift didn't deliver a sane devstack env for swift | 18:08 |
| clarkb | morning /me is running late today | 18:08 |
| *** beagles has quit IRC | 18:09 | |
| mordred | yay! it's clarkb | 18:09 |
| mordred | clarkb: review me! https://review.openstack.org/#/c/56411/ | 18:10 |
| mordred | clarkb: (also, good morning) | 18:10 |
| clarkb | morning! | 18:10 |
| clarkb | mordred: line 246 you still grab distribute | 18:11 |
| clarkb | mordred: but that is just a wheel | 18:11 |
| clarkb | should we remove it from tehre as well? | 18:11 |
| mordred | yes. that's just to build the wheel | 18:11 |
| mordred | clarkb: having it there did not affect any breakages | 18:11 |
| mordred | clarkb: (I actually tested several iterations locally this time) | 18:12 |
| *** jcoufal has quit IRC | 18:12 | |
| mrodden | mordred: "we're using a new enough version of pip and virtualenv now" would it make sense to mention which versions? just a question that i thought of when looking it over | 18:12 |
| *** milki has quit IRC | 18:13 | |
| mordred | mrodden: yeah- I could mention that for sure | 18:14 |
| *** blamar has joined #openstack-infra | 18:14 | |
| mordred | for that matter | 18:14 |
| mordred | clarkb, mrodden: should I put virtualenv into the requirements.txt ? we do kinda rely on it existing | 18:14 |
| mordred | in fact, I do not think that pypi-mirror should depend on pip. I think it should depend on virtualenv - all of the operations it does are inside of virtualenvs | 18:15 |
| mrodden | i think virtualenv pulls in pip | 18:15 |
| mrodden | at least its own copy of it | 18:15 |
| mrodden | not sure if that woudl be an issue or not | 18:15 |
| *** shardy is now known as shardy_afk | 18:16 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/pypi-mirror: Don't upgrade setuptools or distribute https://review.openstack.org/56411 | 18:17 |
| mordred | there you go - I think that's more correct | 18:17 |
| mordred | I updated the commit message | 18:17 |
| clarkb | mrodden: that shouldn't be a problem and I think is the behavior mordred wants to rely on | 18:17 |
| clarkb | we don't explicitly need pip because we have virtualenv which gives us pip | 18:17 |
| clarkb | jog0: so 6GB helped? | 18:17 |
| mordred | yup | 18:18 |
| *** ruhe has joined #openstack-infra | 18:19 | |
| jog0 | I figure wait a week or so and see if that fixes it all together or just makes it happen much much less | 18:19 |
| jog0 | mordred: http://tinyurl.com/mqju53r | 18:19 |
| jog0 | graphite goodness | 18:19 |
| *** dims has joined #openstack-infra | 18:22 | |
| *** nsaje has joined #openstack-infra | 18:24 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Separate mailing list for StackForge projects https://review.openstack.org/56432 | 18:25 |
| *** Ryan_Lane has joined #openstack-infra | 18:25 | |
| jog0 | fungi: woot | 18:25 |
| *** blamar has quit IRC | 18:26 | |
| *** pcm_ has quit IRC | 18:26 | |
| *** blamar has joined #openstack-infra | 18:27 | |
| *** nsaje has quit IRC | 18:27 | |
| *** nsaje has joined #openstack-infra | 18:29 | |
| *** Ryan_Lane has quit IRC | 18:29 | |
| *** Ryan_Lane has joined #openstack-infra | 18:29 | |
| *** nsaje has quit IRC | 18:30 | |
| *** krotscheck has joined #openstack-infra | 18:31 | |
| *** pcm_ has joined #openstack-infra | 18:31 | |
| *** changbl has joined #openstack-infra | 18:35 | |
| *** cody-somerville has joined #openstack-infra | 18:37 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Separate mailing list for StackForge projects https://review.openstack.org/56432 | 18:39 |
| fungi | grr, fat fingers this morning | 18:39 |
| fungi | well, afternoon now | 18:39 |
| *** ni273536 has joined #openstack-infra | 18:39 | |
| *** jpich has quit IRC | 18:40 | |
| *** krtaylor has joined #openstack-infra | 18:41 | |
| mordred | fungi: it's always morning in my brain | 18:41 |
| fungi | moar coffeeeeeeeEEE | 18:42 |
| zaro | mee tooOOOOOO | 18:42 |
| clarkb | mordred: can you review fungi's first folsom killing patches | 18:42 |
| mordred | clarkb: I landed them already I believe | 18:43 |
| clarkb | mordred: and do you have time to figure out why the projects.yaml changes that were merged yesterday didn't do anything | 18:43 |
| clarkb | mordred: oh great | 18:43 |
| mordred | clarkb: race condition. I re-ran manage projects and all is good | 18:43 |
| clarkb | is that a new problem | 18:43 |
| mordred | nope. it's the same old one | 18:44 |
| mordred | we need to put a retry in there I think | 18:44 |
| *** vipul-away is now known as vipul | 18:46 | |
| fungi | the last few times i've witnessed it, the impression i get is that we're making too many calls to github's api and either catching some small percentage of call failures which makes us bomb out prematurely, or tripping a threshold and getting ungracefully throttled/blocked | 18:46 |
| fungi | lately if i try to manually fire a full manage-projects, it rarely runs to completion and usually bombs out after a bunch of github api errors | 18:48 |
| fungi | but running it for just one project usually works fine | 18:48 |
| mordred | yah. this is one of the reasons I want better stateful stuff | 18:48 |
| mordred | I suppose I could rewrite manage-projects to use taskflow | 18:49 |
| *** che-arne has quit IRC | 18:50 | |
| clarkb | :/ we can probably do a lot to optimize the number of github api calls we need | 18:52 |
| clarkb | also make the PR closer bot run once an hour or so | 18:52 |
| anteaya | did we identify this was a process to consider salting? | 18:53 |
| *** mrmartin has joined #openstack-infra | 18:53 | |
| anteaya | wikipedia seems to be down | 18:54 |
| *** HenryG has joined #openstack-infra | 18:54 | |
| anteaya | no sorry, just the page I was after | 18:55 |
| mordred | clarkb, fungi: https://review.openstack.org/#/c/53899/ <-- I think we should add iroinc to the gate jobs before landing that | 18:55 |
| mordred | anteaya: yes, we did identify that as a thing that might want salting | 18:55 |
| fungi | though in this case the benefit from salt is further downstream from where manage-projects is running | 18:56 |
| fungi | for example on the git servers | 18:56 |
| anteaya | mordred: so would you be re-writing manage-projects as well? | 18:56 |
| anteaya | fungi: ah | 18:56 |
| *** markmcclain has quit IRC | 18:57 | |
| mordred | anteaya: dunno. haven't thought enough about it yet | 18:57 |
| anteaya | k | 18:57 |
| anteaya | I'm sure it will come to you in a vision | 18:58 |
| *** mrmartin has quit IRC | 18:58 | |
| fungi | mordred: agreed. self-testing that change would be helpful | 18:58 |
| *** jhesketh__ has quit IRC | 18:58 | |
| *** markmcclain has joined #openstack-infra | 18:58 | |
| clarkb | ++ | 18:59 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add Ironic to the gate https://review.openstack.org/56439 | 19:00 |
| mordred | devananda: ^^ | 19:01 |
| *** changbl has quit IRC | 19:01 | |
| mordred | devananda: right? | 19:01 |
| mrodden | reading through http://ci.openstack.org/stackforge.html, would i just have to push up a patch to add some new projects to stackforge? | 19:01 |
| *** jorisroovers has quit IRC | 19:01 | |
| mordred | mrodden: yup | 19:01 |
| mrodden | looking at adding chef cookbooks for db2 and qpid | 19:01 |
| mordred | mrodden: there's a pile of chef cookbooks on stackforge already in fact | 19:01 |
| mrodden | right | 19:01 |
| mrodden | the db2 one was what i was mainly wondering about | 19:01 |
| mrodden | we use it and would liek to make it available for others outside to use | 19:02 |
| mrodden | but i'm not sure how "openstack related" is it | 19:02 |
| clarkb | jhesketh: you are now a member of turbo-hipster-core. you can add more core members as you like | 19:02 |
| *** ni273536 has quit IRC | 19:02 | |
| mrodden | it is* | 19:02 |
| * mrodden needs more caffiene | 19:03 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Add taskflow (used by cinder + others) to requirements https://review.openstack.org/52725 | 19:03 |
| harlowja | mordred hope thats ok for u | 19:03 |
| harlowja | ^^ | 19:03 |
| devananda | mordred: since i'm not totally sure how to read that, i assume it means those tests will be run at check and gate for changes to ironic and python-ironicclient | 19:06 |
| mordred | devananda: yes | 19:06 |
| dims | harlowja, nice! any reason for "<0.2"? (api changes?) | 19:07 |
| mordred | devananda: romcheg submitted a patch to add ironic ot devstack-gate - so I figured we should start running devstack-gate on ironic changes before turning that on | 19:07 |
| devananda | mordred: and we don't want to enable ironic in check-tempest-devstack-vm-full in general yet | 19:07 |
| harlowja | dims seems safe, 0.2 doesn't exist yet, so it was relatively arbitary :-P | 19:07 |
| harlowja | i can put < 1.0 | 19:07 |
| devananda | mordred: right. that's romcheg's patch. let's not enable it everywhere yet :) | 19:07 |
| mordred | devananda: we don't? | 19:07 |
| harlowja | dims i'm up for making it a different number :) | 19:07 |
| devananda | mordred: well. we do. but let's get a little confidence first | 19:07 |
| mordred | no - romcheg's patch would enable it everywhere | 19:07 |
| devananda | mordred: i think we're saying the same thing now | 19:08 |
| devananda | enable all the check, gate, post, etc, on ironic changes. but don't gate other projects' changes on ironic -yet-. right? | 19:09 |
| dims | harlowja, was curious that's all. | 19:09 |
| harlowja | np :) | 19:09 |
| *** rnirmal has joined #openstack-infra | 19:09 | |
| mordred | devananda: right. which means don't land romcheg's patch yet | 19:09 |
| devananda | right | 19:09 |
| mordred | however, we probably want to make a job that only runs on ironic changes which adds the ironic services ot the list before running devstack | 19:10 |
| reed | mordred, thanks for pushing to public the discussion about core/integrated from the foundation-board list | 19:10 |
| mordred | so that you can start to get comfortable | 19:10 |
| devananda | mordred: yes | 19:10 |
| devananda | mordred: i think taht's waht romcheg was trying to do | 19:10 |
| mordred | devananda: awesome. I will leave some feedback | 19:10 |
| devananda | ty | 19:10 |
| *** CaptTofu has quit IRC | 19:11 | |
| *** CaptTofu has joined #openstack-infra | 19:11 | |
| *** CaptTofu has quit IRC | 19:12 | |
| *** CaptTofu has joined #openstack-infra | 19:12 | |
| *** ben_duyujie has quit IRC | 19:13 | |
| mordred | devananda: I have left feedback on romcheg's patch | 19:15 |
| devananda | awesome | 19:15 |
| *** changbl has joined #openstack-infra | 19:20 | |
| ArxCruz | ALL: Is there a way in jenkins to skip some tests when tempest jobs are running ? | 19:21 |
| *** Ryan_Lane has quit IRC | 19:24 | |
| *** ruhe has quit IRC | 19:25 | |
| *** jorisroovers has joined #openstack-infra | 19:27 | |
| *** jroovers has joined #openstack-infra | 19:27 | |
| *** milki has joined #openstack-infra | 19:28 | |
| *** rhsu1 has joined #openstack-infra | 19:29 | |
| *** Ryan_Lane has joined #openstack-infra | 19:29 | |
| reed | Ryan_Lane, pleia2: the meeting next week is canceled, thanks for offering availability | 19:30 |
| krtaylor | ArxCruz, to clarify, other than manually adding decorators | 19:30 |
| ArxCruz | krtaylor: yes, exactly :) | 19:30 |
| ArxCruz | krtaylor: I believe mordred have the answer ;) | 19:31 |
| krtaylor | always | 19:31 |
| *** rhsu has quit IRC | 19:31 | |
| *** jorisroovers has quit IRC | 19:31 | |
| dripton | So I git pushed a tag for sqlalchemy-migrate 0.8 release to gerrit a couple of hours ago. stackforge sees it but PyPI did not update. Anyone know where any logs would be where I could check for errors? | 19:32 |
| fungi | dripton: did you follow the instructions at https://wiki.openstack.org/wiki/GerritJenkinsGit#Tagging_a_Release to the letter (signed tag, et cetera)? | 19:33 |
| fungi | i'll take a look | 19:33 |
| fungi | looks like you did... https://git.openstack.org/cgit/stackforge/sqlalchemy-migrate/tag/?id=0.8 | 19:34 |
| fungi | i'll dig into the job logs | 19:35 |
| *** toddmorey has quit IRC | 19:35 | |
| dripton | fungi: I did sign the tag, but I didn't do a milestone-proposed branch because I was just tagging existing work on master. I don't *think* that should matter but... | 19:35 |
| fungi | it shouldn't | 19:36 |
| fungi | pypi upload job seems to have gotten a 404 grabbing the tarball: https://jenkins.openstack.org/job/sqlalchemy-migrate-pypi-upload/1/console | 19:37 |
| fungi | checking to see where the tarball build ran and what got uploaded | 19:37 |
| mordred | dripton: a m-p branch is not needed | 19:37 |
| *** ^d is now known as ^demon|lunch | 19:38 | |
| fungi | versioning problem... look at the end of https://jenkins02.openstack.org/job/sqlalchemy-migrate-tarball/1/console | 19:38 |
| fungi | sqlalchemy-migrate-0.8.dev.tar.gz | 19:38 |
| dripton | I also get a 404 trying to get that tarball, but http://tarballs.openstack.org/sqlalchemy-migrate/sqlalchemy-migrate-0.8.dev.tar.gz seems to exist. The "dev" is the difference | 19:38 |
| *** yolanda has quit IRC | 19:39 | |
| *** saper has quit IRC | 19:44 | |
| *** saper has joined #openstack-infra | 19:45 | |
| fungi | dripton: yeah, if i 'git checkout 0.8' and 'tox -evenv python setup.py sdist' i too get a sqlalchemy-migrate-0.8.dev.tar.gz rather than sqlalchemy-migrate-0.8.tar.gz | 19:47 |
| dripton | ok, I'll dig into the setup.py and see if I can make "dev" go away. Thanks fungi | 19:47 |
| fungi | np | 19:48 |
| *** jamesmcarthur has quit IRC | 19:48 | |
| fungi | keep in mind you'll want to release a 0.8.1 or something for that. no take-backsies on replicated tags. causes a huge mess | 19:48 |
| dripton | fungi: ack | 19:48 |
| fungi | dripton: normally the .dev suffix is, i think, something pbr adds when it thinks you're on a commit which isn't tagged. in this case that seems wrong so something's probably being misdetected there | 19:49 |
| fungi | mordred: ^ alternate ideas? | 19:49 |
| dripton | fungi: I see tag_build = .dev in setup.cfg Chasing through docs to see if that means something. | 19:50 |
| *** Ryan_Lane has quit IRC | 19:51 | |
| *** Ryan_Lane has joined #openstack-infra | 19:51 | |
| *** apevec has quit IRC | 19:52 | |
| SpamapS | Hi. We have a "contrib" set of unit tests in Heat that really need to be able to run more like stackforge tests, even with an alternate requirements.txt, because we want contrib to be able to depend on things not in global req's. But we'd like to gate Heat on those unit tests passing. Thoughts? | 19:52 |
| clarkb | SpamapS: I would setup a new job that runs those tests with different config settints | 19:53 |
| clarkb | then you can gate on it and it can have different requirements and so on | 19:53 |
| SpamapS | clarkb: ok cool I thought that would work. Thanks. | 19:54 |
| fungi | dripton: yeah, based on what i've seen in our other projects using pbr we usually don't set that, or we set it to an empty value | 19:54 |
| dripton | fungi: removing that tag_build line from setup.cfg then running your tox line made the .dev go away | 19:54 |
| dripton | fungi: so I'm going to push a review to remove that tag_build line, then try another release | 19:54 |
| clarkb | though making it do different requirements is an exercise left to the reader :) | 19:54 |
| SpamapS | clarkb: tox.ini can do it right? | 19:54 |
| reed | i have finally got my my hands on the archives of the old launchpad list | 19:54 |
| SpamapS | deps = -r{toxinidir/contrib/requirements.txt} | 19:55 |
| clarkb | SpamapS: sort of, pbr will actually use requirements.txt by default even without tox | 19:55 |
| fungi | dripton: from what i saw of the jobs which triggered at least, it should get as far as trying to upload it to pypi, so if it fails next time it's probably either going to be permissions or invalid trove classifiers | 19:55 |
| reed | rockstar, yt? | 19:55 |
| clarkb | so you will need to convince pbr that the requirements it needs are elsewhere | 19:55 |
| clarkb | I think there may be an env var you can set to do that | 19:55 |
| SpamapS | oh lurvely | 19:55 |
| clarkb | or an env var to install no requirements | 19:56 |
| clarkb | then you can do dep management outside of pbr | 19:56 |
| fungi | the skip_pip_somefoo envvar for pbr i think? | 19:56 |
| * fungi doesn't recall the particular envvar name | 19:57 | |
| fungi | clarkb: SpamapS: SKIP_PIP_INSTALL=1 | 19:57 |
| *** vipul is now known as vipul-away | 19:57 | |
| *** markwash has joined #openstack-infra | 19:58 | |
| fungi | unless that went away in favor of single-version-externally-managed? | 19:58 |
| fungi | mordred: ^ ? | 19:59 |
| *** jamesmcarthur has joined #openstack-infra | 19:59 | |
| SpamapS | ok so that would allow us to setup our venv with tox's deps= bit and then run tests? | 19:59 |
| clarkb | SpamapS: yup that is the idea at least | 19:59 |
| fungi | SpamapS: i believe so. you might also have a look at what horizon's tox.ini does for testing with older django | 19:59 |
| fungi | as far as having tox second-guess your deps for a particular testenv | 20:01 |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 20:02 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 20:03 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 20:03 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 20:03 | |
| *** marcrp has joined #openstack-infra | 20:05 | |
| *** marcrp has quit IRC | 20:06 | |
| *** gyee has quit IRC | 20:06 | |
| *** marcrp has joined #openstack-infra | 20:08 | |
| rockstar | reed, hi | 20:09 |
| reed | rockstar, do you have time to merge the old mbox archives of the general list on lists.openstack.org? | 20:10 |
| rockstar | reed, I don't currently, but I can find time soon. | 20:10 |
| *** dstanek has joined #openstack-infra | 20:10 | |
| reed | rockstar, ok, I'll keep you posted | 20:11 |
| clarkb | today is information overload day, I have just managed to finish skimming email. do I read irc scrollback? at this rate I am going to get nothing done :) | 20:12 |
| clarkb | JJB core folks, we have a bug requesting a new JJB release and I think that is a great idea. I haven't been able to keep up with JJB reviews recently. Are there outstanding changes that we should try to review and merge before pushing a new release? | 20:13 |
| clarkb | hashar zaro mgagne ^ | 20:13 |
| mgagne | clarkb: we should check for pending bugs. new features can wait otherwise we will never be ready to release, it's an ongoing work. otherwise we can do a rush and try to merge a much change as possible before the release. does it have to be today? | 20:15 |
| *** denis_makogon has joined #openstack-infra | 20:15 | |
| clarkb | mgagne: it doesn't need to be today | 20:16 |
| clarkb | and ++ to focusing on bug fixes | 20:16 |
| *** julim has quit IRC | 20:17 | |
| *** vipul-away is now known as vipul | 20:18 | |
| *** ni313896 has joined #openstack-infra | 20:18 | |
| clarkb | I don't think we have to rush, more take a look at the queue figure out which things should go in (bug fixes simple features ready to go etc), get those in then releaase | 20:20 |
| zaro | actually last release was on 2013-06-20 so a release is due. | 20:20 |
| *** toddmorey has joined #openstack-infra | 20:21 | |
| zaro | i'm reviewing patches today. i suggest we do one at end of week. | 20:21 |
| clarkb | yup. I just don't want to force one out. Want to make sure we are actually happy with what we release | 20:21 |
| zaro | i'll try to get as many off as i can by then. | 20:21 |
| *** marcrp has quit IRC | 20:22 | |
| zaro | mgagne: you like that idea? | 20:23 |
| LinuxJedi | mordred: (or anyone) can you help marcrp with a git-review -D problem he has hit? | 20:23 |
| LinuxJedi | damn... | 20:23 |
| mgagne | zaro: yes, we can try to merge as much as possible before friday and then release | 20:23 |
| mgagne | zaro: it's not like we cannot do an other release after that one | 20:23 |
| clarkb | LinuxJedi: sure, stop using git review -D :) | 20:23 |
| *** marun has quit IRC | 20:24 | |
| LinuxJedi | clarkb: it is known not to work? | 20:24 |
| zaro | mgagne: you like the idea of 3 month cadence? | 20:25 |
| mgagne | zaro: why not =) | 20:25 |
| *** yolanda has joined #openstack-infra | 20:25 | |
| clarkb | LinuxJedi: no, it should work. But drafts themselves are problematic (it is why we have WIP instead) | 20:25 |
| clarkb | LinuxJedi: so general advice is to not use drafts | 20:25 |
| clarkb | funny things happen when you have a parent whose change is a draft, people get confused when tests don't run or why they can't vote on a change and so on | 20:25 |
| LinuxJedi | clarkb: the problem here is after a rebase it doesn't want to push the draft up :) | 20:26 |
| *** markmcclain has quit IRC | 20:27 | |
| fungi | and the only reasons why we generally see people using drafts are because they think it's secure (which it isn't), because they don't feel the change is ready for review yet (in which case set it wip), or because they're not comfortable working in the open without a lot of polish first (in which case, sorry, get used to making mistakes in public) | 20:28 |
| zaro | mgagne: ok. we don't have permissions to release. so we just need to shout in this channel every 3 months. | 20:28 |
| *** ni313896 has quit IRC | 20:29 | |
| fungi | LinuxJedi: got a paste of the error? i think you can't push a new patchset to someone else's change if the most recent existing patchset on it is a draft | 20:30 |
| fungi | so there's one possible cause | 20:30 |
| LinuxJedi | http://paste.openstack.org/show/52917/ | 20:31 |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Don't upgrade setuptools or distribute https://review.openstack.org/56411 | 20:31 |
| clarkb | zaro: mgagne: I think we can change who has access to do releases. It is currently set to openstack-ci-core. I wouldn't change it to JJB-core simply because we like the release groups to be small, but that small group can be something different than openstack-ci-core and JJB-core | 20:31 |
| Shrews | substituting WIP for draft is not always an acceptable solution. If core devs don't want to see endless emails about reviews they may have to make, then that doesn't work. | 20:31 |
| *** yolanda has quit IRC | 20:32 | |
| Shrews | s/make/look at/ | 20:32 |
| mgagne | clarkb, zaro: is zaro familiar with the release process? | 20:32 |
| LinuxJedi | its ok, I /dev/null most of my openstack mail now anyway, volume is too high because I still get CI stuff :) | 20:32 |
| fungi | LinuxJedi: that change is already merged, so that error is normal | 20:32 |
| LinuxJedi | fungi: that isn't the change he is drafting though | 20:33 |
| LinuxJedi | fungi: that is the one he rebased against | 20:33 |
| fungi | LinuxJedi: did he happen to reuse the change-id header from that merged one though? | 20:33 |
| fungi | also results in the same error message i think | 20:33 |
| LinuxJedi | he probably used whatever the default does, and I suspect that is what the default does | 20:34 |
| *** esker has quit IRC | 20:34 | |
| fungi | the default is for git-review to generate a new change-id. i mean did he reuse the change-id from the earlier merged change, not from an earlier patchset of the current change | 20:34 |
| clarkb | lifeless: comment for you on https://review.openstack.org/#/c/55943/ | 20:35 |
| fungi | i've seen people cut-paste-edit commit messages and forget to throw away the old change-id | 20:35 |
| clarkb | mgagne: he is, he has done them for the gearman jenkins plugin | 20:35 |
| LinuxJedi | no idea. Would be easier if I could show him how to use IRC :< | 20:36 |
| LinuxJedi | :\ | 20:36 |
| fungi | LinuxJedi: you are welcome to borrow my clue by four, or any other lart of your choice ;) | 20:36 |
| fungi | i think i have some cluebats, clueclubs and cluepaddles lying around here too | 20:37 |
| *** hashar has joined #openstack-infra | 20:37 | |
| lifeless | clarkb: answered | 20:38 |
| jog0 | so I am seeing grenade timeout in the check queue | 20:39 |
| jog0 | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQnVpbGQgdGltZWQgb3V0IChhZnRlciA2MCBtaW51dGVzKVwiIEFORCBidWlsZF9uYW1lOlwiY2hlY2stZ3JlbmFkZS1kZXZzdGFjay12bVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg0NDYxNTI2MDQ1LCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9 | 20:39 |
| clarkb | lifeless: so with someFixture() as foo: foo.dostuff ? | 20:39 |
| fungi | jog0: quite possible. check jobs can end up running on slower rackspace slaves | 20:39 |
| lifeless | clarkb: f'rinstance yes | 20:39 |
| lifeless | clarkb: or with someFixture() as foo: r = foo.repo | 20:40 |
| lifeless | or whatever | 20:40 |
| jog0 | fungi: can we bump the timeout by a little bit? | 20:40 |
| jog0 | say 70? | 20:40 |
| jog0 | or is that a bad idea? | 20:40 |
| jog0 | http://logs.openstack.org/30/56430/4/check/check-grenade-devstack-vm/56efb52/console.html | 20:41 |
| jog0 | its timeing out just at the end | 20:41 |
| zaro | mgagne: yes, i'm familiar with how to release. | 20:41 |
| fungi | jog0: seems like an okay choice to me, unless you think you can make grenade faster | 20:41 |
| fungi | jog0: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#n834 | 20:41 |
| fungi | if you want to propose a change to that, we can discuss it in review | 20:41 |
| jog0 | sdague: ^^ | 20:42 |
| jog0 | fungi: cool | 20:42 |
| zaro | clarkb: i'll create a patch to setup new gerrit group to allow releasing jjb? | 20:42 |
| jog0 | I know how to make grenade faster, get more sdague's | 20:42 |
| fungi | jog0: the factory where they made them burned down, or so i heard. extreme supply shortage | 20:42 |
| zaro | clarkb: how about new group name 'JJB-project-managers'? | 20:44 |
| sdague | jog0: can graph the grenade run times? | 20:44 |
| mgagne | zaro: sorry if I can't be more active atm, lot to catch up with here | 20:44 |
| sdague | I'm really surprised something hit 60 minutes | 20:44 |
| sdague | that was running at 24 pretty regularly | 20:45 |
| jog0 | sdague: don't know where we record run times | 20:46 |
| jog0 | oh wait jenkins does taht | 20:46 |
| * jog0 files a bug on ifra | 20:46 | |
| sdague | I thought there was a meter on it in graphite already | 20:47 |
| jog0 | https://jenkins02.openstack.org/job/check-grenade-devstack-vm/buildTimeTrend | 20:47 |
| *** krotscheck has quit IRC | 20:47 | |
| jog0 | sdague: if so I don't know where to look in graphite | 20:47 |
| fungi | sdague: jog0: if you're only seeing it on check jobs and not on gate jobs, it's possible the reason is that some portion of check jobs run in rackspace rather than hpcloud | 20:47 |
| fungi | clarkb: did we get a way in logstash to correlate by provider? | 20:48 |
| jog0 | fungi: that is what I am seeing | 20:48 |
| *** julim has joined #openstack-infra | 20:48 | |
| clarkb | fungi: no it requires extra magic | 20:48 |
| sdague | yeh, so that graph tells me that the only time we hit the timeout, we were going to fail anyway | 20:48 |
| clarkb | since jenkins doesn't actually know anything about providers | 20:48 |
| * jog0 wants to file a bug on rax cloud: your slow | 20:48 | |
| fungi | we are fresh out of extra magic | 20:48 |
| clarkb | you can see the node name in the console logs thouhg | 20:48 |
| *** yolanda has joined #openstack-infra | 20:48 | |
| sdague | and the timeout was hit because one of the tests was wedged and timed out | 20:48 |
| jog0 | sdague: where is the failure in http://logs.openstack.org/30/56430/4/check/check-grenade-devstack-vm/56efb52/console.htm | 20:49 |
| jog0 | or http://logs.openstack.org/40/56440/1/check/check-grenade-devstack-vm/c80ccb7/console.html | 20:50 |
| *** alexpilotti has quit IRC | 20:51 | |
| fungi | yeah, if you're seeing it consistently on a variety of changes, and it got most of the way through before being killed, then bumping the timeout a little may make sense. if you're seeing it early in runs, infrequently, and not correlated well to a particular provider then it probably won't help you | 20:51 |
| jog0 | clarkb: filed a bug for you https://bugs.launchpad.net/openstack-ci/+bug/1251423 | 20:51 |
| *** yolanda has quit IRC | 20:51 | |
| sdague | grenade not running in parallel? | 20:51 |
| jog0 | haha | 20:52 |
| jog0 | lets change that instead of the timeout | 20:52 |
| sdague | so... you'll have to create a second smoke target | 20:52 |
| sdague | right we use smoke for neutron and for grenade, for different reasons | 20:52 |
| sdague | so smoke is set serial right now | 20:53 |
| clarkb | speaking of smoke and exercises we should add gating jobs to devstack that run those | 20:53 |
| jog0 | can we just change smoke to parallel? | 20:53 |
| clarkb | just to make sure we don't regress them | 20:53 |
| jog0 | clarkb: ++ | 20:53 |
| sdague | jog0: no, then you fail all neutron jobs | 20:53 |
| jog0 | don't they fail anyway ;) | 20:54 |
| sdague | jog0: ok, so do this | 20:54 |
| sdague | 1) tox.ini for a smoke serial target | 20:54 |
| sdague | 2) layout.yaml chang to move neutron to smoke serial | 20:54 |
| sdague | 3) tox.ini change for smoke to be parallel | 20:54 |
| sdague | and I'll make sure the tempest reviews get priority to get them through | 20:55 |
| jog0 | sdague: ack | 20:55 |
| sdague | but we need to stepwise them to not break neutron | 20:55 |
| *** rfolco has quit IRC | 20:57 | |
| *** ^demon|lunch is now known as ^d | 20:58 | |
| *** jpeeler has quit IRC | 21:01 | |
| *** datsun180b has quit IRC | 21:01 | |
| *** SergeyLukjanov has quit IRC | 21:03 | |
| *** jpeeler has joined #openstack-infra | 21:03 | |
| *** jpeeler has quit IRC | 21:03 | |
| *** jpeeler has joined #openstack-infra | 21:03 | |
| *** hashar has quit IRC | 21:04 | |
| *** hashar has joined #openstack-infra | 21:05 | |
| *** dstanek has left #openstack-infra | 21:05 | |
| *** vipul is now known as vipul-away | 21:06 | |
| *** sandywalsh has quit IRC | 21:09 | |
| *** pabelanger has joined #openstack-infra | 21:10 | |
| *** HenryG has quit IRC | 21:11 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Wire up support for DEVSTACK_GATE_SMOKE_SERIAL https://review.openstack.org/56466 | 21:11 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Explicitly set smoke tests in neutron test to be serial https://review.openstack.org/56467 | 21:11 |
| jog0 | sdague: ^ | 21:11 |
| portante | just has a check job time out after 60 minutes | 21:11 |
| portante | swift | 21:11 |
| jog0 | portante: what check job? | 21:12 |
| portante | http://logs.openstack.org/40/56440/1/check/check-grenade-devstack-vm/c80ccb7 | 21:12 |
| jog0 | grenade? | 21:12 |
| portante | yes | 21:12 |
| jog0 | portante: we are fixing that right now | 21:12 |
| portante | the tests were running fine | 21:12 |
| portante | okay | 21:12 |
| jog0 | just pushed up 4 patches for it | 21:12 |
| portante | thanks | 21:12 |
| jog0 | the short is rax is slow | 21:12 |
| *** dkliban has quit IRC | 21:12 | |
| *** pcm_ has quit IRC | 21:12 | |
| sdague | jog0: did you push the tempest patch? | 21:13 |
| sdague | for the new target? | 21:13 |
| fungi | did it get narrowed down to that being the deciding factor (timeouts were consistently on rax nodes, not just in teh check pipeline)? | 21:13 |
| jog0 | https://review.openstack.org/#/c/56464/ | 21:13 |
| jog0 | https://review.openstack.org/#/c/56465/ | 21:13 |
| jog0 | fungi: a spot check pointed to only timeouts on rax | 21:13 |
| jog0 | fungi: see https://jenkins02.openstack.org/job/check-grenade-devstack-vm/buildTimeTrend | 21:14 |
| jog0 | fungi: and search for '1 hr 0 min' | 21:14 |
| jog0 | sdague: I am standing by to fix your -1s on my patches | 21:15 |
| sdague | jog0: I'm good on the first one | 21:15 |
| jog0 | I think I missed something in the d-g and config patches | 21:15 |
| sdague | and I got to do office packing for the next hour, so I'll come back and make sure it lands | 21:15 |
| jog0 | fungi: ^ | 21:15 |
| fungi | ah, cool | 21:16 |
| jog0 | clarkb: btw I know we have said this a lot but, I want to say it again elasticsearch is amazing | 21:19 |
| *** vipul-away is now known as vipul | 21:19 | |
| *** julim has quit IRC | 21:19 | |
| *** sandywalsh has joined #openstack-infra | 21:21 | |
| fungi | clarkb: mordred: *sigh* http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 21:21 |
| fungi | from logstash to logcrash | 21:21 |
| *** ianj1 has joined #openstack-infra | 21:22 | |
| fungi | we've nearly burned through 0.5tib of log storage in a little over a week | 21:22 |
| fungi | did we perchance get more verbose on anything of note? (or maybe just started running more tests?) | 21:23 |
| *** svarnau has joined #openstack-infra | 21:24 | |
| *** sarob has joined #openstack-infra | 21:27 | |
| dims | weird, lot more gate jobs are in queued status compared to check jobs in zuul...any ideas? | 21:27 |
| *** dkliban has joined #openstack-infra | 21:27 | |
| *** svarnau has quit IRC | 21:27 | |
| jhesketh | clarkb: thanks for setting up turbo-hipster :-) | 21:28 |
| *** sarob has quit IRC | 21:28 | |
| *** jhesketh_ has joined #openstack-infra | 21:29 | |
| *** sarob has joined #openstack-infra | 21:29 | |
| jog0 | dims: yes | 21:30 |
| fungi | dims: i think it's just a matter of which ones got uploaded/approved first | 21:31 |
| *** sarob has quit IRC | 21:31 | |
| jog0 | dims: if gate is flakey (see http://tinyurl.com/mqju53r) gate will slow way down | 21:31 |
| *** sarob_ has joined #openstack-infra | 21:31 | |
| dims | cool. i should stop staring at a specific job :) | 21:31 |
| jog0 | due to many jobs being restarted | 21:31 |
| *** vipul is now known as vipul-away | 21:31 | |
| *** vipul-away is now known as vipul | 21:31 | |
| dims | i see | 21:31 |
| *** sarob_ has quit IRC | 21:32 | |
| clarkb | jog0: :) | 21:32 |
| clarkb | fungi: no idea | 21:32 |
| *** sarob has joined #openstack-infra | 21:33 | |
| clarkb | elasticsearch hasn't seen huge growth | 21:33 |
| clarkb | but we are filtering everything <= DEBUG out | 21:33 |
| jog0 | dims: with gate jobs failing at about 7% of the time, and 3 jobs, gate will fail 20% oftime | 21:33 |
| clarkb | fungi: maybe our deletes aren't working properly? | 21:33 |
| jog0 | so stablize gate, gate queue goes faster | 21:33 |
| fungi | clarkb: it's possible | 21:33 |
| *** senk has joined #openstack-infra | 21:34 | |
| jog0 | clarkb: can you review https://review.openstack.org/#/q/status:open+project:openstack-infra/devstack-gate+branch:master+topic:grenade,n,z | 21:34 |
| *** Ryan_Lane has quit IRC | 21:34 | |
| jog0 | which is linked with https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:smoke,n,z | 21:34 |
| *** Ryan_Lane has joined #openstack-infra | 21:35 | |
| jog0 | dims: see my attempt at doing basic math about why gate is slow http://lists.openstack.org/pipermail/openstack-dev/2013-November/019042.html | 21:35 |
| dims | ah thanks | 21:36 |
| *** talluri has quit IRC | 21:36 | |
| *** senk1 has joined #openstack-infra | 21:36 | |
| jog0 | clarkb: https://review.openstack.org/#/q/status:open+branch:master+topic:grenade,n,z * | 21:37 |
| fungi | clarkb: i'm doing 'find /srv/static/logs/ -type f -mtime +191 -name \*.gz' to see if stuff got left behind when last weekend's delete ran | 21:38 |
| *** zhiyan has quit IRC | 21:38 | |
| clarkb | fungi: ok | 21:38 |
| fungi | nothing so far | 21:38 |
| *** senk has quit IRC | 21:38 | |
| anteaya | clarkb: is dekehn in the office today? | 21:39 |
| fungi | and i do see a little dip in the graph over the weekend, so it did something at least | 21:39 |
| clarkb | fungi: http://logs.openstack.org/85/53985/1/gate/gate-tempest-devstack-vm-neutron/237df19/logs/ n-cpu is fairly large | 21:39 |
| anteaya | he isn't on irc and someone has a questions about code he wrote | 21:39 |
| clarkb | anteaya: dkehn works from home | 21:39 |
| anteaya | hmmmm | 21:39 |
| clarkb | in another state :) | 21:39 |
| anteaya | oh, I thought he was in seattle | 21:39 |
| fungi | boulder | 21:40 |
| anteaya | cool | 21:40 |
| *** adalbas has quit IRC | 21:40 | |
| anteaya | that's not seattle | 21:40 |
| fungi | nope | 21:40 |
| dims | clarkb, that n-cpu has the unncecessary lines from iso8601.iso8601 | 21:40 |
| *** Ryan_Lane has quit IRC | 21:40 | |
| clarkb | fungi: n-cpu being 8MB compressed is that only thing that stands out to me in that random set of logs | 21:40 |
| fungi | ooh, so maybe it's the iso8601 verbosity issue biting us | 21:40 |
| *** Ryan_Lane has joined #openstack-infra | 21:40 | |
| clarkb | fungi: could be | 21:40 |
| fungi | when did that crop up? | 21:41 |
| anteaya | his irc nick and gerrit nick are off by one e | 21:41 |
| dims | fungi, y, there was a change in oslo, let me sync it to nova | 21:41 |
| anteaya | he is here | 21:41 |
| clarkb | that basically doubled our log size after compression (using back of napkin maths) | 21:41 |
| *** gyee has joined #openstack-infra | 21:41 | |
| fungi | those are the best maths | 21:41 |
| fungi | in a bar setting, those are in fact the only maths | 21:41 |
| *** senk1 has quit IRC | 21:42 | |
| *** senk1 has joined #openstack-infra | 21:42 | |
| *** dstanek has joined #openstack-infra | 21:44 | |
| *** Ryan_Lane has quit IRC | 21:44 | |
| *** toddmorey has quit IRC | 21:46 | |
| *** DennyZhang has joined #openstack-infra | 21:46 | |
| *** vipul is now known as vipul-away | 21:47 | |
| Shrews | fungi: when i'm in a bar, math is the LAST thing on my mind | 21:47 |
| *** slong has joined #openstack-infra | 21:47 | |
| Shrews | fungi: when i'm NOT in a bar, math is the LAST thing on my mind | 21:48 |
| Shrews | speaking of which... we need to go to a bar soon | 21:48 |
| Shrews | and do math or something | 21:48 |
| fungi | Shrews: beermaths. almost but not quite entirely unlike bistromathematics | 21:48 |
| * Shrews pokes olaph | 21:49 | |
| fungi | not to be confused with beermats, which are a totally different sort of thing | 21:49 |
| fungi | Shrews: olaph: name the time and place (or ask me to name them) and i'll be there | 21:49 |
| *** toddmorey has joined #openstack-infra | 21:50 | |
| dims | fungi, clarkb - this https://review.openstack.org/56471 should take care of the log sizes. if not, can look further after this lands | 21:51 |
| *** vipul-away is now known as vipul | 21:51 | |
| clarkb | dims: fungi: thankfully we will see in the check jobs if that helps | 21:52 |
| fungi | dims: okay cool. in the meantime, since the existing logs aren't going to age out until the spring thaw, i should probably add another terabyte to static.o.o | 21:52 |
| *** lcestari has quit IRC | 21:52 | |
| fungi | which at this point is only a 25% increase to /srv/static/logs anyway | 21:53 |
| fungi | clarkb: mordred: ^ opinion? | 21:53 |
| clarkb | fungi: ya I think that is reasonable | 21:54 |
| *** ^d has quit IRC | 21:54 | |
| olaph | bar = very yes | 21:54 |
| *** dkranz has quit IRC | 21:55 | |
| *** dkranz has joined #openstack-infra | 21:55 | |
| olaph | the past several times have been in Durham, so I suspect it's fungi's turn to pick | 21:57 |
| *** dprince has quit IRC | 21:57 | |
| * olaph goes to milk before daylight disappears | 21:58 | |
| openstackgerrit | Peter Liljenberg proposed a change to openstack-infra/jenkins-job-builder: Added support for Jenkins plugin Blame upstream committers https://review.openstack.org/54085 | 21:59 |
| fungi | clarkb: mordred: also, just a bit of trending analysis, even purging logs older than 6 months, what "6-months of logs" means has been steadily increasing at a rate of a terabyte a month since july/august | 21:59 |
| *** dstanek has quit IRC | 22:00 | |
| *** dstanek has joined #openstack-infra | 22:00 | |
| fungi | clarkb: mordred: at this rate, by the end of the year we'll be storing an average of a terabyte of compressed logs a month, and come the icehouse release it'll be more like 2tb a month | 22:01 |
| *** beagles has joined #openstack-infra | 22:03 | |
| clarkb | wow | 22:07 |
| *** dkliban has quit IRC | 22:08 | |
| *** kin is now known as nikhil__ | 22:10 | |
| *** esheffield_ has quit IRC | 22:11 | |
| *** che-arne has joined #openstack-infra | 22:12 | |
| *** Ryan_Lane has joined #openstack-infra | 22:12 | |
| *** UtahDave has joined #openstack-infra | 22:12 | |
| anteaya | wow, as well | 22:14 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove query for bug 1217734 https://review.openstack.org/56473 | 22:17 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1251448 https://review.openstack.org/56474 | 22:17 |
| jog0 | clarkb: quick queries.yaml update ^ | 22:17 |
| jog0 | anteaya: bug 1251448 is a new neutron one | 22:17 |
| *** johnthetubaguy has joined #openstack-infra | 22:17 | |
| anteaya | jog0: thanks | 22:18 |
| openstackgerrit | Peter Liljenberg proposed a change to openstack-infra/jenkins-job-builder: Updated gitblit projectname https://review.openstack.org/56475 | 22:19 |
| anteaya | jog0: 75 hits on it? | 22:21 |
| *** dkliban has joined #openstack-infra | 22:21 | |
| jog0 | anteaya: yeah, I don't know why your thanking me for finding more things for neutron fix ;) | 22:22 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove query for bug 1217734 https://review.openstack.org/56473 | 22:22 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1251448 https://review.openstack.org/56474 | 22:22 |
| jog0 | anteaya: 165 in last week | 22:23 |
| anteaya | jog0: ouch | 22:23 |
| anteaya | well at least you are talking to me about it | 22:23 |
| anteaya | and letting me know | 22:23 |
| jog0 | anteaya: and it got worse recently | 22:23 |
| anteaya | anyway of tracking down why? | 22:23 |
| jog0 | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiIHBvc3NpYmxlIG5ldHdvcmtzIGZvdW5kLCB1c2UgYSBOZXR3b3JrIElEIHRvIGJlIG1vcmUgc3BlY2lmaWMuIChIVFRQIDQwMClcIiBBTkQgZmlsZW5hbWU6XCJjb25zb2xlLmh0bWxcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiYWxsIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4NDQ2Nzg0MzM2MiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ== | 22:24 |
| jog0 | thats the query over all time (2 weeks AFAIK) | 22:24 |
| jog0 | anteaya: not sure about tracking down the cause, I was just trying to identify the bug at all | 22:25 |
| *** senk1 has quit IRC | 22:25 | |
| jog0 | but it did get much worse today | 22:25 |
| anteaya | okay thanks | 22:25 |
| jog0 | perhaps because we fixed other bugs | 22:26 |
| anteaya | anybody else being hit by this or just neutron | 22:26 |
| jog0 | nope | 22:26 |
| anteaya | closing the hole | 22:26 |
| jog0 | so in elasticsearch click on 'build_name' on the left hand side | 22:26 |
| jog0 | that will show you which builds are affected | 22:26 |
| anteaya | okay I will wait until the Monday meeting then, unless someone else brings it up: https://wiki.openstack.org/wiki/Network/Meetings#Neutron_Tempest_.28anteaya.29 | 22:26 |
| jog0 | you can also click on 'build_status' and see this always lines up with a failure | 22:27 |
| jog0 | anteaya: thanks!! | 22:27 |
| anteaya | how handy | 22:27 |
| anteaya | yeah, well if we can close the hole so that the only project affected by neutron bugs is neutron | 22:28 |
| *** loq_mac has joined #openstack-infra | 22:28 | |
| anteaya | I would consider that helpful | 22:28 |
| anteaya | then when they are frustrated enough, they will do something within the group | 22:28 |
| jog0 | anteaya: as long as neutron is in the integrated gate thats impossible AFAIK | 22:28 |
| anteaya | sigh | 22:28 |
| jog0 | anteaya: but we do make thing worse for them at least | 22:29 |
| anteaya | by fixing the bugs | 22:29 |
| anteaya | it is a good direction | 22:29 |
| jog0 | fungi: I hear zuul has an option to run jobs multiple times. Cna we use that in neutron | 22:29 |
| jog0 | just do everything twice for them | 22:29 |
| anteaya | not trying to apply thumbscrews but if all else falls on deaf ears | 22:29 |
| jog0 | or is that a bad idea | 22:29 |
| *** maurosr has quit IRC | 22:31 | |
| *** dstanek has quit IRC | 22:31 | |
| *** maurosr has joined #openstack-infra | 22:32 | |
| clarkb | I don't think zuul has that option | 22:32 |
| clarkb | I think jeblair was suggesting it hsould be added thouhg | 22:32 |
| *** maurosr has quit IRC | 22:32 | |
| *** toddmorey has quit IRC | 22:32 | |
| *** loq_mac has quit IRC | 22:33 | |
| *** ianj1 has left #openstack-infra | 22:34 | |
| *** maurosr has joined #openstack-infra | 22:34 | |
| *** pabelanger has quit IRC | 22:34 | |
| *** loq_mac has joined #openstack-infra | 22:35 | |
| *** flaper87 is now known as flaper87|afk | 22:35 | |
| *** ryanpetrello has quit IRC | 22:37 | |
| jog0 | clarkb: ohh | 22:37 |
| *** mriedem has quit IRC | 22:37 | |
| anteaya | jog0: I'm a little shorter than yourself on the ml | 22:38 |
| anteaya | I salute you | 22:38 |
| anteaya | shorter as in briefer, terse | 22:38 |
| jog0 | WAT http://logs.openstack.org/66/56466/1/check/check-swift-devstack-vm-functional/063a31a/console.html#_2013-11-14_21_18_22_913 | 22:38 |
| *** weshay has quit IRC | 22:38 | |
| jog0 | we try to use pypi.python.org | 22:39 |
| *** vipul is now known as vipul-away | 22:39 | |
| jog0 | I thought we don't trust pypi.python.org to be up | 22:40 |
| jog0 | anteaya: I am going to shame people into stop asking for code reviews on the ML | 22:40 |
| *** johnthetubaguy1 has joined #openstack-infra | 22:41 | |
| *** vipul-away is now known as vipul | 22:41 | |
| *** dstanek has joined #openstack-infra | 22:42 | |
| clarkb | jog0: we don't trust pypi.python.org to be up | 22:42 |
| jog0 | clarkb: see the swift-function test | 22:43 |
| *** dangers is now known as danger_fo_away | 22:44 | |
| jog0 | pip 1.4 install | 22:44 |
| anteaya | w00t! music to my ears: "The policies described here are in effect immediately." http://lists.openstack.org/pipermail/openstack-dev/2013-November/019219.html | 22:45 |
| *** johnthetubaguy has quit IRC | 22:45 | |
| Ryan_Lane | upgrading the wiki for security release | 22:46 |
| jog0 | anteaya: nice | 22:46 |
| *** johnthetubaguy has joined #openstack-infra | 22:47 | |
| anteaya | I wish that one of the requirements of the point of contact was pariticpation in the -neutron irc channel | 22:48 |
| *** johnthetubaguy1 has quit IRC | 22:48 | |
| anteaya | but at least there is a policy | 22:48 |
| *** oubiwann has quit IRC | 22:48 | |
| anteaya | so let's give it some time and see what happens | 22:48 |
| fungi | jog0: mordred: re pypi.p.o, i suspect we want openstack-dev/devstack tools/install_pip.sh to allow us to override PIP_TAR_URL and then pass that through set to our mirror when run devstack-gate-wrap.sh | 22:49 |
| jog0 | R | 22:50 |
| fungi | thanks Ryan_Lane! | 22:50 |
| jog0 | fungi: yeah | 22:50 |
| fungi | anteaya: he did at least say they have to show up in the weekly neutron irc meetings, so that's better than no irc i guess | 22:50 |
| Ryan_Lane | well, I take that back | 22:51 |
| Ryan_Lane | the damn security updates aren't in git yet | 22:51 |
| fungi | jog0: mordred: or maybe have it parse a url out of the usual dotfiles pip looks at? | 22:51 |
| Ryan_Lane | I hate that we do tarball releases and announcements without releasing the patches in git | 22:51 |
| Ryan_Lane | it'll be another hour | 22:51 |
| fungi | Ryan_Lane: we do the opposite ;) | 22:52 |
| *** julim has joined #openstack-infra | 22:52 | |
| Ryan_Lane | yep | 22:52 |
| jog0 | fungi: any reason not to default to pypi.o.o all together? | 22:52 |
| fungi | jog0: stack.sh runs it. we want random people to not get shunted to our mirror | 22:53 |
| fungi | jog0: (see: "devstack is a developer tool first and a ci platform second") | 22:54 |
| jog0 | fungi: ah | 22:54 |
| *** danger_fo_away is now known as dangers | 22:54 | |
| jog0 | fungi: ahh this uses github too | 22:54 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Fix core member count https://review.openstack.org/56421 | 22:54 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Show average reviews per day https://review.openstack.org/56425 | 22:54 |
| fungi | jog0: yeah, several magnitudes of ick there | 22:55 |
| jog0 | well it uses pypi or github | 22:55 |
| anteaya | fungi: yes it is, you are correct | 22:55 |
| anteaya | perhaps he is easing them onto irc | 22:56 |
| * jog0 files an infra bug and will go back to his scheduled programming | 22:56 | |
| fungi | jog0: technically, that's a devstack bug ;) | 22:57 |
| jog0 | fungi: yeah cought that after I said it | 22:57 |
| fungi | or at least we'd need to make changes there to address it | 22:57 |
| *** denis_makogon has quit IRC | 22:59 | |
| *** nati_ueno has joined #openstack-infra | 22:59 | |
| *** johnthetubaguy has quit IRC | 23:00 | |
| *** dcramer_ has quit IRC | 23:00 | |
| jog0 | fungi https://bugs.launchpad.net/openstack-ci/+bug/1251462 | 23:00 |
| fungi | soren: uvirtbot hath forsaken us | 23:01 |
| *** dcramer_ has joined #openstack-infra | 23:01 | |
| *** michchap has quit IRC | 23:02 | |
| *** julim has quit IRC | 23:03 | |
| krtaylor | just curious, how was the contributor list made on http://www.openstack.org/software/havana | 23:03 |
| krtaylor | it looks like it missed contributors with merges after Grizzly freeze, maybe it was started mid April? | 23:04 |
| *** dkranz has quit IRC | 23:04 | |
| devananda | this is an odd looking patchset from jenkins: https://review.openstack.org/#/c/55854/2 | 23:04 |
| *** michchap has joined #openstack-infra | 23:04 | |
| devananda | what's in trunk seems already match the global-requirements file. why is jenkins trying to change it? | 23:05 |
| fungi | devananda: there's been an issue with the requirements proposal job running for requirements stable branch merges, and wanting to propose them to master on the projects | 23:08 |
| devananda | ah | 23:08 |
| devananda | sounds like it | 23:08 |
| devananda | shall i just ignore? | 23:08 |
| fungi | it's not yet branch-aware | 23:08 |
| fungi | yes, just ignore | 23:08 |
| devananda | k | 23:08 |
| jog0 | devananda: don't ignore, yell at mordred instead | 23:09 |
| jog0 | ;) | 23:09 |
| jog0 | and then ignore | 23:09 |
| devananda | mordred: ^^^ :) | 23:09 |
| clarkb | fungi: we should manually abandon all proposed changes and disable the job until it is working | 23:09 |
| *** jamesmcarthur has quit IRC | 23:11 | |
| jog0 | clarkb: ++ | 23:12 |
| jog0 | http://tinyurl.com/m5ogwzf | 23:13 |
| jog0 | proof that neutron is broken | 23:13 |
| jog0 | anteaya: ^ | 23:13 |
| jog0 | neutron fails way more then nova-netwokr | 23:13 |
| *** julim has joined #openstack-infra | 23:13 | |
| * anteaya looks | 23:14 | |
| jog0 | after doign a moving average over 200 data points neutron gate fails 20% of the time | 23:15 |
| jog0 | err 18.33% | 23:15 |
| anteaya | jog0: where is the comparison with nova-netwokr? | 23:15 |
| jog0 | and check fails 28% of time | 23:15 |
| jog0 | *-tempest-devstack-vm-full = nova-network | 23:16 |
| jog0 | *neutron=neutron | 23:16 |
| anteaya | ah | 23:16 |
| jog0 | at what failure rate do we un-vote a project? | 23:16 |
| *** hashar has quit IRC | 23:16 | |
| anteaya | so we are still always testing everything against nova-network? | 23:16 |
| anteaya | jog0: exactly | 23:17 |
| *** paul--_ has joined #openstack-infra | 23:17 | |
| jog0 | anteaya: yes, see results above for why | 23:17 |
| jog0 | and your scary bug list | 23:17 |
| anteaya | jog0: right | 23:17 |
| anteaya | well you can always make a motion to the TC asking for some direction on that question | 23:17 |
| jog0 | sdague: yo uaround? | 23:17 |
| sdague | yes, briefly, what's up? | 23:17 |
| anteaya | when does a project get un-voted | 23:18 |
| jog0 | anteaya: not a bad idea, if a job goes over a certain threshhold we de-integrate until fixed? | 23:18 |
| jog0 | the gate that is* | 23:18 |
| jog0 | sdague: neutron is hurting me (well us) http://tinyurl.com/m5ogwzf | 23:18 |
| *** jlk has quit IRC | 23:18 | |
| sdague | jog0: ok, sure, that's not entirely news | 23:19 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/gerritlib: Add connection attempt limit https://review.openstack.org/52986 | 23:19 |
| jog0 | sdague: so gate-*neutron has a 18% failure rate at the moment (200 point moving average) | 23:19 |
| jog0 | and check has a 28% failure rate | 23:19 |
| anteaya | jog0: I think the TC needs to come to some determination | 23:19 |
| jog0 | sdague: which means its hurting everything and slowing down dev | 23:19 |
| jog0 | sdague: what anteaya said ^ | 23:20 |
| jog0 | sdague: so at what point can we make neutron non-voting? | 23:20 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/gerritlib: Add connection attempt limit https://review.openstack.org/52986 | 23:20 |
| jog0 | sdague: because these numbers are awful | 23:20 |
| sdague | what's our gate queue depth look like? | 23:20 |
| sdague | sure, but they aren't actually that different from normal | 23:21 |
| sdague | you're just spending a lot of time looking at them now :) | 23:21 |
| anteaya | 12 in check, 15 in the gate | 23:21 |
| anteaya | 0 queued events | 23:21 |
| sdague | at a 15 deep gate, at prime PST, I think it's pain we live with | 23:22 |
| anteaya | I do think this kind of decision needs to come from the TC | 23:22 |
| jog0 | sdague: true, I am only now realing *how* bad business as usual is | 23:22 |
| anteaya | yup | 23:22 |
| *** mrodden has quit IRC | 23:22 | |
| jog0 | sdague: it hurts a lot more in the number of rechecks we need in checks | 23:22 |
| anteaya | and they can't make a decision in one meeting | 23:22 |
| sdague | jog0: I think the important thing is to build some long term trending, and focus on the specific races that are the biggest issues | 23:22 |
| jog0 | sdague: thats exactly what we have now | 23:23 |
| sdague | the reality is there is actually more neutron channel activity on working on issues than I've seen in a while | 23:23 |
| jog0 | graphite has long term trends, and elastic-recheck has the bugs | 23:23 |
| sdague | well, what percentage of races does it have? | 23:23 |
| jog0 | sdague: I don't know how to find that yet :( | 23:23 |
| fungi | krtaylor: i think that list came from the same dataset bitergia used to build activity.openstack.org but reed may know more | 23:24 |
| jog0 | I think we need to wire elastic-recheck to emmit graphite data | 23:24 |
| sdague | right, I actually kind of do, but you know, changing jobs means I can't quite get down to it yet | 23:24 |
| jog0 | sdague: eveything falls apart when you change jobs apparently | 23:24 |
| sdague | well... I'll try not to do it that often :) | 23:24 |
| jog0 | :) | 23:24 |
| jog0 | sdague: but the question I want to bring up, which I think anteaya has a good answer to is: | 23:25 |
| reed | fungi, krtaylor: I honestly don't remember anymore where it came from | 23:25 |
| reed | i can dig further or krtaylor just give me the list of people to add | 23:25 |
| jog0 | can we have a policy to make a project non-gating if its gate job fails over x percent of time | 23:25 |
| *** thomasem has quit IRC | 23:25 | |
| jog0 | I guess we don't have great numbers to answer that yet | 23:25 |
| anteaya | I have an item on the neutron meeting agenda for next week: https://wiki.openstack.org/wiki/Network/Meetings#Agenda_for_Next_Neutron_Team_Meeting | 23:26 |
| anteaya | would be great if you both could be there jog0 and sdague | 23:26 |
| sdague | yeh, honestly, I think we need to give folks specifics so they can go after them. So point them at the top 3 reset bugs, and keep tracking those until fixed | 23:26 |
| clarkb | jog0: I don't think we should have that policy | 23:26 |
| jog0 | but what i do know is that 28% of the time you will have to use 'recheck' because of a neutron job failing | 23:26 |
| clarkb | we are a collaborative project people should collaborate | 23:27 |
| sdague | jog0: answer the question of the most important bugs to fix to get the reset race down | 23:27 |
| clarkb | instead of sweeping the pain under the rug we all bear it | 23:27 |
| anteaya | clarkb: are you in -neutron, idling? | 23:27 |
| sdague | agree with clarkb | 23:27 |
| clarkb | anteaya: no, I have tried very hard to not sit in any of those channels | 23:27 |
| jog0 | sdague: I am drafting a list of the most important bugs | 23:27 |
| sdague | jog0: cool | 23:27 |
| clarkb | too distracting and EHARDTONOTBUTTIN | 23:27 |
| jog0 | sdague: I guess I just didn't realize how bad things were | 23:28 |
| sdague | that's an actionable thing, and something to bring into the neutron discussion | 23:28 |
| krtaylor | reed, it should be the same as the people that got a havana contributors badge, right? | 23:28 |
| sdague | jog0: now you understand why my tone is what it was during the session :) | 23:28 |
| anteaya | okay let's just take a peek at this log: http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2013-11-13.log | 23:28 |
| anteaya | timestamp: 2013-11-13T08:18:24 | 23:29 |
| reed | krtaylor, uhm... not sure. The ATC definition is a bit wider than 'people who contributed to the current release' | 23:29 |
| sdague | because I think our path forward is about being specific about what fixes would be the best bang for the buck | 23:29 |
| sdague | and markmcclain already knows they need to rewrite their agent core to handle the races | 23:29 |
| anteaya | clarkb: I think i might need you to butt in for a bit | 23:29 |
| sdague | which he's working on | 23:29 |
| krtaylor | reed, ah, ok, but it should be people that merged a patch that went into the release? | 23:29 |
| clarkb | anteaya: but I explicitly don't want to :) | 23:30 |
| reed | krtaylor, yes, that for sure | 23:30 |
| notmyname | mordred: around? I saw your name on the internet and wanted to ask you about it | 23:30 |
| sdague | so I don't want to go lynch mob on them while they are working towards solutions, that seems un-gentlemenly | 23:30 |
| anteaya | clarkb: I don't blame you | 23:30 |
| reed | notmyname, really? which name was it? | 23:30 |
| notmyname | reed: heh. turns out mordred is listed as an original maintainer of python-eventlet in ubuntu precise. and I want a backport | 23:31 |
| *** dangers is now known as danger_fo_away | 23:31 | |
| jog0 | sdague: true enough, but its at the point where we need to run 'recheck' a lot | 23:32 |
| jog0 | so much that I should just write a 'recheck' babysitter for my own patches | 23:32 |
| sdague | yeh, agreed, but before going crazy, lets get better and quantifying things | 23:33 |
| sdague | in reality the gate stability goes in waves, and people don't go and fix races until we get a rash of bounces | 23:33 |
| sdague | so turning stuff off won't fix things | 23:33 |
| jog0 | sdague: yeah, I think we are getting close to getting good numbers | 23:33 |
| sdague | I agree | 23:33 |
| jog0 | sdague: no but it will mhide the problem from me :) | 23:33 |
| sdague | yeh, but then you wouldn't care about it any more :) | 23:34 |
| sdague | and I like not being alone in caring about it :) | 23:34 |
| *** danger_fo_away is now known as dangers | 23:34 | |
| sdague | anyway, it's time for me to go home and shred this drive. I'll be pretty spotty for being online until end of next week, then back in a normal flow | 23:35 |
| jog0 | sdague: ^_^ | 23:35 |
| anteaya | sdague: congratulations and happy trails | 23:35 |
| anteaya | sdague: it will be email only for me for the last week of Nov and first week of Dec | 23:36 |
| anteaya | so let me know when you know anything about Montreal | 23:36 |
| *** loq_mac has quit IRC | 23:36 | |
| *** adalbas has joined #openstack-infra | 23:36 | |
| *** paul-- has quit IRC | 23:36 | |
| anteaya | time for me to do something else | 23:40 |
| *** ^d has joined #openstack-infra | 23:41 | |
| *** locke105 has joined #openstack-infra | 23:42 | |
| *** paul--_ is now known as paul-- | 23:43 | |
| *** mfer has quit IRC | 23:44 | |
| *** ArxCruz has quit IRC | 23:46 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove query for 1240256 https://review.openstack.org/56484 | 23:46 |
| *** matsuhashi has joined #openstack-infra | 23:47 | |
| krtaylor | reed, I had a few minor merges after Grizzly freeze that went in Havana, but it makes we wonder if others were missed too | 23:48 |
| krtaylor | it may be that the gather date was set after summit | 23:49 |
| reed | krtaylor, it's possible | 23:50 |
| reed | send me an email please with the details to add and i"ll amend the page | 23:50 |
| *** DennyZhang has quit IRC | 23:52 | |
| jog0 | anteaya: http://paste.openstack.org/show/52959 | 23:53 |
| jog0 | top 5 gate bugs | 23:53 |
| *** thomasem has joined #openstack-infra | 23:53 | |
| *** rcleere has quit IRC | 23:55 | |
| *** adalbas has quit IRC | 23:55 | |
| *** dcramer_ has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!