| *** xchu has joined #openstack-infra | 00:00 | |
| *** jcooley_ has quit IRC | 00:02 | |
| *** thomasbiege has quit IRC | 00:02 | |
| *** nicedice has quit IRC | 00:04 | |
| *** dolphm has quit IRC | 00:05 | |
| *** nicedice has joined #openstack-infra | 00:05 | |
| *** jhesketh__ has joined #openstack-infra | 00:05 | |
| *** jcooley_ has joined #openstack-infra | 00:06 | |
| *** jcooley_ has quit IRC | 00:08 | |
| *** jcooley_ has joined #openstack-infra | 00:10 | |
| *** DennyZhang has joined #openstack-infra | 00:14 | |
| *** eharney has quit IRC | 00:15 | |
| *** DennyZhang has quit IRC | 00:15 | |
| *** elasticio has joined #openstack-infra | 00:15 | |
| lifeless | fungi: ^ | 00:16 |
|---|---|---|
| *** DennyZhang has joined #openstack-infra | 00:16 | |
| *** mriedem has joined #openstack-infra | 00:17 | |
| *** rnirmal has quit IRC | 00:19 | |
| *** amotoki has joined #openstack-infra | 00:21 | |
| *** ryanpetrello has joined #openstack-infra | 00:22 | |
| *** ryanpetrello has quit IRC | 00:23 | |
| *** xchu has quit IRC | 00:23 | |
| *** mriedem has quit IRC | 00:23 | |
| *** rfolco has quit IRC | 00:24 | |
| *** dkranz has joined #openstack-infra | 00:24 | |
| *** mriedem has joined #openstack-infra | 00:25 | |
| *** wenlock has quit IRC | 00:27 | |
| *** DennyZhang has quit IRC | 00:27 | |
| *** mutante has joined #openstack-infra | 00:27 | |
| *** hogepodge has quit IRC | 00:27 | |
| mutante | Hi, just wanted to point you to http://burrow.openstack.org/ I ran across it by accident because in our channel, wikimedia, we had a trigger !queue giving you that link, then somebody else figured out what it once was, just letting you know about the "It works" URL of a deprecated thing and #openstack said to do that here for ops attention. maybe worth a redirect because it seems to be out there and "good URLs" bla bla | 00:29 |
| sdague | jog0: http://ubuntuone.com/5zemZEvpdtXPoDNQU87nyr - replication of the 5hr moving average (of success, so inverted) for gate-tempest-dsvm-full in python with pandas off the graphite data | 00:29 |
| lifeless | fungi: / clarkb: if either of you are around; would like to setup the gantt review team | 00:29 |
| sdague | and with that... it's a day, and I'm going to go fend for dinner | 00:29 |
| *** mriedem has quit IRC | 00:29 | |
| lifeless | lines 12 and 13 of https://etherpad.openstack.org/p/icehouse-external-scheduler list the initial core team | 00:31 |
| lifeless | russellb: should we also make it inherit nova-core? | 00:31 |
| *** marun has joined #openstack-infra | 00:31 | |
| *** yamahata has joined #openstack-infra | 00:33 | |
| jog0 | pandas? | 00:33 |
| jog0 | sdague: cool but not even sure waht I am looking at (no x axis label) | 00:35 |
| jog0 | sdague: btw Ilike http://not.mn/gate_status.html | 00:36 |
| jog0 | except it should use moving average | 00:36 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Automatically provide 'name' to project templates https://review.openstack.org/62500 | 00:36 |
| jeblair | lifeless: i'll setup the team | 00:37 |
| *** dims has quit IRC | 00:37 | |
| *** dcramer_ has quit IRC | 00:37 | |
| lifeless | jeblair: thank you! | 00:37 |
| *** esker has quit IRC | 00:38 | |
| *** freyes has joined #openstack-infra | 00:38 | |
| *** esker has joined #openstack-infra | 00:38 | |
| jeblair | lifeless: done except for johnthetubaguy because i don't know his real name | 00:38 |
| jeblair | or gerrit name | 00:38 |
| jeblair | etc | 00:38 |
| jeblair | lifeless: you're on it so you can self-manage now at any rate | 00:39 |
| *** dims has joined #openstack-infra | 00:39 | |
| lifeless | jeblair: ack, thanks | 00:40 |
| *** michchap has quit IRC | 00:42 | |
| *** michchap has joined #openstack-infra | 00:42 | |
| *** freyes has quit IRC | 00:47 | |
| *** dims has quit IRC | 00:47 | |
| fungi | lifeless: back and taking a look now | 00:50 |
| fungi | oh, jeblair seems to have jumped on it | 00:50 |
| fungi | i'll make sure stuff cloned okay | 00:50 |
| *** jcooley_ has quit IRC | 00:50 | |
| fungi | since, as noted, we still have a problem there apparently | 00:50 |
| jeblair | fungi: i have not looked at git repos, thanks for picking that up. | 00:51 |
| lifeless | jeblair: fungi: thanks; | 00:51 |
| lifeless | s/;/./ | 00:51 |
| *** mriedem has joined #openstack-infra | 00:52 | |
| *** jcooley_ has joined #openstack-infra | 00:53 | |
| *** nosnos has joined #openstack-infra | 00:54 | |
| fungi | review and zuul clones/checkouts are okay. git farm managed fine *except* for git01 which has a broken mirror of gantt and python/ganttclient, and git02 which just has broken gantt... fixing | 00:55 |
| lifeless | awesome, thank you! | 00:56 |
| jog0 | jeblair: I realized the phasing of my comments on https://review.openstack.org/#/c/62096/ were sub optimal | 00:56 |
| *** ^d is now known as ^demon|away | 00:56 | |
| jog0 | its I don't only want to know about errors the word error was misleading I think | 00:56 |
| jeblair | jog0: gotcha, the commit msg could have been more explicit in that case... does the expanded explanation make sense? | 00:57 |
| jog0 | jeblair: yes it does | 00:57 |
| *** sarob has joined #openstack-infra | 00:57 | |
| jog0 | so I want a superset | 00:57 |
| jog0 | of what that patch does | 00:58 |
| jog0 | also one thing I am curios about is, how your patch will handle changes in log formats etc | 00:58 |
| jeblair | jog0: it's markov chaining so it's not particular to formats; it's just matching up series of word-tokens | 00:59 |
| jeblair | jog0: so in other words, it should adapt | 00:59 |
| jog0 | jeblair: figured but wonder how that will change the accuracy of this etc | 01:00 |
| jog0 | but one way to find out | 01:01 |
| *** apevec has quit IRC | 01:01 | |
| jog0 | jeblair: as for the logstash vs logs.o.o | 01:01 |
| jog0 | I like logs.o.o because it has all logs and has long term storage | 01:01 |
| jeblair | jog0: yeah, it'll probably take a few runs for it to adapt. and yeah, there's more small-scale testing that could be done, but i think we can go ahead and start seeing what happens when we throw a lot of data at it. | 01:02 |
| jog0 | I like using both | 01:02 |
| jeblair | it's a big experiment. | 01:02 |
| jog0 | jeblair: ++ for just going for it and seeing how well it works | 01:02 |
| *** dims has joined #openstack-infra | 01:02 | |
| jog0 | so long term it would be nice to have some way to get this info into logs.o.o as well as elasticsearch | 01:02 |
| *** mrodden has quit IRC | 01:04 | |
| *** elasticio has quit IRC | 01:05 | |
| *** dcramer_ has joined #openstack-infra | 01:05 | |
| *** nosnos has quit IRC | 01:06 | |
| *** nosnos has joined #openstack-infra | 01:06 | |
| *** sarob has quit IRC | 01:07 | |
| *** mutante has left #openstack-infra | 01:08 | |
| *** sarob has joined #openstack-infra | 01:08 | |
| fungi | it might be possible for os-loganalyze to query some hints out of the result set when viewing the log (though that would require some spooky action at a distance, between logstash workers and logs.o.o) | 01:10 |
| fungi | but i could imagine a particular colorset for highlighting lines which score in particular ranges | 01:11 |
| *** sarob has quit IRC | 01:12 | |
| fungi | arosen: thanks for punching 61437 in. i'm surprised we got it merged as soon as we did | 01:12 |
| arosen | fungi: sure thing | 01:13 |
| arosen | thanks for getting those reviews up :) | 01:13 |
| fungi | though it's still in the gate, so i shouldn't jinx it | 01:13 |
| sdague | jog0: I need to label the series, the point was finally getting to where I could do math on them :) | 01:14 |
| *** freyes has joined #openstack-infra | 01:16 | |
| *** bingbu has joined #openstack-infra | 01:17 | |
| *** sarob has joined #openstack-infra | 01:17 | |
| *** nibalizer has quit IRC | 01:19 | |
| jog0 | sdague: too bad that graph doesn't explain what math that was | 01:22 |
| sdague | yeh, well, I only just got the series stuff working at the end of the day :P | 01:24 |
| sdague | so I decided I could share the raw data stream and declare success, or sit on it. So I decided part 1 | 01:24 |
| sdague | that's the hard part, adding the chrome on the graphs is just time, not learning new things | 01:25 |
| sdague | on the crm114 stuff, error_pr will be our new field? | 01:25 |
| *** bingbu has quit IRC | 01:26 | |
| jog0 | sdague: I like http://not.mn/gate_status.html | 01:27 |
| jog0 | notmyname: you should use moving averge instead | 01:27 |
| sdague | jog0: so I tried for 4 hrs to convert that to moving average this weekend | 01:28 |
| sdague | graphite is not a data analysis tool | 01:28 |
| notmyname | jog0: summarize buckets into what's happening at a certain time interval | 01:28 |
| sdague | so if we actually make it easy to pull the graphite data back into python, where we can do data analysis, then we can make all kinds of pretty graphs | 01:30 |
| notmyname | jog0: I don't care what the average pass rate was over the last day. I care what was the real pass rate for a time block | 01:30 |
| notmyname | sdague: change the format query param. you can get json output | 01:30 |
| *** ^demon|away has quit IRC | 01:30 | |
| sdague | notmyname: yep | 01:30 |
| sdague | actually, that's what I'm doing, getting the counters, pulling them into pandas time series, and then can do manip on those | 01:30 |
| sdague | and put back out to matplotlib when done | 01:31 |
| jog0 | notmyname: moving averge for 1 hour does the same thing but you don't get teh big jumps between hours | 01:32 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Cover the tempest user in select_mirror https://review.openstack.org/61861 | 01:32 |
| notmyname | jog0: heh. are you arguing for 1 hour time blocks now? ;-) | 01:32 |
| jog0 | notmyname: no | 01:32 |
| jog0 | I am arguing moving average for same time interval that you have for smart median or w/e its called | 01:32 |
| notmyname | jog0: moving average version http://not.mn/gate_status2.html | 01:33 |
| notmyname | you'll see that it's the same shape, just noisier | 01:33 |
| jog0 | yeah | 01:33 |
| jog0 | that shape may be the dev cycle | 01:33 |
| notmyname | essentially the same thing you'd see if you have smaller time buckets | 01:33 |
| jog0 | in gate | 01:33 |
| jog0 | because gate is empty at night | 01:33 |
| notmyname | well, actually, you'll see that the gate is only empty at the end of the week | 01:34 |
| jog0 | sdague: are you somehow factoring in amount of data | 01:34 |
| jog0 | for dev cycle | 01:34 |
| sdague | jog0: I'm just getting the data into time series right now | 01:35 |
| sdague | because this morning, I didn't know how to do that. | 01:35 |
| jog0 | you mean the json thing? | 01:35 |
| sdague | in a way that was easy to do series manipulation | 01:35 |
| notmyname | the summarize() method shows what's happening at a certain time bucket (in my case, a 6 hour status for visual clarity over a 2 week span) | 01:36 |
| jog0 | https://review.openstack.org/#/c/61429/ | 01:36 |
| jog0 | notmyname: so right now if the gate is empty it has a 100% pass rate | 01:36 |
| sdague | jog0: I mean once you had the json arrays with timestamps, then what datastructures and math libs do yuo have to be able to do sliding windows and series division | 01:36 |
| *** praneshp has quit IRC | 01:37 | |
| jog0 | which is why we gate a daily cycle for gate rate | 01:37 |
| jog0 | sdague: yeah | 01:37 |
| notmyname | jog0: correct. that's why I show on my graph. the gates only pass when nothing in it | 01:37 |
| notmyname | jog0: and using smaller time buckets is valid to see exactly what's going on, but it just looks worse. in fact, for the average dev, I think using larger buckets (eg 12 hours) is more accurate. and that looks horrible (ie it's never a 100% pass chance) | 01:38 |
| jog0 | notmyname: agreed | 01:38 |
| jog0 | I think we need two graphs | 01:38 |
| jog0 | one that is multi day trends | 01:39 |
| jog0 | another that is much finer grained | 01:39 |
| jog0 | err 4 | 01:39 |
| jog0 | and another two for the check queue | 01:39 |
| notmyname | jog0: http://not.mn/gate_status2.html is now for 12 hour buckets. it looks horrible | 01:39 |
| fungi | i think it's a *bit* of a mischaracterization to suggest that nothing passes when there is something in the gate. graphing the change merged rate would make it apparent that things are actually merging, albeit not everything thrown at the wall sticks on the first attempt | 01:39 |
| jog0 | also notmyname insead of gate depth you can use https://review.openstack.org/#/c/61429/ | 01:39 |
| jog0 | notmyname: it looks accurate | 01:40 |
| notmyname | jog0: I'm using stats.gauges.zuul.pipeline.gate.current_changes | 01:40 |
| notmyname | for the gate queue depth | 01:40 |
| *** yaguang has joined #openstack-infra | 01:40 | |
| jog0 | that is good to see but doesn't reflect how large of a data set you are using | 01:41 |
| fifieldt | fungi, you're too fast :) | 01:41 |
| fungi | fifieldt: i'll slow down | 01:41 |
| *** ljjjustin has joined #openstack-infra | 01:41 | |
| jog0 | sum(stats.zuul.pipeline.gate.job.*{SUCCESS,FAILURE}) will how many datapoints are used to get the stats | 01:42 |
| fifieldt | ^_^ | 01:42 |
| fungi | fifieldt: or maybe i'm just trying to game our peer reviews ;) | 01:42 |
| fifieldt | clearly | 01:42 |
| fifieldt | :) | 01:42 |
| *** bingbu has joined #openstack-infra | 01:42 | |
| notmyname | fungi: correct. obviously some things are passing the gate. I'm looking at my "chance for an otherwise perfectly good patch to pass". that is only 100% when there is nothing in the gate. it /should/ be always 100%. even if we never get there, it should be _very_ high | 01:42 |
| jog0 | notmyname: YES YES YES | 01:43 |
| fungi | notmyname: i couldn't agree more | 01:43 |
| jog0 | notmyname: IMHO this is one of the biggest issues in openstack today | 01:43 |
| fungi | at least with it being *very* high. obviously i don't want to see _everything_ pass because, well, sometimes some people approve some things they oughtn't | 01:44 |
| jhesketh__ | Hi guys | 01:44 |
| fungi | and i'd like for the gate to keep that trash out of master | 01:44 |
| notmyname | and based on the 2 weeks of data we have, it seems that Wed. is the problem day. I think this is because things clear out over the weekend, start to pile up Mon and Tues, and then panic happens on Wed. Then it takes until Fri for the gate to flush out. and I think we're on track again this week | 01:44 |
| fungi | jhesketh__: hi! | 01:44 |
| jhesketh__ | just wondering where in the config the formatting is added to zuul's URL reports | 01:44 |
| jog0 | fungi notmyname: I still want to run gate jobs on trunk when we have free cloud resources | 01:44 |
| jog0 | to get more datasets | 01:44 |
| jog0 | and those jobs should *always* pass | 01:45 |
| jog0 | jhesketh__: o/ | 01:45 |
| jhesketh__ | hey jog0 | 01:45 |
| * jhesketh__ is a bad person and usually hides in shadows but is often here :-) | 01:45 | |
| fungi | jhesketh__: it's in layout.yaml, but some of it depends on gerrit doing magical pattern recognition on comments (so some of that behavior is configured in gerrit itself) | 01:45 |
| sdague | jog0: so those jobs are not going to always pass until all these bugs are squashed - http://status.openstack.org/elastic-recheck/ | 01:46 |
| jog0 | sdague: of course | 01:46 |
| fungi | jhesketh__: we a'splained it to mikal while you were awol, but i'll get you links to the relevant configuration | 01:46 |
| jog0 | I bolded the wrong word | 01:46 |
| jog0 | *should* | 01:46 |
| *** ljjjustin has quit IRC | 01:46 | |
| jog0 | notmyname: http://not.mn/gate_status2.html what does that look like with moving averae for 12 hour | 01:47 |
| jhesketh__ | fungi: yeah, I remember he went looking for it but I don't know why | 01:47 |
| jhesketh__ | I've been hunting in the gerrit config but I'm blind | 01:47 |
| *** melwitt has quit IRC | 01:48 | |
| notmyname | jog0: don't know. I've got to review something internally then bike home. I'm can't work on it any more right now. maybe later tonight | 01:48 |
| *** ljjjustin has joined #openstack-infra | 01:48 | |
| jog0 | notmyname: at 1AM again? | 01:48 |
| jog0 | notmyname: o/ | 01:48 |
| notmyname | jog0: I hope not! | 01:48 |
| notmyname | jog0: but I do have to work on my PTL webinar, so maybe | 01:48 |
| fungi | jhesketh__: so, in the layout.yaml you *can* do fancy things like http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n270 | 01:49 |
| *** oubiwann has joined #openstack-infra | 01:49 | |
| fungi | jhesketh__: as for what gerrit knows how to magically link to places, we have a set of pattern/stream-edits pairs (basically) when get spammed into its configuration from http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/gerrit.pp#n128 | 01:50 |
| jhesketh__ | fungi: yeah that's cool. I was more after making turbo-hipster's reports look nice and neat | 01:50 |
| jhesketh__ | ah that's it | 01:50 |
| jhesketh__ | thanks :-) | 01:50 |
| fungi | er, which... feh i am giving up on accurate typing for the night | 01:51 |
| fungi | the testresult one is probably of most \interest to you | 01:52 |
| jhesketh__ | yep, thanks :-) | 01:52 |
| *** jergerber has quit IRC | 01:53 | |
| *** mrodden has joined #openstack-infra | 01:53 | |
| *** greghaynes has joined #openstack-infra | 01:54 | |
| *** mrodden1 has joined #openstack-infra | 01:55 | |
| *** dkliban has quit IRC | 01:56 | |
| *** mrodden has quit IRC | 01:57 | |
| jhesketh__ | fungi: so it looks like the regex expects the results from zuul to be an html list (<li>). However zuul by default outputs the results with dashes.. am I missing something/somewhere that converts those? | 01:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Process logs with CRM114 https://review.openstack.org/62096 | 02:01 |
| *** nati_ueno has quit IRC | 02:01 | |
| fungi | jhesketh__: does it default to dashes normally? hmmm, yeah i see what you mean at the end of zuul.scheduler.BasePipelineManager.formatReport() | 02:01 |
| jhesketh__ | yeah as far as I can tell | 02:02 |
| jhesketh__ | maybe gerrit already formats dashes into lists? | 02:02 |
| fungi | i'll test the theory | 02:02 |
| *** Hunner has quit IRC | 02:03 | |
| jhesketh__ | :-) | 02:03 |
| *** dolphm has joined #openstack-infra | 02:04 | |
| fungi | aha... if - is the first character on the line, then it seems to do | 02:05 |
| jhesketh__ | ah cool :-) | 02:05 |
| jhesketh__ | thanks for checking | 02:05 |
| fungi | if you precede it with whitespace, then no | 02:05 |
| fungi | so indented bullet lists a la restructuredtext is not an option, but if you don't indent it should prettify automagically | 02:06 |
| jhesketh__ | good to know | 02:06 |
| fungi | doesn't seem to actually display any bullets for me though | 02:06 |
| fungi | but it does seem to format the lines similar to what's in the zuul report comments, so i think that's the main point | 02:07 |
| *** jcooley_ has quit IRC | 02:08 | |
| jhesketh__ | fungi: do you know if the formatting applied on rendering or on save? | 02:14 |
| *** dcramer_ has quit IRC | 02:14 | |
| *** bingbu has quit IRC | 02:19 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Check for whitespace around changeid searches https://review.openstack.org/62523 | 02:25 |
| *** bingbu has joined #openstack-infra | 02:25 | |
| *** dcramer_ has joined #openstack-infra | 02:27 | |
| *** nati_ueno has joined #openstack-infra | 02:30 | |
| fungi | jhesketh__: i'm not sure... i'd have to go digging in the gerrit db if i understand what you're asking | 02:30 |
| jhesketh__ | fungi: nah all good, just wondered if you knew | 02:31 |
| jhesketh__ | fungi: mostly in regards to this change that will help me https://review.openstack.org/62523 | 02:32 |
| *** reed has quit IRC | 02:42 | |
| *** vkozhukalov has joined #openstack-infra | 02:48 | |
| *** SushilKM has joined #openstack-infra | 02:48 | |
| *** slong has joined #openstack-infra | 02:48 | |
| *** freyes has quit IRC | 02:49 | |
| *** dcramer_ has quit IRC | 02:51 | |
| *** SushilKM has quit IRC | 02:52 | |
| *** dims has quit IRC | 02:53 | |
| *** dims has joined #openstack-infra | 02:53 | |
| *** wenlock has joined #openstack-infra | 03:02 | |
| *** SushilKM has joined #openstack-infra | 03:03 | |
| *** thomasem has joined #openstack-infra | 03:06 | |
| *** yamahata__ has quit IRC | 03:06 | |
| *** yamahata__ has joined #openstack-infra | 03:06 | |
| *** oubiwan__ has joined #openstack-infra | 03:09 | |
| *** oubiwann has quit IRC | 03:11 | |
| *** thomasem has quit IRC | 03:12 | |
| *** klrmn has joined #openstack-infra | 03:19 | |
| *** ljjjustin has quit IRC | 03:21 | |
| *** dolphm has quit IRC | 03:25 | |
| *** dolphm has joined #openstack-infra | 03:30 | |
| *** rcleere has joined #openstack-infra | 03:35 | |
| *** rongze has joined #openstack-infra | 03:40 | |
| *** praneshp has joined #openstack-infra | 03:44 | |
| *** slong has quit IRC | 03:44 | |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack-infra/devstack-gate: Copy libvirt log file after tempest run https://review.openstack.org/61892 | 03:48 |
| *** praneshp_ has joined #openstack-infra | 03:49 | |
| *** praneshp has quit IRC | 03:51 | |
| *** praneshp_ is now known as praneshp | 03:51 | |
| *** bingbu has quit IRC | 03:51 | |
| *** mriedem has quit IRC | 03:55 | |
| *** bingbu has joined #openstack-infra | 04:04 | |
| *** dolphm has quit IRC | 04:05 | |
| *** chandankumar has joined #openstack-infra | 04:16 | |
| *** DennyZhang has joined #openstack-infra | 04:18 | |
| *** dkliban has joined #openstack-infra | 04:19 | |
| *** DennyZhang has quit IRC | 04:21 | |
| *** DennyZhang has joined #openstack-infra | 04:22 | |
| *** DennyZhang has quit IRC | 04:23 | |
| *** DennyZhang has joined #openstack-infra | 04:24 | |
| *** zhiyan has joined #openstack-infra | 04:27 | |
| *** DennyZhang has quit IRC | 04:29 | |
| *** rongze has quit IRC | 04:33 | |
| *** dkliban has quit IRC | 04:37 | |
| *** DennyZhang has joined #openstack-infra | 04:38 | |
| *** SushilKM has quit IRC | 04:38 | |
| *** dkliban has joined #openstack-infra | 04:48 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:49 | |
| *** rongze has joined #openstack-infra | 04:55 | |
| notmyname | jeblair: around this evening? I wanted to bounce an idea off of you that may help (or at least mitigate) gate issues | 04:55 |
| *** sarob_ has joined #openstack-infra | 04:56 | |
| *** rcarrillocruz has joined #openstack-infra | 04:57 | |
| *** sarob has quit IRC | 04:58 | |
| *** rongze_ has joined #openstack-infra | 04:59 | |
| *** rongze has quit IRC | 05:00 | |
| *** sarob_ has quit IRC | 05:01 | |
| *** rcarrillocruz has quit IRC | 05:01 | |
| *** rongze_ has quit IRC | 05:04 | |
| *** qs201 has joined #openstack-infra | 05:05 | |
| *** nicedice has quit IRC | 05:06 | |
| *** qs201 has quit IRC | 05:07 | |
| *** DennyZhang has quit IRC | 05:09 | |
| *** senk has joined #openstack-infra | 05:19 | |
| *** wenlock has quit IRC | 05:19 | |
| *** reed has joined #openstack-infra | 05:21 | |
| *** guohliu has joined #openstack-infra | 05:23 | |
| *** sandywalsh_ has quit IRC | 05:23 | |
| *** ryu25 has joined #openstack-infra | 05:25 | |
| notmyname | fungi: jeblair: jog0: explanation and methodology text added to http://not.mn/gate_status.html | 05:26 |
| *** reed has quit IRC | 05:26 | |
| *** pcrews has quit IRC | 05:33 | |
| *** bingbu has quit IRC | 05:36 | |
| *** sandywalsh_ has joined #openstack-infra | 05:39 | |
| *** Ryan_Lane has quit IRC | 05:41 | |
| *** Ryan_Lane has joined #openstack-infra | 05:42 | |
| *** Ryan_Lane has quit IRC | 05:46 | |
| *** senk has quit IRC | 05:48 | |
| *** bingbu has joined #openstack-infra | 05:49 | |
| *** jhesketh__ has quit IRC | 05:50 | |
| *** jhesketh__ has joined #openstack-infra | 05:51 | |
| *** talluri has joined #openstack-infra | 05:51 | |
| *** senk has joined #openstack-infra | 05:54 | |
| *** senk has quit IRC | 05:55 | |
| *** SushilKM has joined #openstack-infra | 06:00 | |
| *** SushilKM__ has joined #openstack-infra | 06:04 | |
| *** SushilKM has quit IRC | 06:04 | |
| *** dkliban has quit IRC | 06:08 | |
| *** adalbas has quit IRC | 06:12 | |
| *** SushilKM__ has quit IRC | 06:14 | |
| *** SushilKM has joined #openstack-infra | 06:14 | |
| *** rongze has joined #openstack-infra | 06:15 | |
| *** DennyZhang has joined #openstack-infra | 06:19 | |
| *** DennyZhang has quit IRC | 06:27 | |
| *** DennyZhang has joined #openstack-infra | 06:28 | |
| *** adalbas has joined #openstack-infra | 06:29 | |
| *** DennyZhang has left #openstack-infra | 06:30 | |
| *** niska has quit IRC | 06:32 | |
| *** niska has joined #openstack-infra | 06:35 | |
| *** boris-42 has joined #openstack-infra | 06:36 | |
| *** zhiyan has quit IRC | 06:39 | |
| *** SushilKM has quit IRC | 06:40 | |
| openstackgerrit | John Dickinson proposed a change to openstack-infra/reviewstats: add swift-bench https://review.openstack.org/62562 | 06:41 |
| *** harlowja is now known as harlowja_away | 06:43 | |
| *** SushilKM has joined #openstack-infra | 06:43 | |
| *** HenryG has quit IRC | 06:43 | |
| *** SergeyLukjanov has quit IRC | 06:45 | |
| *** marun has quit IRC | 06:49 | |
| *** DennyZhang has joined #openstack-infra | 06:51 | |
| *** vkozhukalov has quit IRC | 06:51 | |
| *** AaronGr_Zzz is now known as AaronGr | 06:52 | |
| *** DennyZhang has left #openstack-infra | 06:54 | |
| *** DennyZhang has quit IRC | 06:54 | |
| *** DennyZhang has joined #openstack-infra | 06:56 | |
| *** thomasbiege has joined #openstack-infra | 06:58 | |
| *** bingbu has quit IRC | 07:00 | |
| *** plomakin_ has quit IRC | 07:01 | |
| *** michchap has quit IRC | 07:02 | |
| *** michchap has joined #openstack-infra | 07:03 | |
| *** jcooley_ has joined #openstack-infra | 07:03 | |
| *** denis_makogon has joined #openstack-infra | 07:05 | |
| *** Abhishek_ has joined #openstack-infra | 07:07 | |
| *** jcooley_ has quit IRC | 07:07 | |
| *** jcooley_ has joined #openstack-infra | 07:09 | |
| *** HenryG has joined #openstack-infra | 07:10 | |
| *** bingbu has joined #openstack-infra | 07:12 | |
| *** resker has joined #openstack-infra | 07:13 | |
| *** jcooley_ has quit IRC | 07:14 | |
| *** yolanda has joined #openstack-infra | 07:15 | |
| *** esker has quit IRC | 07:16 | |
| *** zhiyan has joined #openstack-infra | 07:20 | |
| *** jhesketh__ has quit IRC | 07:20 | |
| *** boris-42 has quit IRC | 07:21 | |
| *** fifieldt has quit IRC | 07:22 | |
| *** DennyZhang has quit IRC | 07:22 | |
| *** Abhishek_ has quit IRC | 07:30 | |
| *** tsufiev has joined #openstack-infra | 07:30 | |
| *** AaronGr is now known as AaronGr_Zzz | 07:32 | |
| *** zul has quit IRC | 07:34 | |
| *** vogxn has joined #openstack-infra | 07:34 | |
| *** fifieldt has joined #openstack-infra | 07:35 | |
| *** DennyZhang has joined #openstack-infra | 07:35 | |
| *** afazekas has joined #openstack-infra | 07:37 | |
| *** rcarrillocruz has joined #openstack-infra | 07:39 | |
| *** DennyZhang has quit IRC | 07:40 | |
| *** oubiwan__ has quit IRC | 07:42 | |
| *** talluri_ has joined #openstack-infra | 07:45 | |
| *** flaper87|afk is now known as flaper87 | 07:46 | |
| *** talluri has quit IRC | 07:48 | |
| *** rcarrillocruz1 has joined #openstack-infra | 08:03 | |
| *** rcarrillocruz has quit IRC | 08:05 | |
| ttx | apevec, lifeless, fungi: FWIW that LP error message is classic https://bugs.launchpad.net/launchpad/+bug/1076512 | 08:07 |
| uvirtbot | Launchpad bug 1076512 in launchpad "BugTask:+editstatus timeouts on bugs with many tasks" [Critical,Triaged] | 08:07 |
| ttx | we just use tasks at a scale LP does not like. And it's only getting worse | 08:08 |
| *** amotoki has quit IRC | 08:09 | |
| lifeless | ttx: it's not that, as I keep saying, it's notifications. | 08:09 |
| lifeless | ttx: the problem is a bug, (vs a design problem or an expectation problem) | 08:09 |
| ttx | lifeless: Oh. So it rollbacks the full tasks because it sends too many notifications ? | 08:10 |
| lifeless | the transaction hits the timeout, yes | 08:10 |
| lifeless | I tried to get StevenK's attention earlier to poke at the oops and see if it is indeed the same one.. | 08:11 |
| ttx | well, it's still linked to our usage of tasks. | 08:11 |
| notmyname | ttx: status updates still in #openstack-dev? | 08:11 |
| lifeless | yes | 08:11 |
| ttx | (generate too many notifications) | 08:11 |
| lifeless | but the usage is valid per LP's design | 08:11 |
| ttx | notmyname: we can skip it this week if you don't have anything special to discuss | 08:12 |
| ttx | notmyname: but yes, still in #openstack-dev, haven't had time to set up channel with proper logging yet | 08:12 |
| notmyname | ttx: I've got stuff. a couple of swift things, but the major issue is the gate status (http://not.mn/gate_status.html) | 08:13 |
| ttx | notmyname: ok, i'll be there | 08:13 |
| *** zhiyan_ has joined #openstack-infra | 08:13 | |
| ttx | notmyname: we can do it now if you prefer to stay up late rather than get up early | 08:13 |
| notmyname | ttx: ok, sounds good to me | 08:14 |
| *** zhiyan has quit IRC | 08:15 | |
| *** zhiyan_ is now known as zhiyan | 08:15 | |
| *** pblaho has joined #openstack-infra | 08:16 | |
| *** moted has quit IRC | 08:16 | |
| *** zhiyan has quit IRC | 08:22 | |
| *** vkozhukalov has joined #openstack-infra | 08:26 | |
| *** denis_makogon has quit IRC | 08:26 | |
| *** avishayb has joined #openstack-infra | 08:29 | |
| *** zul has joined #openstack-infra | 08:34 | |
| *** jcoufal has joined #openstack-infra | 08:34 | |
| *** jhesketh_ has quit IRC | 08:34 | |
| *** dizquierdo has joined #openstack-infra | 08:36 | |
| *** DennyZhang has joined #openstack-infra | 08:36 | |
| *** DinaBelova has joined #openstack-infra | 08:37 | |
| *** DennyZhang has quit IRC | 08:41 | |
| *** osanchez has joined #openstack-infra | 08:41 | |
| *** fifieldt has quit IRC | 08:43 | |
| *** HenryG has quit IRC | 08:43 | |
| *** HenryG has joined #openstack-infra | 08:44 | |
| *** elasticio has joined #openstack-infra | 08:44 | |
| *** jhesketh_ has joined #openstack-infra | 08:46 | |
| *** tsufiev has quit IRC | 08:46 | |
| *** fbo_away is now known as fbo | 08:47 | |
| *** talluri has joined #openstack-infra | 08:47 | |
| *** talluri_ has quit IRC | 08:51 | |
| *** thomasbiege has quit IRC | 08:52 | |
| *** thomasbiege has joined #openstack-infra | 08:54 | |
| *** elasticio has quit IRC | 08:57 | |
| StevenK | lifeless: I had a quick glance at it, it's the usual OMG-too-many-subscribers timeout | 08:57 |
| lifeless | StevenK: thanks | 08:58 |
| StevenK | I just got distracted before I actually told you. | 08:58 |
| *** salv-orlando has quit IRC | 09:00 | |
| lifeless | StevenK: lol - thanks! | 09:05 |
| *** praneshp has quit IRC | 09:05 | |
| *** nosnos has quit IRC | 09:07 | |
| *** nosnos has joined #openstack-infra | 09:07 | |
| *** jpich has joined #openstack-infra | 09:07 | |
| *** derekh has joined #openstack-infra | 09:07 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:08 | |
| *** SergeyLukjanov has quit IRC | 09:09 | |
| *** yassine has joined #openstack-infra | 09:09 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:09 | |
| *** Ryan_Lane has joined #openstack-infra | 09:15 | |
| *** jooools has joined #openstack-infra | 09:17 | |
| *** johnthetubaguy has joined #openstack-infra | 09:19 | |
| *** yamahata has quit IRC | 09:19 | |
| *** Ryan_Lane has quit IRC | 09:20 | |
| *** mancdaz_away is now known as mancdaz | 09:21 | |
| *** jooools has quit IRC | 09:29 | |
| *** fifieldt has joined #openstack-infra | 09:31 | |
| *** DennyZhang has joined #openstack-infra | 09:32 | |
| *** vkozhukalov has quit IRC | 09:35 | |
| *** osanchez has quit IRC | 09:37 | |
| *** Abhishek_ has joined #openstack-infra | 09:39 | |
| *** hashar has joined #openstack-infra | 09:39 | |
| *** ryu25 has quit IRC | 09:39 | |
| flaper87 | Anyone that could help me figure out what's happening here? https://review.openstack.org/#/c/32544/ py27 tests run just fine in my laptop. Same code, same test commands etc | 09:41 |
| *** andreaf has quit IRC | 09:43 | |
| *** bingbu has quit IRC | 09:46 | |
| *** masayukig has quit IRC | 09:47 | |
| *** osanchez has joined #openstack-infra | 09:49 | |
| *** vkozhukalov has joined #openstack-infra | 09:50 | |
| *** boris-42 has joined #openstack-infra | 09:52 | |
| *** nati_ueno has quit IRC | 09:53 | |
| *** DennyZhang has quit IRC | 09:54 | |
| *** osanchez has quit IRC | 09:54 | |
| *** nati_ueno has joined #openstack-infra | 09:55 | |
| *** SergeyLukjanov has quit IRC | 09:55 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:57 | |
| *** zhiyan has joined #openstack-infra | 10:05 | |
| *** agordeev has joined #openstack-infra | 10:11 | |
| *** rcleere has quit IRC | 10:15 | |
| *** rcleere has joined #openstack-infra | 10:15 | |
| *** jooools has joined #openstack-infra | 10:28 | |
| *** jcooley_ has joined #openstack-infra | 10:33 | |
| *** jcooley_ has quit IRC | 10:38 | |
| *** vkozhukalov has quit IRC | 10:40 | |
| *** jcooley_ has joined #openstack-infra | 10:43 | |
| *** zul has quit IRC | 10:47 | |
| *** jcooley_ has quit IRC | 10:48 | |
| *** jcooley_ has joined #openstack-infra | 10:48 | |
| *** zul has joined #openstack-infra | 10:48 | |
| *** tma996 has joined #openstack-infra | 10:50 | |
| *** vkozhukalov has joined #openstack-infra | 10:52 | |
| *** jcooley_ has quit IRC | 10:52 | |
| *** jcooley_ has joined #openstack-infra | 10:53 | |
| *** jcooley_ has quit IRC | 10:58 | |
| *** jcooley_ has joined #openstack-infra | 10:58 | |
| *** dizquierdo has quit IRC | 11:00 | |
| *** SergeyLukjanov has quit IRC | 11:01 | |
| *** jcooley_ has quit IRC | 11:03 | |
| *** jcooley_ has joined #openstack-infra | 11:04 | |
| *** guohliu has quit IRC | 11:05 | |
| *** flaper87 is now known as flaper87|afk | 11:07 | |
| *** jcooley_ has quit IRC | 11:08 | |
| *** ruhe has joined #openstack-infra | 11:08 | |
| *** markmc has joined #openstack-infra | 11:09 | |
| *** nati_ueno has quit IRC | 11:10 | |
| *** nati_ueno has joined #openstack-infra | 11:11 | |
| *** DinaBelova has quit IRC | 11:11 | |
| *** nati_uen_ has joined #openstack-infra | 11:12 | |
| *** jcooley_ has joined #openstack-infra | 11:14 | |
| *** nati_ueno has quit IRC | 11:15 | |
| *** marun has joined #openstack-infra | 11:17 | |
| *** jcooley_ has quit IRC | 11:19 | |
| *** jcooley_ has joined #openstack-infra | 11:19 | |
| *** rongze has quit IRC | 11:19 | |
| *** hashar_ has joined #openstack-infra | 11:20 | |
| *** hashar has quit IRC | 11:21 | |
| *** hashar_ is now known as hashar | 11:21 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/62605 | 11:23 |
| *** jcooley_ has quit IRC | 11:23 | |
| *** jcooley_ has joined #openstack-infra | 11:24 | |
| *** DinaBelova has joined #openstack-infra | 11:25 | |
| *** SergeyLukjanov has joined #openstack-infra | 11:26 | |
| *** fifieldt has quit IRC | 11:27 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/62605 | 11:28 |
| *** salv-orlando has joined #openstack-infra | 11:28 | |
| *** jcooley_ has quit IRC | 11:29 | |
| *** jcooley_ has joined #openstack-infra | 11:30 | |
| *** zhiyan has quit IRC | 11:31 | |
| *** lcestari has joined #openstack-infra | 11:33 | |
| *** SergeyLukjanov has quit IRC | 11:34 | |
| *** jcooley_ has quit IRC | 11:34 | |
| *** jcooley_ has joined #openstack-infra | 11:35 | |
| *** rfolco has joined #openstack-infra | 11:36 | |
| *** SergeyLukjanov has joined #openstack-infra | 11:37 | |
| *** jcooley_ has quit IRC | 11:39 | |
| *** jcooley_ has joined #openstack-infra | 11:40 | |
| *** zhiyan has joined #openstack-infra | 11:42 | |
| *** ruhe has quit IRC | 11:42 | |
| *** ruhe has joined #openstack-infra | 11:43 | |
| *** rongze has joined #openstack-infra | 11:44 | |
| *** jcooley_ has quit IRC | 11:45 | |
| *** rongze_ has joined #openstack-infra | 11:45 | |
| *** rcarrillocruz has joined #openstack-infra | 11:45 | |
| *** ArxCruz has joined #openstack-infra | 11:45 | |
| *** jcooley_ has joined #openstack-infra | 11:46 | |
| *** rcarrillocruz1 has quit IRC | 11:47 | |
| *** salv-orlando has quit IRC | 11:47 | |
| *** ruhe has quit IRC | 11:48 | |
| openstackgerrit | Emily Bache proposed a change to openstack-infra/jenkins-job-builder: fix for bug #1260649 https://review.openstack.org/62611 | 11:48 |
| uvirtbot | Launchpad bug 1260649 in openstack-ci "jenkins-job-builder doesn't filter when using template" [Undecided,New] https://launchpad.net/bugs/1260649 | 11:48 |
| *** rongze has quit IRC | 11:49 | |
| *** HenryG has quit IRC | 11:50 | |
| *** jcooley_ has quit IRC | 11:50 | |
| *** jcooley_ has joined #openstack-infra | 11:51 | |
| *** chandankumar has quit IRC | 11:53 | |
| *** chandankumar has joined #openstack-infra | 11:53 | |
| *** hashar has quit IRC | 11:55 | |
| marun | uh, anybody else having issues with restacking due to nova-* services running as daemon and persisting past screen being killed? | 11:55 |
| *** jcooley_ has quit IRC | 11:55 | |
| *** jcooley_ has joined #openstack-infra | 11:56 | |
| *** jcooley_ has quit IRC | 12:01 | |
| *** jcooley_ has joined #openstack-infra | 12:01 | |
| *** CaptTofu has quit IRC | 12:02 | |
| *** CaptTofu has joined #openstack-infra | 12:03 | |
| *** sharwell_ has quit IRC | 12:03 | |
| *** rongze_ has quit IRC | 12:03 | |
| *** jcooley_ has quit IRC | 12:05 | |
| *** salv-orlando has joined #openstack-infra | 12:10 | |
| *** salv-orlando has quit IRC | 12:10 | |
| *** jcooley_ has joined #openstack-infra | 12:11 | |
| *** fifieldt has joined #openstack-infra | 12:13 | |
| *** CaptTofu has quit IRC | 12:14 | |
| *** CaptTofu has joined #openstack-infra | 12:14 | |
| *** jcooley_ has quit IRC | 12:15 | |
| *** dstanek has quit IRC | 12:16 | |
| *** jcooley_ has joined #openstack-infra | 12:16 | |
| *** akscram has quit IRC | 12:17 | |
| *** Abhishek_ has quit IRC | 12:17 | |
| *** jcooley_ has quit IRC | 12:21 | |
| *** jcooley_ has joined #openstack-infra | 12:21 | |
| *** jcooley_ has quit IRC | 12:25 | |
| *** jcooley_ has joined #openstack-infra | 12:26 | |
| *** dolphm has joined #openstack-infra | 12:29 | |
| *** jcooley_ has quit IRC | 12:30 | |
| *** jcooley_ has joined #openstack-infra | 12:31 | |
| *** yaguang has quit IRC | 12:33 | |
| *** rongze has joined #openstack-infra | 12:34 | |
| *** yolanda has quit IRC | 12:35 | |
| *** yolanda has joined #openstack-infra | 12:36 | |
| *** jcooley_ has quit IRC | 12:36 | |
| *** jcooley_ has joined #openstack-infra | 12:37 | |
| *** jcooley_ has quit IRC | 12:37 | |
| *** jcooley_ has joined #openstack-infra | 12:38 | |
| *** hashar has joined #openstack-infra | 12:40 | |
| *** rongze has quit IRC | 12:42 | |
| *** jcooley_ has quit IRC | 12:42 | |
| *** jcooley_ has joined #openstack-infra | 12:43 | |
| *** jcooley_ has quit IRC | 12:43 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add bug for cinder lvremove stray error https://review.openstack.org/62628 | 12:46 |
| *** jcooley_ has joined #openstack-infra | 12:49 | |
| *** vogxn has joined #openstack-infra | 12:50 | |
| *** nosnos has quit IRC | 12:51 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1261554 https://review.openstack.org/62487 | 12:54 |
| uvirtbot | Launchpad bug 1261554 in python-keystoneclient "Keystoneclient revocation list is not multi-thread/process safe" [Undecided,New] https://launchpad.net/bugs/1261554 | 12:54 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add bug for cinder lvremove stray error https://review.openstack.org/62628 | 12:54 |
| *** jcooley_ has quit IRC | 12:54 | |
| *** andreaf has joined #openstack-infra | 12:54 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Cleanup and clarify README https://review.openstack.org/62214 | 12:54 |
| *** jcooley_ has joined #openstack-infra | 12:54 | |
| *** SergeyLukjanov has quit IRC | 12:56 | |
| *** dizquierdo has joined #openstack-infra | 12:56 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: don't only match on nova logs https://review.openstack.org/62632 | 12:57 |
| *** SergeyLukjanov has joined #openstack-infra | 12:58 | |
| *** jcooley_ has quit IRC | 12:59 | |
| *** jcooley_ has joined #openstack-infra | 13:00 | |
| *** afazekas has quit IRC | 13:02 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: don't only match on nova logs https://review.openstack.org/62632 | 13:03 |
| *** jcooley_ has quit IRC | 13:04 | |
| *** jcooley_ has joined #openstack-infra | 13:04 | |
| *** yassine has quit IRC | 13:06 | |
| *** nati_uen_ has quit IRC | 13:06 | |
| *** yassine has joined #openstack-infra | 13:06 | |
| *** jcooley_ has quit IRC | 13:09 | |
| *** jcooley_ has joined #openstack-infra | 13:10 | |
| *** flaper87|afk is now known as flaper87 | 13:13 | |
| *** jcooley_ has quit IRC | 13:14 | |
| *** jcooley_ has joined #openstack-infra | 13:14 | |
| *** ruhe has joined #openstack-infra | 13:16 | |
| *** yamahata has joined #openstack-infra | 13:17 | |
| *** rongze has joined #openstack-infra | 13:18 | |
| *** dolphm has quit IRC | 13:18 | |
| *** jcooley_ has quit IRC | 13:19 | |
| *** jcooley_ has joined #openstack-infra | 13:19 | |
| *** zhiyan has quit IRC | 13:21 | |
| *** thomasem has joined #openstack-infra | 13:21 | |
| *** afazekas has joined #openstack-infra | 13:22 | |
| *** dolphm has joined #openstack-infra | 13:23 | |
| *** ilyashakhat has joined #openstack-infra | 13:23 | |
| *** resker has quit IRC | 13:24 | |
| *** weshay has joined #openstack-infra | 13:24 | |
| *** jcooley_ has quit IRC | 13:24 | |
| *** ilyashakhat_ has quit IRC | 13:24 | |
| *** jcooley_ has joined #openstack-infra | 13:26 | |
| *** dolphm has quit IRC | 13:27 | |
| *** sandywalsh_ has quit IRC | 13:29 | |
| *** ruhe is now known as ruhe_ | 13:29 | |
| *** jcooley_ has quit IRC | 13:30 | |
| *** ruhe_ is now known as ruhe | 13:31 | |
| *** jcooley_ has joined #openstack-infra | 13:32 | |
| *** fifieldt has quit IRC | 13:33 | |
| *** jcooley_ has quit IRC | 13:36 | |
| *** boris-42 has quit IRC | 13:37 | |
| *** jcooley_ has joined #openstack-infra | 13:37 | |
| *** ruhe is now known as ruhe_ | 13:38 | |
| *** rongze_ has joined #openstack-infra | 13:39 | |
| *** ruhe_ is now known as ruhe | 13:41 | |
| *** sandywalsh_ has joined #openstack-infra | 13:42 | |
| openstackgerrit | Chuck Short proposed a change to openstack/requirements: Bump webob requirements to include 1.3 https://review.openstack.org/61742 | 13:42 |
| *** jcooley_ has quit IRC | 13:42 | |
| *** jcooley_ has joined #openstack-infra | 13:43 | |
| *** rongze has quit IRC | 13:43 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Alter zookeeper params for Fedora. https://review.openstack.org/62645 | 13:46 |
| *** rongze_ has quit IRC | 13:48 | |
| *** jcooley_ has quit IRC | 13:48 | |
| *** jcooley_ has joined #openstack-infra | 13:48 | |
| *** rongze has joined #openstack-infra | 13:48 | |
| *** rongze_ has joined #openstack-infra | 13:49 | |
| *** dolphm has joined #openstack-infra | 13:49 | |
| *** dolphm has quit IRC | 13:51 | |
| *** dolphm_ has joined #openstack-infra | 13:51 | |
| *** afazekas has quit IRC | 13:51 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: allow facets to work at different resolutions https://review.openstack.org/62652 | 13:52 |
| *** jcooley_ has quit IRC | 13:52 | |
| *** rongze has quit IRC | 13:53 | |
| *** dprince has joined #openstack-infra | 13:54 | |
| *** jcooley_ has joined #openstack-infra | 13:54 | |
| *** jcooley_ has quit IRC | 13:54 | |
| *** thomasbiege has quit IRC | 13:56 | |
| *** jcooley_ has joined #openstack-infra | 13:56 | |
| *** rfolco has quit IRC | 13:56 | |
| *** thomasbiege has joined #openstack-infra | 13:56 | |
| *** pblaho has quit IRC | 13:57 | |
| *** dolphm_ has quit IRC | 13:58 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Initial test set added https://review.openstack.org/61120 | 13:59 |
| *** HenryG has joined #openstack-infra | 14:00 | |
| *** jcooley_ has quit IRC | 14:01 | |
| *** jcooley_ has joined #openstack-infra | 14:01 | |
| *** yaguang has joined #openstack-infra | 14:03 | |
| *** dolphm has joined #openstack-infra | 14:03 | |
| *** jcooley_ has quit IRC | 14:06 | |
| *** jcooley_ has joined #openstack-infra | 14:07 | |
| *** heyongli has joined #openstack-infra | 14:07 | |
| *** dolphm has quit IRC | 14:08 | |
| *** rfolco has joined #openstack-infra | 14:08 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: allow facets to work at different resolutions https://review.openstack.org/62652 | 14:09 |
| *** jcooley_ has quit IRC | 14:12 | |
| *** jcooley_ has joined #openstack-infra | 14:13 | |
| *** akscram has joined #openstack-infra | 14:13 | |
| *** oubiwann has joined #openstack-infra | 14:14 | |
| *** eharney has joined #openstack-infra | 14:15 | |
| *** jcooley_ has quit IRC | 14:17 | |
| *** vogxn has quit IRC | 14:18 | |
| *** jcooley_ has joined #openstack-infra | 14:18 | |
| *** jcooley_ has quit IRC | 14:18 | |
| *** jcooley_ has joined #openstack-infra | 14:20 | |
| fungi | marun: at least in tests, we don't reuse devstack nodes so we don't bother to unstack (we just delete them and grab fresh ones). if you're interested in a clean unstack, maybe adding a step in devstack-gate scripts after tempest/grenade runs to do that and then use something like grenade's check-sanity step to confirm the worrysome processes are no longer running | 14:21 |
| marun | fungi: gotcha. just wanted to make sure I wasn't doing something dumb before I filed a bug. | 14:22 |
| *** afazekas has joined #openstack-infra | 14:23 | |
| *** jcooley_ has quit IRC | 14:24 | |
| *** dolphm has joined #openstack-infra | 14:25 | |
| fungi | marun: i have heard several people mention that in recent months unstack no longer stops everything, for what it's worth | 14:25 |
| *** jcooley_ has joined #openstack-infra | 14:25 | |
| *** vogxn has joined #openstack-infra | 14:25 | |
| marun | fungi: ah, someone has filed it since I started seeing the problem: https://bugs.launchpad.net/devstack/+bug/1254795 | 14:26 |
| uvirtbot | Launchpad bug 1254795 in devstack "nova-api is not alway killed with clean.sh and unstack.sh" [Medium,Confirmed] | 14:26 |
| marun | fungi: I'm going to add your suggestion to the bug so that the assignee can pursue it | 14:26 |
| openstackgerrit | Darragh O'Reilly proposed a change to openstack-infra/os-loganalyze: Provide a script that starts a simple WSGI server https://review.openstack.org/60325 | 14:26 |
| *** yamahata__ has quit IRC | 14:27 | |
| *** yamahata__ has joined #openstack-infra | 14:28 | |
| *** esker has joined #openstack-infra | 14:28 | |
| *** jcooley_ has quit IRC | 14:29 | |
| *** jcooley_ has joined #openstack-infra | 14:30 | |
| fungi | marun: well, my suggestwas was merely to test that it works, once it's been fixed, so that it doesn't creep back in later | 14:30 |
| fungi | er, suggestion was | 14:31 |
| *** andreaf has quit IRC | 14:31 | |
| *** julim has joined #openstack-infra | 14:31 | |
| *** ruhe has quit IRC | 14:31 | |
| marun | fungi: I understood, but it's still a good suggestion to prevent regression | 14:31 |
| *** yamahata has quit IRC | 14:32 | |
| *** markmcclain has quit IRC | 14:33 | |
| *** yamahata has joined #openstack-infra | 14:34 | |
| *** jcooley_ has quit IRC | 14:34 | |
| *** mriedem has joined #openstack-infra | 14:34 | |
| *** zhiyan has joined #openstack-infra | 14:35 | |
| *** davidhadas has joined #openstack-infra | 14:39 | |
| zaro | fungi: morning | 14:39 |
| *** jcooley_ has joined #openstack-infra | 14:39 | |
| fungi | morning zaro! | 14:41 |
| zaro | fungi: i just noticed a few git pull requests for jjb hanging around. https://github.com/openstack-infra/jenkins-job-builder/pulls | 14:41 |
| fungi | you're early... the sun's barely in the sky over there | 14:41 |
| fungi | zaro: yup, i started looking into it but haven't had time to figure out where it broke this time. i'll link a paste of the traceback | 14:42 |
| zaro | tired yesterday, so went to sleep early. now up early. | 14:42 |
| fungi | heh | 14:42 |
| zaro | need any help with that? | 14:42 |
| *** talluri has quit IRC | 14:43 | |
| *** talluri has joined #openstack-infra | 14:43 | |
| fungi | sure! i'm still unwinding my stack of things which are broken, and this is one i simply haven't had time to look at yet http://paste.openstack.org/show/55147/ | 14:44 |
| *** jcooley_ has quit IRC | 14:44 | |
| *** ruhe has joined #openstack-infra | 14:44 | |
| fungi | the orgs_dict is built at http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/close_pull_requests.py#n77 and for some reason seems to be incomplete | 14:45 |
| *** jcooley_ has joined #openstack-infra | 14:45 | |
| *** heyongli has quit IRC | 14:45 | |
| *** esker has quit IRC | 14:46 | |
| fungi | not sure when it started breaking, so not too easy to bisect unless we introduced an error directly into that script itself (which doesn't get changes very often) | 14:46 |
| *** esker has joined #openstack-infra | 14:46 | |
| *** talluri has quit IRC | 14:48 | |
| *** yolanda has quit IRC | 14:48 | |
| fungi | last change to it seems to have been me fixing another issue https://review.openstack.org/51905 | 14:48 |
| *** mtreinish has quit IRC | 14:48 | |
| *** yolanda has joined #openstack-infra | 14:49 | |
| *** jcooley_ has quit IRC | 14:49 | |
| fungi | and review.o.o has that fix (pip freeze reports a jeepyb build consistent with the tip of master) | 14:49 |
| fungi | always possible they decided to change their api again though | 14:50 |
| *** jcooley_ has joined #openstack-infra | 14:51 | |
| *** heyongli has joined #openstack-infra | 14:51 | |
| *** mtreinish has joined #openstack-infra | 14:51 | |
| *** mfer has joined #openstack-infra | 14:54 | |
| zaro | was there a recent change to projects.yaml ? looks like that's what the python script is reading to get the projects. | 14:54 |
| *** jcooley_ has quit IRC | 14:55 | |
| *** jcooley_ has joined #openstack-infra | 14:55 | |
| fungi | that's where i'm leaning as well... there are always tons of changes to projects.yaml so maybe something's being mis-parsed | 14:56 |
| fungi | might be a good idea to just make sure it's all well-formed yaml for starters | 14:57 |
| *** david-lyle_ has joined #openstack-infra | 14:57 | |
| fungi | zaro: but the exception suggests orgs_dict doesn't contain an openstack-dev key, and that's built not from projects.yaml but from a github api call... orgs = ghub.get_user().get_orgs() | 14:58 |
| fungi | so more likely something has happened to the get_orgs() method causing it to start returning an empty set (or at least a set missing openstack-dev) | 14:59 |
| *** chandankumar has quit IRC | 14:59 | |
| fungi | which could mean either a change in the api which the github python module hasn't kept up with, or a change in the module which we were unaware of | 15:00 |
| *** vogxn has quit IRC | 15:00 | |
| *** jcooley_ has quit IRC | 15:00 | |
| *** markmcclain has joined #openstack-infra | 15:01 | |
| notmyname | https://jenkins02.openstack.org/job/gate-swift-python26/2205/ seems to be looping again. | 15:01 |
| *** mriedem has quit IRC | 15:01 | |
| *** Ryan_Lane has joined #openstack-infra | 15:01 | |
| fungi | notmyname: yep. someone rechecked it | 15:01 |
| *** mriedem has joined #openstack-infra | 15:02 | |
| fungi | current thought there is that something's weird about that job causing jenkins not to return a job status when it eventually kills it for the 60-minute timeout, so zuul thinks something unusual went wrong with the job and restarts it | 15:02 |
| notmyname | fungi: should I leave a comment on it asking for no more rechecks on it? | 15:02 |
| *** SushilKM has quit IRC | 15:02 | |
| *** jcooley_ has joined #openstack-infra | 15:02 | |
| *** _cyril_ has joined #openstack-infra | 15:03 | |
| fungi | notmyname: if you like | 15:03 |
| notmyname | I asked ttx to add gate status issues to today's openstack meeting (re http://not.mn/gate_status.html) | 15:03 |
| fungi | notmyname: i'm just going to abort it in jenkins when i get a moment | 15:03 |
| * _cyril_ would like to know whether libtidy is installed on the Jenkins slaves | 15:03 | |
| notmyname | fungi: thanks | 15:03 |
| zaro | fungi: i gotta be AFK for a few hours now. but can continue to look into it if you don't have time. just let me know. | 15:03 |
| fungi | zaro: okay, thanks@ | 15:03 |
| *** ryanpetrello has joined #openstack-infra | 15:03 | |
| fungi | _cyril_: i'll look | 15:04 |
| *** davidhadas has quit IRC | 15:05 | |
| *** davidhadas has joined #openstack-infra | 15:08 | |
| *** david-lyle_ has quit IRC | 15:08 | |
| fungi | _cyril_: on our ubuntu precise slaves, we seem to have no libtidy-0.99-0 installed, but 20091223cvs-1ubuntu2 debs are available in main | 15:08 |
| *** david-lyle_ has joined #openstack-infra | 15:08 | |
| *** senk has joined #openstack-infra | 15:09 | |
| fungi | is that where it's going to be needed? | 15:09 |
| _cyril_ | fungi: jd__ recently posted a patch that adds libtidy-0.99-0 on the slaves, so I'm not sure why they are not installed | 15:09 |
| fungi | _cyril_: have a link to the review? | 15:10 |
| _cyril_ | fungi: https://review.openstack.org/#/c/60872/ | 15:11 |
| fungi | _cyril_: that's what i get for reviewing too slowly. the change is wrong (he's transposed the red hat and debian package names) | 15:12 |
| *** esker has quit IRC | 15:12 | |
| *** dkliban has joined #openstack-infra | 15:13 | |
| fungi | and this is also going to break any further puppet configuration getting applied to any of our unit test slaves, so i should go ahead and propose a fix | 15:13 |
| fungi | in other news, precise40.slave has gone missing. i'm going to force-reboot it through the rs dashboard | 15:14 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/zuul: Update the url for the zuul git repo https://review.openstack.org/62676 | 15:15 |
| _cyril_ | jd__: ^ | 15:15 |
| *** esker has joined #openstack-infra | 15:16 | |
| jd__ | fungi: facepalm | 15:16 |
| fungi | jd__: no worries. we need to do a better job of confirming package names on platforms when we review changes which add new packages | 15:17 |
| jd__ | definitely :( | 15:17 |
| *** wenlock has joined #openstack-infra | 15:20 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/zuul: Update the url for the zuul git repo https://review.openstack.org/62676 | 15:21 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: add swift-bench https://review.openstack.org/62562 | 15:22 |
| *** heyongli has quit IRC | 15:23 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Change libtidy in Jenkins::Params to match Fedora and Ubuntu packages https://review.openstack.org/62681 | 15:23 |
| *** wenlock has quit IRC | 15:24 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct distro package names for libtidy https://review.openstack.org/62682 | 15:25 |
| *** yamahata has quit IRC | 15:26 | |
| fungi | _cyril_: jd__: ^ | 15:26 |
| jd__ | thanks fungi | 15:26 |
| fungi | no problem. i checked via yum and apt-cache to make sure those match the intended systems | 15:27 |
| fungi | ArxCruz: we should probably coordinate. you and i seem to be pushing nearly identical reviews | 15:29 |
| ArxCruz | fungi: oh :) | 15:29 |
| ArxCruz | fungi: didn't knew that | 15:29 |
| *** dcramer_ has joined #openstack-infra | 15:29 | |
| ArxCruz | fungi: would you like me to abandon mine? | 15:29 |
| *** DinaBelova has quit IRC | 15:30 | |
| *** dcramer_ has quit IRC | 15:30 | |
| fungi | ArxCruz: probably easiest to abandon the libtidy one, though you may have something to contribute on the zookeeper one (all i did was remove the package from red hat) | 15:31 |
| *** dcramer_ has joined #openstack-infra | 15:31 | |
| fungi | your patch makes it sound like fedora (which we're not using for the moment) has zookeeper rpms | 15:31 |
| fungi | not that it solves the challenge of testing zookeeper on a python 2.6 interpreter unfortunately | 15:32 |
| *** wenlock has joined #openstack-infra | 15:33 | |
| *** rnirmal has joined #openstack-infra | 15:34 | |
| *** oubiwann has quit IRC | 15:34 | |
| *** rpodolyaka has joined #openstack-infra | 15:35 | |
| fungi | i'm going to self-approve https://review.openstack.org/62411 and https://review.openstack.org/62682 so we get puppet running again (and then check to see whether there's anything else broken) | 15:35 |
| *** nicedice has joined #openstack-infra | 15:36 | |
| *** sarob has joined #openstack-infra | 15:40 | |
| *** sarob has quit IRC | 15:40 | |
| *** sarob has joined #openstack-infra | 15:40 | |
| *** luisg has quit IRC | 15:40 | |
| *** reed has joined #openstack-infra | 15:40 | |
| *** dcramer_ has quit IRC | 15:41 | |
| *** DinaBelova has joined #openstack-infra | 15:42 | |
| *** sarob has quit IRC | 15:43 | |
| *** CaptTofu has quit IRC | 15:44 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Alter zookeeper params for Fedora. https://review.openstack.org/62645 | 15:44 |
| *** jergerber has joined #openstack-infra | 15:45 | |
| *** sarob has joined #openstack-infra | 15:45 | |
| *** alcabrera has joined #openstack-infra | 15:46 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Always dump errors to console https://review.openstack.org/61850 | 15:46 |
| *** alcabrera has left #openstack-infra | 15:46 | |
| *** jcoufal has quit IRC | 15:47 | |
| *** sarob has quit IRC | 15:48 | |
| *** sarob_ has joined #openstack-infra | 15:48 | |
| *** SergeyLukjanov has quit IRC | 15:48 | |
| *** DinaBelova has quit IRC | 15:48 | |
| *** esker has quit IRC | 15:49 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add support for classification rate https://review.openstack.org/62694 | 15:49 |
| *** luisg has joined #openstack-infra | 15:49 | |
| *** esker has joined #openstack-infra | 15:50 | |
| *** vkozhukalov has quit IRC | 15:51 | |
| *** sarob_ has quit IRC | 15:53 | |
| *** salv-orlando has joined #openstack-infra | 15:53 | |
| jeblair | hashar: can you take a look at https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul+branch:master+topic:templates,n,z when you have a minute? | 15:53 |
| *** alexpilotti has joined #openstack-infra | 15:54 | |
| jeblair | hashar: i want to make sure you're okay with those two changes, and that they won't mess your configuration up (i don't think they should, unless you use a variable called 'name' for the second one) | 15:54 |
| *** mriedem has quit IRC | 15:55 | |
| hashar | jeblair: yup noticed them this morning, got to figure out a way to tests them for my use cases :) | 15:57 |
| hashar | definitely on my todo list for today / tomorrow | 15:57 |
| *** dolphm is now known as dolphm_afk | 15:57 | |
| *** johnthetubaguy1 has joined #openstack-infra | 15:57 | |
| *** johnthetubaguy has quit IRC | 15:58 | |
| *** boris-42 has joined #openstack-infra | 15:59 | |
| *** senk has quit IRC | 15:59 | |
| *** ^d has joined #openstack-infra | 16:00 | |
| *** bnemec_ is now known as bnemec | 16:02 | |
| *** eharney has quit IRC | 16:03 | |
| *** eharney has joined #openstack-infra | 16:03 | |
| openstackgerrit | A change was merged to openstack-infra/config: No zookeeper packages on CentOS slaves https://review.openstack.org/62411 | 16:03 |
| openstackgerrit | A change was merged to openstack-infra/config: Correct distro package names for libtidy https://review.openstack.org/62682 | 16:03 |
| *** Guest53164 is now known as antonym | 16:12 | |
| *** ryanpetrello has quit IRC | 16:12 | |
| *** tma996 has quit IRC | 16:12 | |
| ArxCruz | fungi: jeblair do you know someone from vmware working on CI ? I see some results from vmware ci team, and I wondering if they can share how they are skipping some tests in tempest (if it's hardcoded or something else) | 16:13 |
| *** rnirmal has quit IRC | 16:13 | |
| *** ryanpetrello has joined #openstack-infra | 16:14 | |
| *** hcc has joined #openstack-infra | 16:14 | |
| *** DennyZhang has joined #openstack-infra | 16:14 | |
| *** rnirmal has joined #openstack-infra | 16:14 | |
| *** HenryG has quit IRC | 16:14 | |
| boris-42 | fungi hi | 16:14 |
| *** HenryG has joined #openstack-infra | 16:14 | |
| boris-42 | fungi could you help with +2 https://review.openstack.org/#/c/61215/ ?) | 16:15 |
| dims | ArxCruz, hop onto #openstack-vmware and ask tjones | 16:17 |
| *** hcc has quit IRC | 16:18 | |
| fungi | ArxCruz: all i know is that they requested a service account in gerrit for their minesweeper thing. i think the request came through the infra ml | 16:18 |
| dims | or hartsocks | 16:18 |
| russellb | hrm, seems this didn't get populated, though I see it in infra-config ... https://review.openstack.org/#/admin/projects/openstack/gantt,access | 16:18 |
| ArxCruz | dims: thanks, none of them are online now :( | 16:19 |
| ArxCruz | fungi: okay, thanks | 16:19 |
| dims | ArxCruz, y, they are on the US west coast | 16:19 |
| ArxCruz | :/ | 16:19 |
| *** dizquierdo has quit IRC | 16:20 | |
| russellb | fungi: ^ gerrit permissions issue | 16:20 |
| russellb | not sure what to poke at | 16:21 |
| *** mattray has joined #openstack-infra | 16:21 | |
| *** SushilKM has joined #openstack-infra | 16:21 | |
| *** thomasbiege1 has joined #openstack-infra | 16:23 | |
| *** DennyZhang has quit IRC | 16:24 | |
| *** DennyZhang has joined #openstack-infra | 16:24 | |
| *** thomasbiege1 has quit IRC | 16:25 | |
| sdague | fungi: https://jenkins01.openstack.org/job/gate-tempest-dsvm-large-ops/5395/ we failed to use clean nodes again? | 16:25 |
| *** markmcclain has quit IRC | 16:26 | |
| *** pcrews has joined #openstack-infra | 16:26 | |
| *** thomasbiege has quit IRC | 16:26 | |
| fungi | sdague: does searching logstash for that slavename return more than one job run? | 16:26 |
| fungi | russellb: looking | 16:27 |
| jeblair | sdague: or possibly https://bugs.launchpad.net/keystone/+bug/1253482 | 16:27 |
| uvirtbot | Launchpad bug 1253482 in keystone "Keystone default port in linux local ephemeral port range. Devstack should shift range." [Undecided,In progress] | 16:27 |
| sdague | fungi: yes http://logstash.openstack.org/#eyJzZWFyY2giOiJkZXZzdGFjay1wcmVjaXNlLWhwY2xvdWQtYXoyLTg4MDI0NCIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg3Mjk3NjY3OTI4fQ== | 16:28 |
| sdague | jeblair: so that bug, is really wrongly categorized | 16:28 |
| *** thomasbiege has joined #openstack-infra | 16:28 | |
| sdague | because all those failures in the gate have been devstack node reuse | 16:28 |
| jeblair | sdague: i'm sorry i don't understand what you're saying. are you saying 1253482 is not a bug? | 16:29 |
| jeblair | sdague: what failures in the gate are you talking about? | 16:29 |
| sdague | jeblair: I'm saying every time we hit 1253482 in the gate the root cause was the fact that keystone was already running | 16:30 |
| *** SushilKM has quit IRC | 16:30 | |
| *** jcooley_ has quit IRC | 16:30 | |
| jeblair | sdague: then 1253482 wasn't being hit -- 1253482 is about keystone not starting because its port is not available | 16:31 |
| sdague | jeblair: right | 16:31 |
| fungi | russellb: fixed. it looks like manage-projects may still have a race in the geritt group creation api call | 16:31 |
| sdague | we should probably capture a netstat and ps at the end of the runs as well to debug these kinds of things | 16:31 |
| *** zhiyan has quit IRC | 16:32 | |
| sdague | jeblair: that's why I added comment #14 on that bug | 16:32 |
| russellb | fungi: great, thank you! | 16:32 |
| sdague | because basically ER assigned everything to that bug, but the real issue was nodepool was reusing nodes | 16:33 |
| *** DennyZhang has quit IRC | 16:33 | |
| sdague | so it would be good to know if we had a ci bug for it | 16:33 |
| fungi | russellb: lifeless: should gantt-milestone start as the two of you and include nova-milestone? | 16:33 |
| jeblair | sdague: oh, so the e-r query is bad. i thought you said the _bug_ was wrong. | 16:33 |
| *** DennyZhang has joined #openstack-infra | 16:33 | |
| russellb | fungi: yeah guess so | 16:33 |
| jeblair | sdague: i don't think we need a ci bug at this point, that was like a week ago. | 16:33 |
| jeblair | sdague: if it happens again, we should open a bug. | 16:34 |
| sdague | jeblair: well, that's where this started | 16:34 |
| sdague | because we have another fail, and I wanted to know if the same thign was happening | 16:34 |
| *** dolphm_afk has quit IRC | 16:35 | |
| *** CaptTofu has joined #openstack-infra | 16:35 | |
| *** herndon has joined #openstack-infra | 16:35 | |
| sdague | also... is er backed up? | 16:36 |
| *** alcabrera has joined #openstack-infra | 16:36 | |
| sdague | it looks like we are at least 5hrs behind now | 16:36 |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Alter zookeeper params for Fedora. https://review.openstack.org/62645 | 16:36 |
| *** rcleere has quit IRC | 16:37 | |
| *** alcabrera has left #openstack-infra | 16:37 | |
| *** CaptTofu has quit IRC | 16:37 | |
| sdague | I suspect the bayesian classifier might be too much effort to do on the pipeline | 16:37 |
| *** CaptTofu has joined #openstack-infra | 16:37 | |
| *** SushilKM has joined #openstack-infra | 16:38 | |
| *** CaptTofu has quit IRC | 16:38 | |
| sdague | http://logstash.openstack.org/index.html#eyJzZWFyY2giOiIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjQzMjAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4NzI5ODI5NDU4MH0= | 16:38 |
| *** CaptTofu has joined #openstack-infra | 16:38 | |
| sdague | there is a pretty distinct flat lining there | 16:39 |
| jeblair | that merged? | 16:40 |
| jeblair | sdague: the run you cited was only used once. | 16:41 |
| *** adalbas has quit IRC | 16:41 | |
| *** mattray has left #openstack-infra | 16:41 | |
| sdague | jeblair: ok, so we should probably add netstat and ps at the end of the logs on fails just to see? | 16:41 |
| jeblair | sdague: the node you cited in that run was used only once. | 16:41 |
| sdague | jeblair: ok, cool | 16:41 |
| sdague | jeblair: https://review.openstack.org/#/c/62096/ | 16:42 |
| sdague | yes it merged | 16:42 |
| jeblair | that change needs someone babysitting it :( | 16:43 |
| sdague | and since last night we are now backed up at least 5hr delay on elastic search | 16:43 |
| sdague | so I'd actually say we should revert that | 16:43 |
| yolanda | hi, can i have some feedback on this bug? https://bugs.launchpad.net/openstack-ci/+bug/1247866 | 16:43 |
| uvirtbot | Launchpad bug 1247866 in openstack-ci "jjb fails on creating job with spaces in the name" [Medium,Triaged] | 16:43 |
| *** mrodden1 has quit IRC | 16:44 | |
| yolanda | i found that is problem of python-jenkins not encoding url sent, but maybe it's easier to limit chars in name | 16:44 |
| jeblair | fungi: have you looked into the performance of the crm114 change? | 16:44 |
| yolanda | if not, i should send some patch to python-jenkins | 16:44 |
| sdague | because unless we're indexing within 15 minutes, elastic recheck is basically not useful | 16:44 |
| jeblair | sdague: yes we should fix or revert it. | 16:45 |
| fungi | jeblair: not yet--this was the first i knew it was slowing down the logstash loading pipe | 16:45 |
| jeblair | fungi: okay, i'll look into it | 16:46 |
| fungi | sorry about that--clark and i'd both +2'd it and i hoped having it run overnight would get us a good body of data | 16:48 |
| *** miqui has quit IRC | 16:49 | |
| zaro | yolanda: i have added a comment. | 16:49 |
| sdague | is there a way to out of band process it? | 16:49 |
| sdague | or are documents idempotent? | 16:49 |
| fungi | sdague: i'm guessing teeing that stream and queuing the analysis separately is one possible solution | 16:49 |
| yolanda | ok, gread | 16:50 |
| yolanda | great | 16:50 |
| jeblair | fungi: it's like a 10x performance hit | 16:50 |
| fungi | eek | 16:50 |
| sdague | fungi: you wouldn't even need to do that, you could go fetch the docs process, annotate them | 16:50 |
| sdague | completely out of bounds | 16:50 |
| jeblair | i thought we chatted about it; i know i spoke with clarkb about it | 16:51 |
| sdague | then process as much data as yuo had cpu for | 16:51 |
| fungi | jeblair: i thought i'd paid attention to the discussions, but missed it needing babysitting | 16:51 |
| fungi | my fault | 16:51 |
| jeblair | sdague: regardless of when it's processed, we need to keep up. so making an process that's oob of the already oob process won't help; it just needs more hardware thrown at it. | 16:52 |
| clarkb | is it affecting things poorly? | 16:52 |
| *** yaguang has quit IRC | 16:53 | |
| jeblair | clarkb: is the logstash queue length recorded anywhere? | 16:53 |
| clarkb | jeblair: gearman | 16:53 |
| jeblair | a graph of that would really help | 16:53 |
| *** rongze_ has quit IRC | 16:53 | |
| jeblair | clarkb: i mean historic data | 16:53 |
| jeblair | push-log7595867 | 16:53 |
| *** rongze has joined #openstack-infra | 16:53 | |
| clarkb | jeblair: no we don't have historic data | 16:53 |
| jeblair | that's the current queue | 16:53 |
| *** adalbas has joined #openstack-infra | 16:54 | |
| sdague | jeblair: well, while it's experimental, we could oob process to figure out if it was giving us any useful data | 16:54 |
| sdague | as it only makes sense to throw a ton of cpu at it if we feel that it will | 16:54 |
| *** SergeyLukjanov has joined #openstack-infra | 16:54 | |
| clarkb | one thing I think we should try is use performance flavors for the worker VMs | 16:55 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Disable the crm114 filter https://review.openstack.org/62710 | 16:55 |
| sdague | clarkb: sure, though if this is a 10x hit, I don't think it will help that much | 16:55 |
| sdague | because we were seeing about 1.5 improvement factor on tempest runs | 16:55 |
| sdague | so I'd be surprised if we'd close on order of magnitude | 16:55 |
| clarkb | sdague: right so performance flavor + more machines | 16:55 |
| sdague | sure | 16:56 |
| jeblair | it's a cloud, that's what it's there for. | 16:56 |
| sdague | but it would be good to fast path that revert so e-r can catch up | 16:56 |
| sdague | jeblair: sure :) | 16:56 |
| jeblair | clarkb, fungi: https://review.openstack.org/62710 | 16:56 |
| fungi | lgtm | 16:56 |
| clarkb | right, get it working now, engineer the fixes for other thing not right this second | 16:57 |
| *** esker has quit IRC | 16:57 | |
| jeblair | clarkb: logstash-worker3 is not registered with gearman | 16:58 |
| fungi | presumably we could grow the worker cluster onto performance flavor nodes and then retire the older ones in a fairly hitless fashion? | 16:58 |
| jeblair | fungi: i believe so | 16:58 |
| clarkb | jeblair: :( | 16:58 |
| clarkb | fungi: jeblair: correct | 16:58 |
| sdague | jeblair: which classifier did you turn on? | 16:59 |
| jeblair | sdague: ? | 16:59 |
| *** rcleere has joined #openstack-infra | 16:59 | |
| *** afazekas has quit IRC | 16:59 | |
| jog0 | notmyname: I saw the explanation very awesome. there was one issue I noticed | 16:59 |
| sdague | jeblair: http://crm114.sourceforge.net/docs/classify_details.txt | 17:00 |
| jog0 | notmyname: with NFFI scheduling a patch can fail in the gate and then get merged. if you have head->a->b->c | 17:00 |
| jog0 | notmyname: c fails first, but b fails, so c failure is invalidated, and gate tries head->a->c | 17:00 |
| jeblair | sdague: https://review.openstack.org/#/c/62096/3/modules/openstack_project/files/logstash/classify-log.crm | 17:01 |
| jeblair | clarkb: logs end abruptly on nov 21 on worker3 | 17:01 |
| jeblair | clarkb: i'll try restarting | 17:01 |
| clarkb | jeblair: ok | 17:02 |
| sdague | jeblair: so I assume that means the osb markov, which is supposed to be the most performant | 17:02 |
| sdague | sorry, all this is new to me | 17:02 |
| jeblair | sdague: yes, and space efficient | 17:02 |
| yolanda | zaro, actually i found this is changed in python-jenkins version in LP: https://code.launchpad.net/~bigkevmcd/python-jenkins/fix-bug-1000799-urlquote/+merge/106206 | 17:02 |
| *** mrodden has joined #openstack-infra | 17:02 | |
| jeblair | clarkb: restarted it, seems to be running | 17:03 |
| jeblair | (had to manually remove pidfile) | 17:04 |
| yolanda | i'll do some tests using latest python-jenkins version | 17:04 |
| *** hashar has quit IRC | 17:05 | |
| *** katyafervent has joined #openstack-infra | 17:06 | |
| fungi | clarkb: unrelated, but i think our group creation race is still there. got "ERROR:manage_projects:Unable to get UUID for group gantt-milestone." on one run, then success on a second run | 17:06 |
| *** DennyZhang has quit IRC | 17:07 | |
| *** Abhishek_ has joined #openstack-infra | 17:07 | |
| clarkb | fungi: that makes me so sad | 17:07 |
| *** davidhadas has quit IRC | 17:09 | |
| *** reed has quit IRC | 17:09 | |
| jeblair | clarkb: how hard would it be to run 2 workers per node? | 17:10 |
| ArxCruz | fungi: http://paste.openstack.org/show/55170/ | 17:10 |
| ArxCruz | so, what's better here? change mysql-devel to community-mysql-devel or use mariadb ? | 17:10 |
| clarkb | jeblair: not too hard, we would just have the logstash processes listen on different ports | 17:10 |
| clarkb | and match up the gearman workers to those ports | 17:10 |
| clarkb | jeblair: I hadn't done that previously because the single pipeline per host uses a good chunk of the 2 cpus available | 17:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Disable the crm114 filter https://review.openstack.org/62710 | 17:11 |
| jeblair | clarkb: it looks like currently (with crm114) 1.5 out of 2 cpus are used | 17:12 |
| jeblair | clarkb: with crm114, the gearman workers are the bottleneck, not logstash; what about one logstash worker and 2 gearman workers per node? | 17:12 |
| *** flaper87 is now known as flaper87|afk | 17:13 | |
| *** markmc has quit IRC | 17:13 | |
| fungi | ArxCruz: fedora specific, or is this also affecting rhel/centos 6? | 17:13 |
| fungi | ArxCruz: probably use community-mysql-devel on newer fedora, i'd guess (though mordred is in favor of switching our mysql testing to drizzle for a variety of reasons) | 17:14 |
| jeblair | clarkb: if we switch to 4g performance nodes, we'll have 4 vcpus | 17:15 |
| clarkb | jeblair: 4gb performance nodes sound perfect | 17:16 |
| jeblair | clarkb: maybe we should make the logstash worker threaded and have it pull a configurable number of jobs to run simultaneously | 17:16 |
| jeblair | clarkb: yeah, but we won't be using them | 17:16 |
| clarkb | jeblair: why wouldn't we use them? | 17:16 |
| jeblair | clarkb: because the pipeline is currently single threaded | 17:16 |
| clarkb | jeblair: two pipelines + performance node is what I am thinking | 17:16 |
| *** thomasbiege has quit IRC | 17:17 | |
| jeblair | clarkb: yes, that would be better. so then, what are your thoughts on logstash workers? should we keep the 1:1 logstash:gearman ratio, or do 1:2, 1:3, 1:4? | 17:18 |
| pleia2 | clarkb: traffic is still a bit bad, but heading out soon | 17:18 |
| jeblair | clarkb: i lean toward increasing the # of gearman workers, because we're not trying to scale up the logstash processing | 17:18 |
| jeblair | clarkb: we're trying to get data to logstash as fast as we were before | 17:18 |
| clarkb | jeblair: I think that makes sense | 17:18 |
| clarkb | jeblair: logstash's input is smart enough to treat two different tcp connections and different sources of info | 17:19 |
| jeblair | clarkb: cool, so as a first pass, i'll see if we can colocate multiple gearman workers on a node | 17:19 |
| clarkb | jeblair: sounds good | 17:20 |
| clarkb | pleia2: see you when you get here. people are trickling in | 17:20 |
| *** gyee has joined #openstack-infra | 17:20 | |
| *** ruhe has quit IRC | 17:21 | |
| *** ruhe has joined #openstack-infra | 17:22 | |
| ArxCruz | fungi: community-mysql-devel then :) | 17:23 |
| jeblair | sdague: crm114 is off and the queue is falling | 17:24 |
| *** klrmn has left #openstack-infra | 17:24 | |
| sdague | jeblair: cool | 17:25 |
| jog0 | what has to be done to get http://docs.openstack.org/developer/elastic-recheck working | 17:26 |
| *** jswarren has joined #openstack-infra | 17:27 | |
| *** talluri has joined #openstack-infra | 17:28 | |
| sdague | there is a clarkb change for that | 17:31 |
| sdague | https://review.openstack.org/#/c/61300/ | 17:32 |
| *** andreaf has joined #openstack-infra | 17:32 | |
| jog0 | sdague: ahh that answers my question it will be under docs.o.o/infra | 17:32 |
| clarkb | jog0: it will actually be docs.o.o/developer/elastic-recheck | 17:33 |
| jog0 | clarkb: jeblair disagrees | 17:34 |
| clarkb | oh /me looks | 17:34 |
| clarkb | oh he wants it in a different place. thats fine just need to update change | 17:34 |
| *** gothicmindfood has joined #openstack-infra | 17:34 | |
| anteaya | gothicmindfood: hey there | 17:37 |
| *** ruhe has quit IRC | 17:37 | |
| anteaya | gothicmindfood: good to see you | 17:37 |
| fungi | i see the new blood are being indoctrinated into the benefits of staying on irc ;) | 17:37 |
| gothicmindfood | hey there anteaya. | 17:37 |
| gothicmindfood | fungi: I think the benefits of caffeine still slightly outweigh the benefits of IRC, but I'm getting there. | 17:38 |
| fungi | the one makes the other possible | 17:38 |
| *** rongze has quit IRC | 17:39 | |
| * gothicmindfood is still waking up, possibly... | 17:40 | |
| *** rongze has joined #openstack-infra | 17:40 | |
| *** praneshp has joined #openstack-infra | 17:40 | |
| jeblair | sdague: empty queue eta ~2 hours | 17:41 |
| *** mancdaz is now known as mancdaz_away | 17:42 | |
| *** dstanek has joined #openstack-infra | 17:42 | |
| *** mancdaz_away is now known as mancdaz | 17:42 | |
| jog0 | jeblair: am I seeing this right: gate is almost emtpy? | 17:42 |
| *** jcooley_ has joined #openstack-infra | 17:43 | |
| *** vkozhukalov has joined #openstack-infra | 17:44 | |
| zaro | fungi: clarkb i'm testing the scp log copy thing. do we have a remote server i can test with? or do i need to setup my own? | 17:44 |
| fungi | zaro: you could just try to have the server scp to itself on 127.0.0.1 | 17:45 |
| clarkb | zaro: just setup your own | 17:45 |
| clarkb | or do what fungi says | 17:45 |
| *** atiwari has joined #openstack-infra | 17:45 | |
| fungi | (in other words, you already *have* set up your own) | 17:45 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Add Python3 deprecated assert* to HACKING.rst https://review.openstack.org/62136 | 17:45 |
| zaro | ohh yeah, good idea. | 17:45 |
| fungi | no point in letting a good multi-user system go to waste after all. unixunixunix | 17:45 |
| *** reed has joined #openstack-infra | 17:49 | |
| *** sarob has joined #openstack-infra | 17:49 | |
| *** simonmcc has quit IRC | 17:51 | |
| *** sarob has quit IRC | 17:51 | |
| *** jcooley_ has quit IRC | 17:51 | |
| *** sarob has joined #openstack-infra | 17:51 | |
| *** mancdaz is now known as mancdaz_away | 17:51 | |
| *** jcooley_ has joined #openstack-infra | 17:51 | |
| *** zhiyan has joined #openstack-infra | 17:52 | |
| sdague | jog0: yes | 17:52 |
| jog0 | I am shocked its empty | 17:53 |
| *** rongze has quit IRC | 17:53 | |
| portante | perhaps no one wants to run the gauntlet? | 17:53 |
| portante | ;) | 17:54 |
| *** rongze has joined #openstack-infra | 17:54 | |
| *** gothicmindfood has quit IRC | 17:54 | |
| *** simonmcc has joined #openstack-infra | 17:55 | |
| *** markmcclain has joined #openstack-infra | 17:55 | |
| *** sarob has quit IRC | 17:55 | |
| *** jcooley_ has quit IRC | 17:56 | |
| sdague | jog0: if you have some cycles, we definitely need to land fingerprints / bugs on things that -2ed in the gate | 17:56 |
| *** nati_ueno has joined #openstack-infra | 17:56 | |
| *** simonmcc has joined #openstack-infra | 17:57 | |
| *** simonmcc has quit IRC | 17:57 | |
| *** simonmcc has joined #openstack-infra | 17:58 | |
| *** derekh has quit IRC | 17:58 | |
| *** dolphm_afk has joined #openstack-infra | 17:59 | |
| *** simonmcc has joined #openstack-infra | 17:59 | |
| *** dolphm_afk is now known as dolphm | 18:00 | |
| mgagne | zaro: ping | 18:01 |
| zaro | mgagne: yo! | 18:02 |
| *** nati_uen_ has joined #openstack-infra | 18:02 | |
| mgagne | zaro: FYI, I think I found why maven project tests are failing in JJB: https://review.openstack.org/#/c/61518/ | 18:02 |
| *** thomasbiege has joined #openstack-infra | 18:02 | |
| jog0 | sdague: how about powow in 30 minutes? | 18:03 |
| *** nati_uen_ has quit IRC | 18:04 | |
| zaro | mgagne: cool. that helps. | 18:04 |
| *** nati_uen_ has joined #openstack-infra | 18:04 | |
| *** nati_uen_ has quit IRC | 18:05 | |
| mgagne | zaro: didn't have time to think about how it could be refactor though | 18:05 |
| sdague | jog0: I'm sort of flat out today, tomorrow yuo got time? | 18:05 |
| *** nati_ueno has quit IRC | 18:05 | |
| *** nati_uen_ has joined #openstack-infra | 18:05 | |
| jog0 | sdague: sure, I can also run with some stuff on my own today if you want | 18:06 |
| zaro | mgagne: don't have time to look into it today either :( | 18:06 |
| mgagne | zaro: alright, just wanted to let you know ;) | 18:06 |
| *** weshay has quit IRC | 18:06 | |
| sdague | honestly, the classifications are pretty high on the list | 18:07 |
| zaro | mgagne: it looks like just need to handle another project besides freeesytle. | 18:07 |
| *** jcooley_ has joined #openstack-infra | 18:08 | |
| *** praneshp has quit IRC | 18:08 | |
| *** gothicmindfood has joined #openstack-infra | 18:11 | |
| zaro | mgagne: thanks. your review helps. | 18:11 |
| *** davidhadas has joined #openstack-infra | 18:13 | |
| *** jcooley_ has quit IRC | 18:13 | |
| fungi | notmyname: i finally managed to stop that looping py26 unit test job... i had to catch it quickly enough to cancel the job before it progressed far enough to reach the "hanging" test_leak_1 test | 18:15 |
| fungi | once it got there, manually stopping the job resulted in the same slave agent communication issues which would prevent it from returning a status | 18:16 |
| *** jkyle_ is now known as jkyle | 18:16 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Add cassandra-driver dependency https://review.openstack.org/62726 | 18:16 |
| *** jkyle is now known as Guest75169 | 18:16 | |
| *** dhellmann is now known as dhellmann_ | 18:17 | |
| mtreinish | fungi: I just backported the devstack change for the lxml package deps: https://review.openstack.org/#/c/62725/ and https://review.openstack.org/#/c/62727/ | 18:18 |
| *** rongze has quit IRC | 18:18 | |
| *** Guest75169 has quit IRC | 18:18 | |
| fungi | mtreinish: did you coordinate with sdague on that? i know he mentioned something about having a work tree to add tempest deps to devstack | 18:18 |
| * fungi looks | 18:18 | |
| mtreinish | fungi: that's what the backport is | 18:18 |
| fungi | i see! | 18:19 |
| *** weshay has joined #openstack-infra | 18:19 | |
| fungi | mtreinish: it's got my +1 but of course i can't progress on the tox.ini patch for havana until that merges to grizzly devstack | 18:20 |
| *** rfolco has quit IRC | 18:20 | |
| *** rfolco has joined #openstack-infra | 18:21 | |
| *** harlowja_away is now known as harlowja | 18:22 | |
| *** praneshp has joined #openstack-infra | 18:23 | |
| *** davidhadas_ has joined #openstack-infra | 18:24 | |
| *** thomasbiege has quit IRC | 18:25 | |
| *** vipul is now known as vipul-away | 18:26 | |
| sdague | fungi: I just +Aed the backports | 18:27 |
| fungi | sdague: thanks much! | 18:27 |
| fungi | hopefully that's the last hurdle for the tox consistency change | 18:28 |
| *** davidhadas has quit IRC | 18:28 | |
| *** blamar has quit IRC | 18:29 | |
| *** UtahDave has joined #openstack-infra | 18:30 | |
| *** vkozhukalov has quit IRC | 18:36 | |
| *** vkozhukalov has joined #openstack-infra | 18:37 | |
| *** Abhishek_ has quit IRC | 18:38 | |
| *** blamar has joined #openstack-infra | 18:39 | |
| *** alexpilotti has quit IRC | 18:42 | |
| *** jcoufal has joined #openstack-infra | 18:42 | |
| *** oubiwann has joined #openstack-infra | 18:43 | |
| sdague | jeblair: how would I determine if a job is part of the integrated gate queue outside of zuul? | 18:44 |
| sdague | or is there a way we can tag er with that metadata? | 18:44 |
| *** sarob has joined #openstack-infra | 18:44 | |
| *** jcooley_ has joined #openstack-infra | 18:44 | |
| *** jcooley_ has quit IRC | 18:44 | |
| fungi | sdague: if it shares at least one exact job name with any other project in that same queue | 18:45 |
| *** jcooley_ has joined #openstack-infra | 18:45 | |
| sdague | fungi: ok, so how do I do that from the data sitting in er? | 18:45 |
| clarkb | sdague: you can query the zuul json, it is organized by pipeline with a list of projects | 18:45 |
| fungi | sdague: so parsing layout.yaml should be sufficient to build the graph to determine whether or not that's the case | 18:45 |
| sdague | fungi: that would tell me if it was true now, but not about yesterday | 18:45 |
| fungi | well, the zuul status.json is a great option, but he said outside of zuul (so i assumed he meant data at rest) | 18:46 |
| *** jcooley_ has quit IRC | 18:46 | |
| fungi | sdague: that file is managed in revision control | 18:46 |
| sdague | fungi: sure | 18:46 |
| *** jcooley_ has joined #openstack-infra | 18:46 | |
| sdague | ok, so here's the problem | 18:46 |
| *** vipul-away is now known as vipul | 18:46 | |
| fungi | so modulo puppet propagation and zuul service reload time, you could tell from git | 18:46 |
| sdague | when going back through es data I want to know if a build_uuid / build_name pair was part of the integrated gate | 18:47 |
| sdague | and I'd rather not have to write a git engine as part of that | 18:47 |
| fungi | as mentioned earlier, the queue name is exceedingly long, but maybe there's some way of submitting that with the log artifacts as they're processed... | 18:48 |
| jeblair | yeah, we could add the queue name as a job parameter which would mean it could be added to es | 18:48 |
| fungi | not sure what ways we can inject metadata along with them | 18:48 |
| jeblair | i think our principal issue is what to name the queues | 18:48 |
| sdague | so is the name exceedlingly long bug relatively stable? | 18:48 |
| fungi | aha, parameters work for that? awesome | 18:48 |
| fungi | sdague: the name of a queue is the list of all the projects in the queue (rather than a uuid or something) | 18:49 |
| sdague | ok | 18:49 |
| fungi | so it changes as projects are added/removed | 18:49 |
| sdague | we have some way to squash that down to a rolling idea of what "the integrated gate" is? | 18:49 |
| jeblair | sdague: it is currently: "openstack-dev/devstack, openstack-dev/grenade, openstack-dev/pbr, openstack-infra/devstack-gate, openstack-infra/pypi-mirror, openstack/ceilometer, openstack/cinder, openstack/gantt, openstack/glance, openstack/heat, openstack/horizon, openstack/keystone, openstack/neutron, openstack/nova, openstack/oslo.config, openstack/oslo.messaging, openstack/oslo.version, openstack/python-ceilometerclient,... | 18:49 |
| jeblair | sdague: "...openstack/python-cinderclient, openstack/python-ganttclient, openstack/python-glanceclient, openstack/python-heatclient, openstack/python-keystoneclient, openstack/python-neutronclient, openstack/python-novaclient, openstack/python-swiftclient, openstack/requirements, openstack/swift, openstack/tempest, z/tempest" | 18:50 |
| sdague | honestly, I basically want a one byte field on er records that is "is integrated gate" | 18:50 |
| fungi | keeping in mind that over time, shared queues may combine or divide, so stateless names for them may be hard to allocate and track | 18:50 |
| sdague | however I could get that | 18:50 |
| *** johnthetubaguy1 has quit IRC | 18:50 | |
| jeblair | sdague: i do not have an idea of how to do that; i'd love a suggestion. if we found one, we could solve your problem and make more meaningful graphite stats as jog0 was suggesting. | 18:50 |
| *** jcooley_ has quit IRC | 18:50 | |
| *** jcooley_ has joined #openstack-infra | 18:51 | |
| jeblair | we could pick a project and tag it with a label; and whatever queue it ends up with gets that label | 18:51 |
| jeblair | so we could attach the label to tempest | 18:51 |
| jeblair | and queues without labels are named after their projects as now | 18:51 |
| sdague | so if we did that would it be a correct approximation? | 18:52 |
| sdague | correct ish | 18:52 |
| *** freyes has joined #openstack-infra | 18:52 | |
| jeblair | sdague: it would be exactly correct as long as tempest participated in the integrated gate, which i think will hold us for a while | 18:53 |
| *** johnthetubaguy has joined #openstack-infra | 18:53 | |
| sdague | re.search("(^openstack/|devstack|grenade)", result.project) is my currently crappy approximation | 18:53 |
| fungi | alternatively, i guess if we just pass the queue name in a job parameter, substring matching on whether or not openstack/tempest is part of it could also work | 18:53 |
| *** nati_ueno has joined #openstack-infra | 18:53 | |
| jeblair | fungi: that's true | 18:54 |
| fungi | again, so long as that project remains part of the integrated queue | 18:54 |
| fungi | but i do like the labels idea. might help us clean up the status page a little too | 18:54 |
| jeblair | fungi: might solve sdague's need but doesn't help us put metrics for queues in graphite | 18:54 |
| sdague | http://paste.openstack.org/show/55185/ | 18:54 |
| fungi | oh, right, i keep forgetting about the graphite metric name problem | 18:55 |
| *** jcooley_ has quit IRC | 18:55 | |
| sdague | that's using the approximation for the gate that I have | 18:55 |
| sdague | to figure out if a build was part of the gate | 18:56 |
| *** nati_uen_ has quit IRC | 18:57 | |
| sdague | mtreinish: we're discussing exactly your point in https://review.openstack.org/#/c/62694/ here :) | 18:57 |
| *** SushilKM__ has joined #openstack-infra | 18:58 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add per job percentages for responsibility of fails https://review.openstack.org/62738 | 18:58 |
| *** SushilKM has quit IRC | 18:58 | |
| mtreinish | sdague: I guess that means I should start paying attention | 18:58 |
| jeblair | sdague: i can hack on this tomorrow | 18:58 |
| sdague | mtreinish: no, it's a really good point | 18:58 |
| sdague | jeblair: awesome | 18:58 |
| jeblair | shouldn't be hard | 18:58 |
| sdague | my bad approximation will limp us through for now | 18:59 |
| sdague | better would be good | 18:59 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Always dump errors to console https://review.openstack.org/61850 | 18:59 |
| *** vipul is now known as vipul-away | 19:01 | |
| jeblair | infra meeting time in #openstack-meeting | 19:02 |
| sdague | mtreinish: you might appreciate that pastebin | 19:02 |
| sdague | it has per the percentage of failures of a particular job that a bug was responsible for | 19:02 |
| reed | fungi, I saw you committed a new patch to https://review.openstack.org/#/c/61250/ Ready for approval? | 19:02 |
| fungi | reed: i hope--didn't want to go self-approving it without final feedback, but can try to drive up the review urgency there | 19:03 |
| *** thomasm_ has joined #openstack-infra | 19:03 | |
| reed | fungi, ship it :) | 19:04 |
| *** jcooley_ has joined #openstack-infra | 19:04 | |
| mtreinish | sdague: very cool | 19:05 |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Change mysql-devel to community-mysql-devel in Fedora https://review.openstack.org/62739 | 19:05 |
| *** salv-orlando has quit IRC | 19:05 | |
| mtreinish | although the first one, the jenkins timeout error, we're hitting that 100% on the ceilo unit tests? | 19:05 |
| mtreinish | or 100% of the failures | 19:05 |
| *** thomasem has quit IRC | 19:07 | |
| *** jcooley_ has quit IRC | 19:08 | |
| *** oubiwann has quit IRC | 19:09 | |
| *** nati_ueno has quit IRC | 19:10 | |
| notmyname | jog0: now with a moving average http://not.mn/gate_status.html | 19:11 |
| *** melwitt has joined #openstack-infra | 19:12 | |
| ArxCruz | fungi: I need to know what's the best approach: | 19:13 |
| ArxCruz | mongodb isn't supported on ppc64 | 19:13 |
| *** rongze has joined #openstack-infra | 19:13 | |
| *** nati_ueno has joined #openstack-infra | 19:13 | |
| jog0 | notmyname: cool, although you have two orange lines | 19:13 |
| ArxCruz | Can I just put an if statement in params.pp and ignore the $::jenkins::params::mongodb_package, in slave.pp or you have other idea | 19:14 |
| dkranz | fungi: Do you know who has +2 on devstack-gate? | 19:14 |
| mtreinish | dkranz: it's the infra core team | 19:14 |
| fungi | dkranz: me, clarkb, jeblair, mordred | 19:14 |
| notmyname | jog0: one bold, one slim. kinda hard to get distinct colors when graphing 9 things | 19:14 |
| *** weshay has quit IRC | 19:14 | |
| jog0 | notmyname: and why not moving average over 6 hours? | 19:14 |
| jog0 | notmyname: heh yeah | 19:15 |
| dkranz | fungi: ok, thanks. This is needed to get the heat tests going https://review.openstack.org/#/c/61576/ | 19:15 |
| fungi | ArxCruz: you'd probably be fine adding a fact check on the machine architecture and then wrapping that package in a conditional or something | 19:15 |
| fungi | ArxCruz: easier to discuss when you have a first stab at it in review | 19:15 |
| fungi | dkranz: i'll try to take a look after meetings wrap up | 19:15 |
| notmyname | jog0: because I wanted an "in general, what's going on" line. the stairstep lines show very well what's happening for a given 6 hour time slot | 19:15 |
| *** dolphm has quit IRC | 19:16 | |
| *** herndon has quit IRC | 19:16 | |
| *** jooools has quit IRC | 19:17 | |
| *** sarob has quit IRC | 19:17 | |
| *** rongze has quit IRC | 19:18 | |
| *** gyee has quit IRC | 19:18 | |
| *** gothicmindfood has quit IRC | 19:19 | |
| *** herndon has joined #openstack-infra | 19:20 | |
| jog0 | notmyname: 1 day and 14 day moving averge are pretty similar | 19:20 |
| jog0 | so never mind | 19:20 |
| jog0 | I am surprisedby that | 19:20 |
| *** jcoufal-mob has joined #openstack-infra | 19:20 | |
| *** esker has joined #openstack-infra | 19:21 | |
| *** jcoufal-mob has quit IRC | 19:22 | |
| *** jcoufal-mob has joined #openstack-infra | 19:22 | |
| *** jcooley_ has joined #openstack-infra | 19:23 | |
| fungi | notmyname: the first paragraph of your explanation says 'the "Patch Pass Chance" is the starting point for the likelihood of that patch being merged into master' but it looks like you're graphing the chance that it will be merged without being automatically retried because of a failure ahead of it. that wording is a bit misleading if so | 19:23 |
| *** ruhe has joined #openstack-infra | 19:23 | |
| jog0 | fungi: agreed | 19:23 |
| *** jcoufal has quit IRC | 19:23 | |
| *** johnthetubaguy has quit IRC | 19:24 | |
| jog0 | notmyname: btw view-source:http://status.openstack.org/elastic-recheck/ | 19:24 |
| jog0 | not sure if that link will workbut we have some javascript that generates the graph URLs | 19:25 |
| fungi | if it's aborted and retried in the gate automatically, it still may merge without any manual intervention, after all | 19:25 |
| *** jcooley_ has quit IRC | 19:25 | |
| *** yolanda has quit IRC | 19:26 | |
| *** jcooley_ has joined #openstack-infra | 19:28 | |
| *** sarob has joined #openstack-infra | 19:29 | |
| *** thomasem has joined #openstack-infra | 19:29 | |
| *** harlowja is now known as harlowja_away | 19:30 | |
| *** jcooley_ has quit IRC | 19:31 | |
| *** jcooley_ has joined #openstack-infra | 19:31 | |
| notmyname | fungi: it's the chance that it will pass if it's at the top of the gate queue | 19:31 |
| fungi | ahh, okay. so that's only measuring changes which either get ejected or merged | 19:32 |
| *** thomasm_ has quit IRC | 19:32 | |
| notmyname | fungi: correct. and also therefore if you have a 80% chance of pass, a queue of depth 10 has a (100* .8**10) percent chance of passing | 19:33 |
| notmyname | (10.7% | 19:33 |
| fungi | curious then why the length of the gate queue at a particular point in time decreases the chances that the change at the head of the gate will merge. i would expect them to be mostly independent numbers | 19:35 |
| notmyname | fungi: 3 things are shown: (1) gate depth (2) actual pass rates for 6 gate jobs (3) computed chance of a new patch at the top of the gate passing | 19:37 |
| fungi | oh! i think i see why. they're both symptoms of gate job stability | 19:37 |
| lifeless | correlated:) | 19:37 |
| *** weshay has joined #openstack-infra | 19:37 | |
| notmyname | fungi: right, so a bigger gate means that more stuff is passing which means that the pass rate goes down (ie nondeterministic errors). which lowers (3) | 19:37 |
| fungi | the more frequently jobs fail in the gate, both the less likely it is for a change to merge and the larger the pile-up in the gate becomes | 19:37 |
| notmyname | fungi: right | 19:37 |
| fungi | whereas if failure rate is low, the gate clears out quickly (leading to low queue length numbers on the graph) | 19:38 |
| notmyname | fungi: and the corollary is that times like now (ie 1 job in the gate) are the only time you have of a high baseline likelihood that your patch will land | 19:38 |
| notmyname | fungi: right, so we want a high pass rate, regardless of the gate queue depth | 19:39 |
| notmyname | but the moving average shows that over the past 2 weeks we've not really gotten above about a 65% pass rate | 19:40 |
| fungi | just making sure it's apparent that the long gate is a symptom, not a cause | 19:40 |
| notmyname | yes, absolutely | 19:40 |
| jog0 | so correlation != causation here | 19:40 |
| jog0 | or rather its backwards | 19:40 |
| fungi | makes sense to me then | 19:40 |
| notmyname | fungi: I'd love to also graph the times that the gate queue is flished and rechecked (ie how many times a patch is rechecked), but that isn't tracked by zuul now | 19:40 |
| jog0 | notmyname: that doesn't normalize for how quickly things are being added to the queue | 19:41 |
| notmyname | fungi: jog0: yes, the gate queue depth is the consequence of the other metrics, not the cause | 19:41 |
| jog0 | notmyname: we can add more metrics to zuul | 19:41 |
| fungi | it's under discussion how best to do that, but it seems doable | 19:42 |
| *** ^d is now known as ^demon|lunch | 19:44 | |
| *** dprince has quit IRC | 19:44 | |
| *** CaptTofu has quit IRC | 19:45 | |
| *** dprince has joined #openstack-infra | 19:45 | |
| *** CaptTofu has joined #openstack-infra | 19:46 | |
| *** yassine has quit IRC | 19:49 | |
| *** jcooley_ has quit IRC | 19:51 | |
| *** jcooley_ has joined #openstack-infra | 19:51 | |
| *** vipul-away is now known as vipul | 19:51 | |
| *** jcoufal-mob has quit IRC | 19:52 | |
| *** vipul-away has joined #openstack-infra | 19:52 | |
| *** atiwari has quit IRC | 19:53 | |
| *** senk has joined #openstack-infra | 19:53 | |
| jeblair | 19:41 < notmyname> fungi: and the corollary is that times like now (ie 1 job in the gate) are the only time you have of a high baseline likelihood that your patch will land | 19:54 |
| jeblair | notmyname: that statement is not correct | 19:54 |
| jeblair | notmyname: a patch's chances of landing are not affected by those ahead of it | 19:55 |
| fungi | well, insofar as that an emptyish gate is a sign that changes are probably succeeding more often and not causing a pile-up | 19:55 |
| fungi | but again, that goes back to the symptom-not-disease point | 19:55 |
| notmyname | jeblair: ok, I should add "assuming your patch ok" at the end there | 19:56 |
| notmyname | *Patch is ok | 19:56 |
| *** jcooley_ has quit IRC | 19:56 | |
| jeblair | notmyname: that statement looks very close to saying "gate queue length affects the probability a patch will land", and it doesn't | 19:56 |
| *** yolanda has joined #openstack-infra | 19:57 | |
| fungi | i think my concern is that focusing on the queue depth as a metric is misleading, unless you're specifically looking at overall aggregate resource consumption | 19:57 |
| jeblair | notmyname: that's not a causal relationship (there may be a relationship as fungi pointed out in that a deep gate is a symptom of low prabability) | 19:57 |
| notmyname | jeblair: yes, I agree with that | 19:57 |
| jeblair | notmyname: to those less familiar, they might assume that means that if a patch at the head fails, that means their patch will fail too; that's the misapprehension i want to avoid. | 19:58 |
| notmyname | jeblair: sorry, I guess it was frustration coming though. the patch will still land (with the computed probablility), but it may take a long time if it's not at the top of the gate queue | 19:58 |
| jeblair | notmyname: ye | 19:58 |
| jeblair | s | 19:58 |
| notmyname | jeblair: ok. that shoudl be more clear. I agree | 19:58 |
| *** SushilKM__ has quit IRC | 19:59 | |
| *** jcooley_ has joined #openstack-infra | 19:59 | |
| notmyname | jeblair: so I have a few suggestions I'd like to propose during the meeting. want to hear them early? | 19:59 |
| *** SushilKM__ has joined #openstack-infra | 19:59 | |
| *** dhellmann_ is now known as dhellmann | 19:59 | |
| *** dcramer_ has joined #openstack-infra | 19:59 | |
| jog0 | notmyname: you should measure the gate depth in number of beers you can drink between +A and merge | 19:59 |
| notmyname | jog0: with a "red line" alcohol poisoning line? | 20:00 |
| jog0 | notmyname: yup | 20:00 |
| fungi | jog0: and the failure rate in beers you drank before you +A'd them | 20:00 |
| jog0 | haha | 20:00 |
| *** jcooley_ has quit IRC | 20:00 | |
| notmyname | jeblair: I just added some additional text to clarify (at the end of the explanation section) that http://not.mn/gate_status.html | 20:04 |
| *** ArxCruz has quit IRC | 20:06 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/zuul: Update the url for the zuul git repo https://review.openstack.org/62676 | 20:07 |
| fungi | notmyname: i like the clarification, but still the question in paragraph 3 "Is the patch pass chance high because the queue is empty?" is wrong. the pass chance won't ever be anything in particular because of the queue length, though the queue length may be empty because the pass chance is high... | 20:09 |
| *** heckj has joined #openstack-infra | 20:10 | |
| *** heckj has quit IRC | 20:10 | |
| *** senk has quit IRC | 20:10 | |
| fungi | reversed to "Is the queue empty because the patch pass chance is high?" would make more sense there | 20:10 |
| notmyname | fungi: well, the initial reason for adding the queue depth is because I wanted to see what was happening in the rare case that I graphed a 100% pass chance). Since I assume pass if there is no data, I added the gate queue depth to see if the "pass" rate really just meant nothing was happening | 20:11 |
| *** heckj has joined #openstack-infra | 20:11 | |
| *** vkozhukalov has quit IRC | 20:11 | |
| fungi | oh, i see. so "Is the patch pass chance high because there is no data to analyze?" in that case | 20:11 |
| mgagne | zaro: ping | 20:11 |
| *** rcarrillocruz1 has joined #openstack-infra | 20:12 | |
| notmyname | right. I'll add a "graph no data at 0 instead of 100" version just to see how it looks | 20:12 |
| notmyname | fungi: this is where the balance of "accurate data vs visual communication" is hard | 20:12 |
| mgagne | zaro: you have 2 accounts with the same email in gerrit. You should get it fixed :P | 20:12 |
| fungi | notmyname: totally agree there | 20:12 |
| *** lcestari has quit IRC | 20:13 | |
| *** rcarrillocruz has quit IRC | 20:13 | |
| fungi | mgagne: i can help zaro with that | 20:13 |
| mgagne | fungi: cool =) | 20:13 |
| fungi | zaro: i'll take a look and see what's going on in the db | 20:13 |
| *** hashar has joined #openstack-infra | 20:14 | |
| *** harlowja_away is now known as harlowja | 20:14 | |
| *** denis_makogon_ has joined #openstack-infra | 20:16 | |
| notmyname | fungi: nope. not even worth showing it. adding in zeros doesn't change much one way or simply graphs the multiplication a zero the other (anything *0=0) | 20:16 |
| *** salv-orlando has joined #openstack-infra | 20:16 | |
| *** dolphm has joined #openstack-infra | 20:17 | |
| *** alexpilotti has joined #openstack-infra | 20:17 | |
| fungi | notmyname: i figured. trending when there are periods of no data is not something i've ever seen a good visualization for (except possibly density maps) | 20:17 |
| *** heckj has left #openstack-infra | 20:19 | |
| *** vipul-away is now known as vipul | 20:19 | |
| notmyname | fungi: reload to check for updated words | 20:19 |
| *** markmc has joined #openstack-infra | 20:19 | |
| *** denis_makogon_ is now known as denis_makogon | 20:19 | |
| fungi | notmyname: yeah, that wording is more accurate i think | 20:20 |
| notmyname | k, thanks | 20:20 |
| *** mtaylor is now known as mordred | 20:20 | |
| *** dolphm has quit IRC | 20:21 | |
| fungi | we don't want devs to be afraid of committing changes, we just want them to be mindful of possibly merging bad code | 20:21 |
| fungi | and being better able to analyze when someone has done so | 20:22 |
| notmyname | (1) I want devs to understand the project-wide impact a single bad test makes and (2) I want to consider that the current state of things isn't acceptable and we should discuss changes necessary to fix it long-term | 20:24 |
| notmyname | and I have 3 suggestions I'll proposed, and I hope others come up too | 20:24 |
| jog0 | notmyname: ayes | 20:24 |
| notmyname | hmm...maybe 4 ideas (but one isn't all that great) | 20:25 |
| *** johnthetubaguy has joined #openstack-infra | 20:25 | |
| jog0 | notmyname: these will be raised at the proejct meeting? | 20:25 |
| notmyname | that's my plan | 20:25 |
| jog0 | notmyname: excellent, I'll be there | 20:25 |
| fungi | with bells on | 20:26 |
| *** miqui has joined #openstack-infra | 20:27 | |
| *** dkliban has quit IRC | 20:27 | |
| harlowja | fungi u got a sec, trying to continue investigation of why https://review.openstack.org/#/c/60850/ is blocking up, https://review.openstack.org/#/c/61689/ i think u pointed me at earlier last week, can u guys let me know the owner of that, since i try to view it and it says 'no permissions' | 20:29 |
| harlowja | i think i know who the owner is, but not 100% sure (need name to bug to tell them to stop using drafts) | 20:30 |
| *** talluri has quit IRC | 20:30 | |
| *** talluri has joined #openstack-infra | 20:30 | |
| fungi | harlowja: sure, i'll find it for you | 20:30 |
| harlowja | thx much | 20:30 |
| reed | jeblair, fungi: https://bugs.launchpad.net/openstack-ci/+bug/1261902 let me know if you need Evgeny to provide more details | 20:31 |
| uvirtbot | Launchpad bug 1261902 in openstack-community "Ask OpenStack host hitting RAM limits" [Undecided,New] | 20:31 |
| fungi | or if the taskflow ptl just wants that change un-drafted, that's also an option | 20:31 |
| harlowja | hmmm, i guess that ptl is me, but let me see who it is first :-P | 20:31 |
| *** vipul is now known as vipul-away | 20:31 | |
| fungi | harlowja: Ivan Melnikov | 20:32 |
| harlowja | k | 20:33 |
| harlowja | i knew its, ha | 20:33 |
| *** rfolco has quit IRC | 20:33 | |
| *** johnthetubaguy has quit IRC | 20:33 | |
| harlowja | thx fungi , a taskflow-core memeber that knows where ivan is will bug him | 20:33 |
| *** talluri has quit IRC | 20:35 | |
| fungi | zaro: so you definitely have two gerrit accounts (6580 and 6987) with separate openids. the first uses khaido as an ssh username while the second uses zaro0508. both have submitted contact information in gerrit and agreed to the icla. based on recent activity I think 6987 must be the one you're using to upload changes. can you confirm it's what also shows for you at | 20:37 |
| fungi | https://review.openstack.org/#/settings/ | 20:37 |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/requirements: Allow Falcon versions <0.2.0 https://review.openstack.org/62751 | 20:38 |
| fungi | zaro: if so, i can clear the accounts.preferred_email field for the old 6580 account so people will be able to add you as a reviewer and/or group member | 20:38 |
| *** rcarrillocruz1 has quit IRC | 20:39 | |
| *** gyee has joined #openstack-infra | 20:39 | |
| *** zhiyan has quit IRC | 20:42 | |
| *** ^demon|lunch is now known as ^d | 20:43 | |
| *** vipul-away is now known as vipul | 20:43 | |
| *** dstufft|laptop has joined #openstack-infra | 20:44 | |
| *** denis_makogon has left #openstack-infra | 20:45 | |
| *** Ajaeger1 has joined #openstack-infra | 20:46 | |
| *** sarob_ has joined #openstack-infra | 20:47 | |
| Ajaeger1 | Hi, I need some help with https://review.openstack.org/#/c/61204/, please. | 20:48 |
| Ajaeger1 | clarkb suggested to package openstack-doc-tools and distribute it somehow. | 20:48 |
| Ajaeger1 | is there any example that I can follow? | 20:49 |
| fungi | Ajaeger1: you may in that case be interested in http://git.openstack.org/cgit/openstack-dev/cookiecutter/tree/ | 20:49 |
| *** sarob has quit IRC | 20:49 | |
| miqui | clarkb: i would like to create blueprints against zuul.... | 20:49 |
| fungi | the readme is very short and i think covers most of what you need to know | 20:50 |
| Ajaeger1 | thanks, fungi | 20:50 |
| miqui | jblair: i would like to create blueprints against zuul.... | 20:51 |
| miqui | jeblair: i would like to create blueprints against zuul.... | 20:51 |
| fungi | miqui: looks like we haven't configured blueprints as an lp feature for the zuul project there. i guess the first question is whether there are any objections to doing so (in which case i'd be happy to configure that if it's desirable) | 20:52 |
| *** vipul is now known as vipul-away | 20:52 | |
| *** sarob_ has quit IRC | 20:52 | |
| fungi | but i don't want to go turning it on if we omitted that for some particular reason i'm unaware of | 20:53 |
| miqui | thanks fungi: | 20:53 |
| miqui | fungi: sure please let me know...the area for docs org and ramp up, i can add here from our experience.... | 20:54 |
| fungi | miqui: jeblair will hopefully be free to discuss it after the tc and project meetings wrap up around 2200 utc (unless he needs to disappear for a late lunch) | 20:54 |
| miqui | plus would like to contrib in other areas of the project... | 20:54 |
| miqui | ok... | 20:54 |
| miqui | thanks...fungi | 20:54 |
| fungi | miqui: that's wonderful. we love having more contributors | 20:55 |
| Ajaeger1 | fungi: I converted it already into a package - without cookiecutter. Should have known earlier about it but will check it out again. | 20:56 |
| zaro | fungi: i prefer zaro0508. | 20:56 |
| *** dolphm has joined #openstack-infra | 20:56 | |
| Ajaeger1 | How do I get the package build and distributed so that we can use it for gating? | 20:57 |
| fungi | Ajaeger1: okay, so next steps are going to be making sure you can sdist and install it, and then making sure we register it on pypi with the right permissions so jenkins can upload there | 20:57 |
| *** rcarrillocruz has joined #openstack-infra | 20:57 | |
| fungi | Ajaeger1: what's the repository called? | 20:58 |
| Ajaeger1 | openstack-doc-tools | 20:58 |
| *** SushilKM__ has quit IRC | 20:58 | |
| Ajaeger1 | will you register it on pypi? | 20:59 |
| Ajaeger1 | How do I trigger that? | 20:59 |
| fungi | Ajaeger1: yes, i can register that in a bit (about to start cooking dinner) | 20:59 |
| Ajaeger1 | fungi: no need to burn your dinner for that ;) | 21:00 |
| fungi | uploads get triggered by jenkins jobs which act on signed tags being pushed to the repository | 21:00 |
| Ajaeger1 | who would add the signed tags? | 21:01 |
| fungi | i hope to be able to successfully lurk through the project meeting starting now while i cook, so trying to do three things at once may exceed my manufactured tolerances | 21:01 |
| fungi | Ajaeger1: possibly you would (we'd need a release coordinator for that project, so it's defined by a group in gerrit) | 21:01 |
| Ajaeger1 | fungi: Can we continue discussing via email? I need to dropoff soon as well... | 21:01 |
| fungi | Ajaeger1: sure thing--a thread on the infra ml would be fine | 21:01 |
| Ajaeger1 | fungi: ok, will do. Enjoy dinner! | 21:02 |
| fungi | i can follow up with the relevant details there | 21:02 |
| fungi | thanks@ | 21:02 |
| Ajaeger1 | fungi: thank you ! | 21:03 |
| *** hunner1 has joined #openstack-infra | 21:04 | |
| *** hunner1 is now known as Hunner | 21:05 | |
| *** dkliban has joined #openstack-infra | 21:06 | |
| *** sandywalsh_ has quit IRC | 21:06 | |
| *** dims has quit IRC | 21:08 | |
| *** dims has joined #openstack-infra | 21:09 | |
| *** dstufft|laptop has quit IRC | 21:10 | |
| *** yamahata has joined #openstack-infra | 21:10 | |
| *** vipul-away is now known as vipul | 21:12 | |
| *** dcramer_ has quit IRC | 21:15 | |
| *** jasond` has left #openstack-infra | 21:17 | |
| *** sandywalsh_ has joined #openstack-infra | 21:17 | |
| *** nati_ueno has quit IRC | 21:20 | |
| *** yolanda has quit IRC | 21:24 | |
| *** yassine has joined #openstack-infra | 21:26 | |
| *** dizquierdo has joined #openstack-infra | 21:27 | |
| *** yfried has joined #openstack-infra | 21:27 | |
| *** nati_ueno has joined #openstack-infra | 21:28 | |
| yfried | Hi guys, I'm looking to get the external connectivity (ie - ping 8.8.8.8) on the devstack-neutron gate so this patch can work: https://review.openstack.org/#/c/55146/ | 21:29 |
| yfried | can anyone help me? | 21:29 |
| *** dcramer_ has joined #openstack-infra | 21:30 | |
| *** headsman has joined #openstack-infra | 21:32 | |
| clarkb | yfried: what do you need help with? | 21:32 |
| *** Ajaeger1 has quit IRC | 21:32 | |
| yfried | clarkb: for starters - is it doable? | 21:33 |
| clarkb | yfried: should be, the infrastructure won't prevent such a thing | 21:33 |
| clarkb | I know fungi has suggested you ping several external hosts and pass if at least one responds | 21:34 |
| yfried | clarkb: that's a different issue | 21:34 |
| clarkb | is there an issue? | 21:34 |
| clarkb | I think that is what I am trying to get at. Infra can help but we don't know what help is needed | 21:34 |
| yfried | clarkb: make it so the neutron gate will be configured with external connectivity | 21:35 |
| jhesketh_ | Hey | 21:35 |
| clarkb | yfried: there is nothing preventing that | 21:35 |
| clarkb | yfried: devstack/neutron/whatever can just do it | 21:35 |
| yfried | clarkb: I'm sorry if I sound weird. I'm very new to this | 21:36 |
| clarkb | yfried: no problem. Let me start again | 21:36 |
| clarkb | the infra team provides the infrastructure that the tests are run on but does not write the tests or configure the clouds the tests run against | 21:37 |
| yfried | clarkb: AFAIK the devstack instances created by jenkins when checking patches are configured without external connectivity | 21:37 |
| clarkb | there is nothing on the existing infrastructure that would prevent external connectivity. Instead you would need to have devstack configure neutron appropriately to ping the outside world | 21:37 |
| yfried | clarkb: so who is responsible for that | 21:37 |
| clarkb | yfried: the devstack and neutron teams | 21:37 |
| *** headsman has quit IRC | 21:38 | |
| yfried | clarkb: ok. tnx. sorry I botherd you | 21:38 |
| clarkb | yfried: no problem. I just don't think the infra team will be able to help with neutron configs much | 21:38 |
| *** dprince has quit IRC | 21:38 | |
| *** briancline has quit IRC | 21:39 | |
| *** sigmacorp has joined #openstack-infra | 21:41 | |
| *** mriedem has joined #openstack-infra | 21:43 | |
| *** SushilKM has joined #openstack-infra | 21:43 | |
| *** sigmacorp has quit IRC | 21:46 | |
| *** thomasbiege has joined #openstack-infra | 21:46 | |
| *** yfried has left #openstack-infra | 21:50 | |
| *** rcarrillocruz has quit IRC | 21:50 | |
| * fungi clearly didn't pop enough popcorn for the project meeting | 21:50 | |
| *** thomasbiege has quit IRC | 21:51 | |
| *** briancline has joined #openstack-infra | 21:52 | |
| pleia2 | heh | 21:52 |
| *** bknudson has quit IRC | 21:53 | |
| *** rnirmal has quit IRC | 21:53 | |
| *** ryanpetrello has quit IRC | 21:53 | |
| *** bknudson has joined #openstack-infra | 21:54 | |
| *** SushilKM has quit IRC | 21:56 | |
| jeblair | sdague: logstash workers are caught up | 21:58 |
| sdague | jeblair: \o/ | 21:59 |
| *** jcoufal has joined #openstack-infra | 21:59 | |
| *** nati_uen_ has joined #openstack-infra | 22:01 | |
| fungi | jeblair: miqui was inquiring whether we had disabled blueprints for launchpad.net/zuul or if it was okay for us to turn them on | 22:02 |
| fungi | er, whether we had disabled them intentionally | 22:02 |
| *** nati_uen_ has quit IRC | 22:03 | |
| jog0 | that was a fun meeting | 22:03 |
| *** branen has quit IRC | 22:03 | |
| *** nati_ueno has quit IRC | 22:03 | |
| *** nati_uen_ has joined #openstack-infra | 22:03 | |
| jeblair | fungi: well, we don't really use them. i suppose we could, but usually a mailing list message will do just as well | 22:03 |
| jeblair | miqui: what's the blueprint you want to file? | 22:04 |
| *** thomasbiege has joined #openstack-infra | 22:04 | |
| *** dstanek has quit IRC | 22:04 | |
| jeblair | if it's not too big, let's just use the mailing list; if it is a big project, we can turn on blueprints... | 22:04 |
| miqui | jeblair: at this stage about documentation | 22:04 |
| jeblair | how does that sound? | 22:04 |
| *** melwitt has quit IRC | 22:04 | |
| *** melwitt has joined #openstack-infra | 22:04 | |
| *** mfer has quit IRC | 22:04 | |
| jeblair | (i expect we'll use storyboard's version of blueprints when we start using it) | 22:05 |
| miqui | jeblair: there is a defect about docs and would like to instead treat that as a blueprint and add out team's notes.... | 22:05 |
| miqui | ...and as we learn more...we add more... | 22:05 |
| *** melwitt1 has joined #openstack-infra | 22:06 | |
| miqui | what is storyboard? another agile tool? | 22:06 |
| jeblair | miqui: can you just propose patches to zuul's docs? | 22:06 |
| *** melwitt has quit IRC | 22:06 | |
| miqui | jeblair: sure....i can do that via etherpad? | 22:07 |
| miqui | oops sorry.. | 22:07 |
| miqui | i meatn launchpad | 22:07 |
| jeblair | miqui: storyboard is our bug/feature tracker in development | 22:07 |
| SergeyLukjanov | miqui, it should eventually replace launchpad | 22:07 |
| miqui | hmm not using launchpad anymore? | 22:07 |
| miqui | ahh...interesting... | 22:08 |
| miqui | whats the link for it..i would like to take a look... | 22:08 |
| SergeyLukjanov | miqui, not yet, storyboard is under dev now, it's in PoC status | 22:08 |
| jeblair | miqui: you can propose patches to zuul using the regular openstack dev workflow | 22:08 |
| SergeyLukjanov | miqui, https://github.com/openstack-infra/storyboard | 22:08 |
| jeblair | miqui: see http://git.openstack.org/cgit/openstack-infra/zuul/tree/README.rst | 22:08 |
| jeblair | miqui: and https://wiki.openstack.org/wiki/How_To_Contribute | 22:08 |
| *** thomasbiege has quit IRC | 22:08 | |
| miqui | jeblair: ok.... | 22:09 |
| *** ArxCruz has joined #openstack-infra | 22:09 | |
| jeblair | miqui: the doc sources are here: http://git.openstack.org/cgit/openstack-infra/zuul/tree/doc/source | 22:09 |
| jeblair | miqui: and get published here: http://ci.openstack.org/zuul/ | 22:09 |
| miqui | ...great...thanks... | 22:09 |
| dcramer_ | Hi zaro. I noticed you were talking about the clouddocs-maven-plugin stuff recently. | 22:09 |
| dcramer_ | Wanted to touch base and see if you have questions. | 22:10 |
| *** melwitt1 has quit IRC | 22:10 | |
| *** melwitt has joined #openstack-infra | 22:10 | |
| *** markmcclain has quit IRC | 22:10 | |
| miqui | SergeyLukjanov: is there way i can look at storyboard even in alpha version? | 22:10 |
| miqui | or whatever you have.... | 22:11 |
| SergeyLukjanov | miqui, it could be easily started from sources, instructions are in README | 22:11 |
| miqui | ok....where is the project in github? | 22:11 |
| SergeyLukjanov | https://github.com/openstack-infra/storyboard | 22:11 |
| miqui | SergeyLukjanov: great...thanks... | 22:12 |
| fungi | dcramer_: i was going to fill out the sonatype jira request for an org.openstack.whatever groupid and was just wanting to confirm the various details they ask for when filing it. i really know next to nothing about maven/java so a lot of the terms are a bit foreign to me | 22:12 |
| SergeyLukjanov | miqui, but it's really a PoC, and there're plans to rewrite it, so, the main thing that you can see now is the data model stuff | 22:12 |
| *** DennyZhang has joined #openstack-infra | 22:13 | |
| miqui | SergeyLukjanov: ok......question why the switch from launchpad ? | 22:13 |
| fungi | SergeyLukjanov: miqui: or https://git.openstack.org/cgit/openstack-infra/storyboard/tree/ | 22:13 |
| zaro | dcramer_: yes. we had discussions with sharwell. | 22:13 |
| SergeyLukjanov | miqui, there are tons of gaps like subscribing, release management, projects grouping | 22:13 |
| *** davidhadas_ has quit IRC | 22:14 | |
| fungi | SergeyLukjanov: miqui: remember, we don't really use github, we just happen to mirror a copy of the repositories to it | 22:14 |
| zaro | dcramer_: not sure if you talked to him yet, but it's about this change https://review.openstack.org/#/c/58349 | 22:14 |
| SergeyLukjanov | fungi, yup, I know, just an old habbit | 22:14 |
| miqui | fungi: then what do you use? | 22:15 |
| SergeyLukjanov | fungi, using git.o.o as origin everywhere ;) | 22:15 |
| miqui | SergeyLukjanov: exactly..... | 22:15 |
| jeblair | fungi: i don't think the free space graphs in cacti are automated (beyond / partition) | 22:15 |
| fungi | miqui: we use a code review system called gerrit at review.openstack.org and then mirror to a farm of git servers at git.openstack.org | 22:15 |
| jeblair | fungi: do you want to manually add them? | 22:16 |
| fungi | jeblair: ahh, okay, i can add them by hand then | 22:16 |
| SergeyLukjanov | miqui, here you can find a etherpad about storyboard from the last design summit - https://etherpad.openstack.org/p/icehouse-summit-storyboard-basic-concepts-and-next | 22:16 |
| jeblair | fungi: k; let me know if you have q's | 22:16 |
| zaro | dcramer_: is this change inline with original plans? | 22:16 |
| miqui | fungi: ah right.....gerrit... | 22:16 |
| miqui | yeah ok...sure forgot about those... | 22:16 |
| SergeyLukjanov | miqui, and we have a channel (#storyboard) and irc weekly meeting | 22:17 |
| miqui | SergeyLukjanov: awesome... thanks.. | 22:17 |
| *** ruhe has quit IRC | 22:17 | |
| fungi | jeblair: i may have questions later, but should be able to suss it out myself. currently tending several gallons of lentil soup for another half hour or so, then i'll take a look | 22:18 |
| *** markmc has quit IRC | 22:18 | |
| miqui | fungi: btw...noticed that good 'ol pip install /opt/zuul "seems to fail" quietly (i.e. seems installed - but no) - we had to use --upgrade flag... | 22:19 |
| miqui | re-ran with that and it was happy again... | 22:19 |
| dcramer_ | zaro: So there have been discussions since then regarding the need to be sure that the snapshot version is committed with the pom.xml. It looks like you guys were discussing that in your recent meeting. | 22:20 |
| * miqui thanks for your replies..... ttyl | 22:20 | |
| dcramer_ | zaro: I'm looking at http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-12-17-19.02.log.html and https://etherpad.openstack.org/p/java-release-process | 22:21 |
| dcramer_ | fungi: As I understand it, you request org.openstack and then you can create whatever you want under that. | 22:22 |
| jeblair | dcramer_: here's what i'd like to see happen on that: we land zaro's change and finish the plan that we started on a year ago. you guys try out the process and see how it works and what obstacles you have, and then propose improvements to it | 22:22 |
| dcramer_ | jeblair: But that doesn't address sharwell's concern regarding those doing local builds. It seems like if we just omit 58349, we can use everything else as is. | 22:23 |
| zaro | dcramer_: you won't get auto versioning and deployments. | 22:24 |
| zaro | dcramer_: alternative for you is to use maven release plugin, but you'll need to do that manually. | 22:25 |
| dcramer_ | zaro: No autoversionsing, but the deployments would still happen from Jenkins, correct? | 22:25 |
| zaro | dcramer_: there is alternative proposal in the etherpad that you guys can try out, the version plugin | 22:25 |
| dcramer_ | Yes, I was looking at that: https://etherpad.openstack.org/p/java-release-process | 22:26 |
| jeblair | dcramer_: there is a very small cost to you as developers to do this, and since this project has a small development team, it should not be a big burden to give it a try. | 22:26 |
| zaro | dcramer_: if there is no auto versioning then jenkins will keep deploying the same version. | 22:27 |
| dcramer_ | jeblair: I was seeing it as "Well the cost of me doing two git reviews per release is very low and does not change anything else about the workflow" | 22:27 |
| *** yassine has quit IRC | 22:28 | |
| dcramer_ | zaro: sharwell's proposal was for us to commit the pom changes manually to increment the verison. | 22:28 |
| zaro | dcramer_: ohhh i guess that's the problem. (manual) :( | 22:29 |
| dcramer_ | zaro: From my point of view there's a benefit that justifies it... | 22:29 |
| dcramer_ | zaro: If you're going to have your community use gerrit to review anything, it should be the decision to cut a release. | 22:29 |
| *** boris-42 has quit IRC | 22:29 | |
| jeblair | dcramer_: none of these proposals get you that | 22:30 |
| dcramer_ | zaro: By using gerrit to review the pom.xml change in preparation for a new release, you give your community an opportunity to raise concerns or ask for time to finish pending work. | 22:30 |
| sdague | jeblair: so in your copious free time, if you wanted to ever review devstack changes with yuor +2, it would be appreciated :) | 22:31 |
| dcramer_ | jeblair: Yes, when I do "git review" on a pom.xml change that removes the -SNAPSHOT, someone can -1 it and ask for another day. | 22:31 |
| jeblair | sdague: i'd like to; most of them are out of my area of competency, but if you happen to see any that are, feel free to add me | 22:31 |
| dcramer_ | jeblair: But I must admit, I'm not fully clear on who does the tagging in the tag based approach. | 22:32 |
| jeblair | sdague: my review load has been so high lately i haven't had time to go looking | 22:32 |
| jeblair | dcramer_: here's the thing -- we have this workflow that we use for _all_ of openstack. we'd like to use it for this project too. we talked about it, you agreed, zaro implemented. | 22:32 |
| sdague | yeh no worries. dtroyer_zz and I sort of decided we'd go to single approver on easy stuff over the holidays to make sure people aren't held up | 22:33 |
| jeblair | dcramer_: it's a great system and i still think you should try it. | 22:33 |
| sdague | but extra eyes always good. | 22:33 |
| sdague | ok, shoveling time | 22:33 |
| jeblair | dcramer_: but if you're not interested, i'm not sure how we can help. | 22:34 |
| dcramer_ | jeblair: Indeed, you guys do impressive work. But I see this as gerrit doing its thing, giving someone (in this case sharwell) an opportunity to bring forward a concern we didn't anticipate. | 22:34 |
| *** esker has quit IRC | 22:34 | |
| dcramer_ | jeblair: I'm very interested. | 22:34 |
| jeblair | dcramer_: he left a -2 | 22:35 |
| jeblair | dcramer_: that means he's opposed to it merging | 22:35 |
| jeblair | dcramer_: it's the strongest rejection of a patch possible | 22:35 |
| dcramer_ | jeblair: and still would get what I really want (and especially Anne Gentle will get what she wants) from the compromise approach. | 22:35 |
| dcramer_ | jeblair: Right, but just for that one change...the change that removes the version from the pom. | 22:36 |
| jeblair | dcramer_: yeah, it's the final change; the one that lets the system work as designed | 22:36 |
| dcramer_ | jeblair: As I understand the semantics -1 means "I'd prefer you don't merge, but you can if you really want to" and -2 is blocking. | 22:36 |
| *** oubiwann has joined #openstack-infra | 22:37 | |
| jeblair | dcramer_: yes, a blocking vote from someone who isn't at all familiar with the effort is surprising. | 22:37 |
| jeblair | dcramer_: that's why i think we should defer to the original plan, and ask sharwell to come back and improve it. | 22:38 |
| dcramer_ | jeblair: Right, but from someone who knows a lot about Maven and whose opinion we should take into account. I don't see -2 as having any kind of moral or derogatory content. | 22:38 |
| dcramer_ | jeblair: What's the problem with his suggested alternative? | 22:39 |
| dcramer_ | jeblair: That's the part I haven't heard (why it feels like we're talking at cross-purposes). | 22:39 |
| jeblair | dcramer_: it ignores everything we were trying to accomplish, and i don't think we should keep going back to the design phase on this. | 22:40 |
| dcramer_ | jeblair: But it doesn't ignore EVERYTHING we were trying to accomplish. The only thing it breaks is that the tag name is no longer the single point of truth for the version. All the other benefits remain. | 22:41 |
| jeblair | dcramer_: sharwell's proposal was basically "don't do what you wanted to do and just use maven instead" | 22:41 |
| dcramer_ | jeblair: Let me outline some benefits of what you're doing: | 22:42 |
| mgagne | jeblair: I was reading the conversation but have to go now. Although it adds no new value to the discussion, I found an article talking about this subject: http://axelfontaine.com/blog/final-nail.html | 22:42 |
| mgagne | jeblair: same conclusion: just inject from jenkins build number (or git tag in our case) | 22:43 |
| dcramer_ | mgagne: Thanks I'll look at that. It's not something I've read. | 22:43 |
| *** sarob has joined #openstack-infra | 22:44 | |
| *** DennyZhang has quit IRC | 22:44 | |
| dcramer_ | 1) Code is part of OpenStack and reviewed via gerrit. Bringing more disciplined code reviews and letting IBMers contribute (for legal reasons they need to protection of the CLA to contribute) | 22:45 |
| jeblair | dcramer_: being on stackforge doesn't necessarily mean it's part of openstack. as far as i know the repo is not part of an existing program. if it's part of docs, then we put it in the wrong place. | 22:46 |
| dcramer_ | 2) Release builds are (again) coming from a Jenkins machine instead of "some guy in a corner pushing up commits". Currently, I could accidentally cut a release that isn't committed anywhere. | 22:46 |
| zaro | clarkb: ping | 22:46 |
| *** SergeyLukjanov has quit IRC | 22:46 | |
| jeblair | dcramer_: if it's part of docs, we should clear that up with docs, and maybe the tc. it could also be part of infra. | 22:47 |
| dcramer_ | jeblair: So my understanding from annegentle_ was that this would enable IBMers to contribute due to CLA stuff. I don't mean to imply it's part of "OpenStack" proper. | 22:47 |
| jeblair | dcramer_: i can't speak to that kind of legal issue. the only thing i can say is that i'm pretty sure the tc (which gets to decide these things along with the board) don't think it's part of openstack. | 22:48 |
| jeblair | dcramer_: but if being on stackforge makes their legal team happy, ok. | 22:48 |
| dcramer_ | jeblair: That's fine. I really just want Anne to be happy. | 22:48 |
| dcramer_ | trying to remember 3)...had it a second ago. | 22:49 |
| lifeless | dcramer_: IBM *require* the CLA to contribute? | 22:50 |
| lifeless | dcramer_: so they can't contribute to projects that don't have teh CLA-required bit set? | 22:50 |
| dcramer_ | lifeless: You'll have to check with Anne. | 22:50 |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Add cassandra-driver dependency https://review.openstack.org/62726 | 22:51 |
| dcramer_ | lifeless: But you just reminded me of 3) | 22:51 |
| clarkb | zaro: hi | 22:51 |
| lifeless | annegentle_: ^ | 22:51 |
| fungi | zaro: btw i cleared up your e-mail address and external ids in gerrit for the old account, so should be all set now | 22:51 |
| dcramer_ | lifeless: yes, annegentle_ | 22:51 |
| fungi | (i think he was attempting to perform a summoning incantation) | 22:51 |
| annegentle_ | lifeless: dcramer_ reading | 22:51 |
| zaro | fungi: why thanks a bunch. | 22:52 |
| dcramer_ | fungi: Ok, thanks, I suck at irc. | 22:52 |
| jeblair | it worked | 22:52 |
| zaro | clarkb: i just installed latest jenkins and i can't find the copy log option anymore. | 22:52 |
| fungi | indeed | 22:52 |
| *** thomasem has quit IRC | 22:52 | |
| jeblair | zaro: it's in an unreased version of the plugin | 22:52 |
| annegentle_ | ok I can't read that much and be on this call. Summary? :) | 22:52 |
| zaro | jeblair: that's good to know. | 22:53 |
| annegentle_ | ok that was oct 2012 so who knows if other member companies want a "blessing" to work on the maven plugin | 22:53 |
| jeblair | zaro: try git master | 22:53 |
| annegentle_ | but at the time, the IBM legal didn't want them to put devs on a rackspace org tool | 22:54 |
| lifeless | annegentle_: rumour that IBMers can't contribute to projects that *do not* require a CLA ? | 22:54 |
| fungi | sounds more like they just wanted it to be open-source and community-developed | 22:54 |
| dcramer_ | 3) Ensure that if I'm hit by a truck the release process is owned by a team she trusts. | 22:54 |
| *** loq_mac has joined #openstack-infra | 22:55 | |
| dcramer_ | I do bike to work sometimes and there are lots of trucks in Texas. | 22:55 |
| annegentle_ | dcramer_: eeek. yes. that articulates my concern. With of course a God forbid. | 22:56 |
| jeblair | dcramer_: okay, so i think we did a pretty good job coming up with a nice system that addresses those concerns and makes the development and release of the plugin match what we're doing for (literally) more than one hundred other projects | 22:57 |
| *** sarob has quit IRC | 22:57 | |
| dcramer_ | jeblair: So it seems that we're hung up over one small (to my mind esoteric) detail: Is the version of the release stored only in the tag or also in the pom? | 22:57 |
| dcramer_ | jeblair: Yes, your stuff is awesome and amazing. | 22:58 |
| zaro | dcramer_: it's in both. | 22:59 |
| *** johnthetubaguy has joined #openstack-infra | 22:59 | |
| jeblair | dcramer_: i don't want to start that process over again. we have a lot of things we need to be working on, even in the java realm. | 23:00 |
| dcramer_ | jeblair: Right, and I really don't want you guys to do any more work than you already have. | 23:00 |
| sdague | dcramer_: jeblair's point on over 100 other projects isn't bragging, it's that *no one* gets to be a unique snow flake. It doesn't scale. :) | 23:01 |
| *** dims has quit IRC | 23:01 | |
| zaro | jeblair: just saw the history. yikes! scp hasn't released new version for 3yrs. | 23:01 |
| *** jcoufal has quit IRC | 23:01 | |
| *** jpich has quit IRC | 23:01 | |
| jeblair | dcramer_: great, so as a stackforge project, you can manage your own infrastructure. you can write your own jenkins jobs, etc. | 23:01 |
| *** harlowja is now known as harlowja_away | 23:02 | |
| dcramer_ | jeblair: Nooo...no asking for that. | 23:02 |
| dcramer_ | sdague: Ok, point taken. | 23:02 |
| fungi | sdague: we're well over 200 counting stackforge too | 23:02 |
| dcramer_ | jeblair, sdague: so I'm beginning to see the root of the issue: | 23:03 |
| sdague | fungi: I said over :) | 23:03 |
| *** iccha_ is now known as iccha-dinner | 23:03 | |
| fungi | you definitely did | 23:03 |
| sdague | anyway, I need to meet people for beers and planning for our LUG. see you folks later | 23:04 |
| *** hashar has quit IRC | 23:04 | |
| *** johnthetubaguy has quit IRC | 23:04 | |
| jeblair | dcramer_: if you aren't doing things the way we do the rest of the java projects in infra, what we do is probably going to evolve over time and you'll want to make sure to allocate some time to keep up with changes, etc. but all the tools are available; you might have seen zaro's changes where he wrote the jenkins jobs for instance. you can do that too. | 23:04 |
| *** melwitt1 has joined #openstack-infra | 23:05 | |
| *** melwitt has quit IRC | 23:05 | |
| *** harlowja_away is now known as harlowja | 23:05 | |
| jeblair | zaro: yeah, i have jenkins commit access and can merge changes there; i don't really like it though because the copyright status is ambiguous. i'm not sure i'd make a release | 23:05 |
| dcramer_ | jeblair: I see the source of your concern now: if we do this one small thing in a way that's different from your other projects, you're worried that at some point changes you make for the 100 other projects will break us. | 23:06 |
| jeblair | zaro: but i also think i broke upgrade compatibility on master, so that needs to be fixed. | 23:06 |
| *** eharney has quit IRC | 23:07 | |
| jeblair | dcramer_: it's only a handful of other java projects (most of our projects are python) but it's based on the same process so even if the techical details are shared with 4 projects, the dev workflow and release process is shared with all 100+ | 23:07 |
| *** wenlock has quit IRC | 23:07 | |
| jeblair | dcramer_: so: qualified yes. | 23:08 |
| *** senk1 has joined #openstack-infra | 23:08 | |
| dcramer_ | jeblair: Ok, at least I understand the source of your concern now. I had been quite confused previously. | 23:08 |
| jeblair | dcramer_: we're an industrial scale software factory. :) | 23:08 |
| dcramer_ | jeblair: I don't think I ever saw zaro's jenkins job. | 23:09 |
| dcramer_ | jeblair: You can have a car that's any color you want as long as it's black. | 23:09 |
| dcramer_ | jeblair: Henry Ford is supposed to have said something like that. | 23:09 |
| jeblair | dcramer_: https://review.openstack.org/#/c/46099/ | 23:09 |
| dcramer_ | jeblair: I understand the constraints that kind of scale imposes. | 23:10 |
| jeblair | dcramer_: my favorite color | 23:10 |
| *** david-lyle_ has quit IRC | 23:10 | |
| *** dstanek has joined #openstack-infra | 23:10 | |
| devananda | hi guys. I just tagged another rev of python-ironicclient. zuul did its thing, and its on pypi.python now, but i don't see python-ironicclient /anywhere/ on http://pypi.openstack.org/openstack/ | 23:14 |
| *** thomasem has joined #openstack-infra | 23:15 | |
| jeblair | devananda: check out http://status.openstack.org/zuul/ | 23:15 |
| jeblair | devananda: under 'release' you'll see the mirror update jobs are running | 23:15 |
| jeblair | devananda: if python-orinicclient is in openstack/requirements, then it will show up there when they're done | 23:15 |
| devananda | jeblair: right. and a countdown to ~ when they finish | 23:15 |
| *** dims has joined #openstack-infra | 23:16 | |
| jeblair | sorry i can't spell the name of your project | 23:16 |
| *** thomasem has quit IRC | 23:16 | |
| devananda | jeblair: ah! that's the missing bit of info. It's *not* in requirements yet .... https://review.openstack.org/#/c/60638/1 | 23:16 |
| jeblair | devananda: however, it isn't in requirements... | 23:16 |
| jeblair | yeah | 23:16 |
| devananda | hence why even the prior relesae isn't up there | 23:16 |
| jeblair | so it won't show up yet | 23:16 |
| devananda | gotcha. thanks | 23:16 |
| *** thomasem has joined #openstack-infra | 23:16 | |
| * devananda looks for who to poke about that review | 23:17 | |
| jeblair | devananda: the requirements repo has similar post jobs, so once that change lands, it'll show up | 23:17 |
| devananda | cool | 23:17 |
| fungi | (offset by the run-length duration of the mirror jobs at any rate) | 23:17 |
| jeblair | devananda: that seems like one i can weigh in on. | 23:17 |
| dcramer_ | jeblair: So the risk in sharwell's proposal is (say a neuton bomb takes me and sharwell out) and someone comes along, assuming our project works like all the others, does NOT commit the change to the pom and cuts a release by setting the tag. | 23:18 |
| devananda | jeblair: great. thanks much! | 23:18 |
| dcramer_ | jeblair: That's a legitimate concern. Let me discuss it with sharwell. | 23:19 |
| dcramer_ | jeblair: The next release will be in January, so that's when we need to decide something. | 23:19 |
| dcramer_ | jeblair: Have to run now. | 23:20 |
| *** yamahata has quit IRC | 23:20 | |
| ekarlso | strptime('17/Dec/2013:20:11:51 +0100', '%d/%b/%Y:%X %z') < works in py3 but 2.7 ? | 23:20 |
| ekarlso | wrong chan | 23:20 |
| *** reed has quit IRC | 23:22 | |
| *** thomasem has quit IRC | 23:23 | |
| *** dcramer_ has quit IRC | 23:24 | |
| *** MarkAtwood has joined #openstack-infra | 23:24 | |
| *** weshay has quit IRC | 23:25 | |
| *** ^d has quit IRC | 23:25 | |
| *** shardy is now known as shardy_afk | 23:26 | |
| lifeless | I am confuse | 23:33 |
| lifeless | how did https://review.openstack.org/#/c/62520/ fail ? The tempest results are green | 23:33 |
| lifeless | in the first failure | 23:33 |
| jeblair | lifeless: 2013-12-17 02:41:13.050 | Logs have errors | 23:34 |
| jeblair | lifeless: that should probably be a bit more shouty. | 23:34 |
| jeblair | 2013-12-17 02:41:12.844 | 2013-12-17 02:24:45.716 ERROR nova.network.manager [req-0fec8653-af83-402a-b397-8f9aa437e8c0 ListServersNegativeTestJSON-tempest-1193702949-user ListServersNegativeTestJSON-tempest-1193702949-tenant] Unable to release 10.1.0.5 because vif doesn't exist. | 23:34 |
| jeblair | lifeless: so that's the new 'make sure we aren't outputting new errors that aren't on the whitelist' check. | 23:35 |
| lifeless | oh you | 23:35 |
| lifeless | bah | 23:35 |
| lifeless | oh joy, vif races | 23:35 |
| lifeless | now to find the recheck bug for that | 23:36 |
| lifeless | jeblair: thanks | 23:36 |
| jeblair | lifeless: the postgres job failed with a _different_ error log (but tests passed) | 23:36 |
| *** ^d has joined #openstack-infra | 23:36 | |
| jeblair | lifeless: and neutron actually did fail tests | 23:36 |
| jeblair | lifeless: so, wow that's a _lucky_ change! | 23:36 |
| lifeless | jeblair: yeah. 'Lucky'. | 23:37 |
| *** ^d has quit IRC | 23:41 | |
| *** branen has joined #openstack-infra | 23:46 | |
| *** rcleere has quit IRC | 23:47 | |
| *** dolphm has quit IRC | 23:47 | |
| *** dstanek has quit IRC | 23:51 | |
| *** jhesketh__ has joined #openstack-infra | 23:53 | |
| jog0 | lifeless: isn't the gate fun | 23:54 |
| lifeless | jog0: for sadomasochists... | 23:54 |
| jog0 | yeah ... so I think we can map the gate issues to not treating critical bugs as critical | 23:55 |
| *** fbo is now known as fbo_away | 23:55 | |
| jog0 | now that we can track what bugs are causing issues moderately well we just need to fix em | 23:55 |
| *** praneshp has quit IRC | 23:56 | |
| *** ^d has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!