| fungi | k | 00:00 |
|---|---|---|
| *** yamahata has joined #openstack-infra | 00:01 | |
| *** wenlock has quit IRC | 00:01 | |
| *** weshay has quit IRC | 00:09 | |
| *** ^d has quit IRC | 00:12 | |
| *** fallenpegasus has joined #openstack-infra | 00:14 | |
| *** dcramer_ has quit IRC | 00:14 | |
| fungi | the graphs seem to be recovering | 00:16 |
| fungi | sparklines too | 00:16 |
| *** sarob has quit IRC | 00:17 | |
| *** sarob has joined #openstack-infra | 00:17 | |
| *** _cjones_ has quit IRC | 00:20 | |
| *** changbl has quit IRC | 00:21 | |
| fungi | precise7 is back online in jenkins01 after a clean reboot, and is running jobs, no jenkins agent errors now | 00:21 |
| *** sarob has quit IRC | 00:22 | |
| *** fallenpegasus has quit IRC | 00:23 | |
| *** ArxCruz has joined #openstack-infra | 00:23 | |
| *** fallenpegasus has joined #openstack-infra | 00:24 | |
| *** jasondotstar has quit IRC | 00:25 | |
| *** ryanpetrello has joined #openstack-infra | 00:26 | |
| *** ryanpetrello has quit IRC | 00:27 | |
| *** fallenpegasus has quit IRC | 00:27 | |
| *** fallenpegasus has joined #openstack-infra | 00:28 | |
| *** ryanpetrello has joined #openstack-infra | 00:29 | |
| *** slong has joined #openstack-infra | 00:30 | |
| *** hogepodge has quit IRC | 00:31 | |
| *** slong_ has quit IRC | 00:31 | |
| dims_ | fungi was this the error from jenkins agent? "[EnvInject] - [ERROR] - SEVERE ERROR occurs: java.lang.InterruptedException" | 00:32 |
| *** fallenpegasus has quit IRC | 00:32 | |
| dims_ | 127 hits in last 4 hours | 00:32 |
| *** ArxCruz has quit IRC | 00:32 | |
| fungi | dims_: yeah, we had several slaves go off the rails and need restarts. if it was precise7, 9 or 37 those should all be okay again now | 00:33 |
| *** dstanek has joined #openstack-infra | 00:35 | |
| fungi | ugh, spotted another havana change devs keep blindly reverifying even though grenade is *not* going to succeed | 00:37 |
| dims_ | cool. is there a way i can add that precise7/9/37 as a field? (in logstash) right now i have to search for it in the logs. we may be able to tell if certain jobs fail on certain hosts | 00:38 |
| dims_ | ouch | 00:38 |
| clarkb | dims if you add thst info to the zmq event publisher plugin for jenkins | 00:39 |
| fungi | which means touching java, so be sure to wear gloves | 00:40 |
| dims_ | :) | 00:40 |
| dims_ | clarkb, which repo? i can give it a shot | 00:41 |
| clarkb | openstack-infra/zmq-mumblemumble | 00:42 |
| dims_ | thx | 00:42 |
| *** ArxCruz has joined #openstack-infra | 00:47 | |
| *** ryanpetrello has quit IRC | 00:48 | |
| *** fallenpegasus has joined #openstack-infra | 00:51 | |
| marun | the dumb stuff continues with 3rd party jobs f'ing up review comment logs: https://review.openstack.org/#/c/61105/ | 00:52 |
| marun | is there anyone else seeing this is a problem and considering better ways of integrating advisory testing than review comments? | 00:53 |
| fungi | wow. crazy comment loop | 00:53 |
| fungi | marun: i think our "proprietary technology plugin" architecture is an attractive nuisance, but i'm probably not representative of a majority of the community there | 00:54 |
| marun | fungi: I have had similar thoughts, but I've been convinced it's worth it. | 00:55 |
| fungi | companies want to sink time into writing driver shims rather than collaborating on standard protocols | 00:55 |
| marun | fungi: The alternative is those vendors doing things entirely outside of openstack and then we don't know enough to know what different vendors share (with an eye to factoring it out for reuse). | 00:56 |
| marun | fungi: But to your point, it's api/protocol stability vs just having the source and doing whatevere. | 00:56 |
| fungi | yes, that has its own fragmentation risks, and none of us who survived the unix wars wants to relive it | 00:56 |
| marun | fungi: the former is certainly preferable, but requires a certain maturity that I'm not sure any openstack project really has yet. | 00:56 |
| fungi | agreed | 00:57 |
| *** dcramer_ has joined #openstack-infra | 00:57 | |
| marun | fungi: so, in the meantime, I think we need a better way of reporting advisory testing results. | 00:57 |
| marun | fungi: something explicit, ideally. say, a per-patch section reporting the results of different testing mechanisms | 00:58 |
| fungi | to your original question, weigh in on https://review.openstack.org/63478 where we're at least trying to wrangle it a bit | 00:58 |
| marun | fungi: (in a perfect world) | 00:58 |
| marun | fungi: ah, ok. | 00:58 |
| fungi | marun: part of the problem, i think, is that this was originally expected to support a handful of smokestack-like systems, and we went from that to 30+ in the span of a few months | 00:59 |
| *** rwsu has quit IRC | 01:00 | |
| fungi | i agree that if every neutron change is going to start having 20 +/-1 vrfy votes on it, this becomes a bit of an interface usability issue | 01:00 |
| marun | fungi: review comments are already a poor substitute for a mailing list discussion, and introducing spam just makes it impossible | 01:01 |
| fungi | marun: i concur. there are some things gerrit lets you do which aren't really as easy on a mailing list, but it has its own drawbacks as well | 01:02 |
| *** cyeoh has quit IRC | 01:02 | |
| marun | fungi: i was initially a pretty big fan of tools like reviewboard and gerrit, but the gloss has come off for me. | 01:03 |
| marun | fungi: the side-by-side colorized diffs are definite pluses bug I think the lack of coherency to a review conversation may negate its advantage | 01:03 |
| fungi | overall, having done development in a ml-driven community and in openstack's gerrit setup, i prefer the latter for scalability. if we were to try to review the change volume for all of openstack's subprojects on an ml, i think it would be untenable | 01:03 |
| fungi | having the ability to add comments in context and subthread discussions around different chunks of a change and easily browse between them is pretty helpful | 01:04 |
| dims_ | clarkb, jeblair has already added the node name in the zmq publisher (https://github.com/openstack-infra/zmq-event-publisher/commit/36ca349ad10c3f17e69a70c73fefcb2ae3405ca7) | 01:05 |
| marun | fungi: fair enough | 01:05 |
| fungi | having humans organize similar review discussions between multiple lists and threads gets nasty (lkml nasty) | 01:05 |
| marun | fungi: in any case, regardless of merit, gerrit isn't going away | 01:06 |
| marun | fungi: I'm hoping to prevent a regression in utility that these advisory jobs appear poised to deliver. | 01:06 |
| fungi | agreed | 01:06 |
| fungi | marun: so far that proposed doc update is the best rallying point we have around this discussion, though we could consider changing venue if it needs more actual discussion and less general document review | 01:07 |
| marun | fungi: I don't think policy is going to be enough - I think we need a different reporting mechanism. | 01:08 |
| fungi | marun: you very well may be right. just suggesting it needs to be raised as a point to a wider audience. right now the interested audience has been gathering on that doc review | 01:08 |
| marun | fungi: ah, fair enough. I'll raise the issue there then. | 01:09 |
| fungi | it could make for a great -dev ml thread though. so far there have been individual discussions within neutron, nova, infra and so on | 01:09 |
| *** jroovers has quit IRC | 01:09 | |
| fungi | the ideas and experiences to date need to merge into a larger debate | 01:10 |
| *** ArxCruz has quit IRC | 01:10 | |
| marun | fungi: Hmmm, and I guess I can point to a ml thread on the review in any case. | 01:11 |
| fungi | on the infra plus side, the recent onslaught of third-party testing requirements has gained numerous entities a much deeper understanding of our test infrastructure, so i think that's a great outcome | 01:11 |
| marun | Definitely a good thing | 01:14 |
| *** thuc has quit IRC | 01:14 | |
| marun | OpenStack efforts pretty much live and die by the quality of the CI effort, whether upstream or down. Better not to have everyone reinvent the wheel. | 01:15 |
| *** thuc has joined #openstack-infra | 01:15 | |
| *** dstanek has quit IRC | 01:16 | |
| *** thuc has quit IRC | 01:19 | |
| openstackgerrit | Auston McReynolds proposed a change to openstack-infra/reviewstats: Update Trove Core https://review.openstack.org/65587 | 01:20 |
| *** fallenpegasus has quit IRC | 01:21 | |
| *** melwitt has quit IRC | 01:28 | |
| *** nosnos has joined #openstack-infra | 01:29 | |
| *** amotoki has quit IRC | 01:31 | |
| *** mriedem has joined #openstack-infra | 01:31 | |
| *** UtahDave has quit IRC | 01:35 | |
| *** fifieldt has joined #openstack-infra | 01:37 | |
| clarkb | fungi: I just marked precise4 offline on jenkins02, it was very quickly fialing things :( you don't happen to still be about do you? | 01:39 |
| fungi | i am | 01:39 |
| clarkb | fungi: did you want to poke at it or should I? I will probably just reboot it then reconnect it to jenkins which isn't very informative | 01:39 |
| fungi | i'll see if i can shepherd it back in, but we've been losing slaves like that right and left on both jenkins01 (old version) and 02 (newer) so it's not a new bug i guess | 01:40 |
| clarkb | that is annoying | 01:41 |
| fungi | i think we're just grinding them into dust with the current gate volume | 01:41 |
| fungi | (the old masters, i mean) | 01:41 |
| *** slong has quit IRC | 01:42 | |
| fungi | they've both been restarted in the past 48 hours, so it's not an uptime thing either | 01:43 |
| *** prad has joined #openstack-infra | 01:46 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Add jenkins slave name to the logstash records https://review.openstack.org/65593 | 01:52 |
| clarkb | fungi: fyi, the logstash processors are slightly behind. We may need to readd logstach-worker05-08 back to the mix | 01:53 |
| clarkb | fungi: but I think it is ok for now | 01:53 |
| fungi | okay, i'll keep that in mind. thanks | 01:53 |
| dims_ | clarkb, added a review (https://review.openstack.org/#/c/65593/) thanks for the pointer | 01:53 |
| clarkb | dims_: thank you | 01:54 |
| fungi | clarkb: looks like maybe https://issues.jenkins-ci.org/browse/JENKINS-19453 but it says remoting 2.33 should be used in 1.542 (jenkins02 is on 1.543) | 01:59 |
| fungi | my eyes still have a tendency to gloss over when i look at java tracebacks. i think it must be a subconscious aversion of some kind | 02:00 |
| clarkb | survival instinct | 02:00 |
| fungi | could be | 02:00 |
| *** tian has joined #openstack-infra | 02:04 | |
| fungi | i'm going to do what i've done with the other for now, which is ssh into the slave, sudo poweroff, connect to the rackspace dashboard, hard reset the vm, then bring it back up in jenkins once it boots and watch the next few jobs it runs | 02:04 |
| *** fifieldt has quit IRC | 02:05 | |
| *** fifieldt has joined #openstack-infra | 02:06 | |
| *** yaguang has joined #openstack-infra | 02:12 | |
| *** fallenpegasus has joined #openstack-infra | 02:12 | |
| *** GheRiver1 has joined #openstack-infra | 02:13 | |
| *** plomakin_ has joined #openstack-infra | 02:14 | |
| *** mikap has joined #openstack-infra | 02:14 | |
| *** dkehn_ has joined #openstack-infra | 02:14 | |
| *** dkehn has quit IRC | 02:15 | |
| *** plomakin has quit IRC | 02:15 | |
| *** GheRivero has quit IRC | 02:15 | |
| *** mika has quit IRC | 02:15 | |
| *** mikap is now known as mika | 02:15 | |
| *** pcrews has quit IRC | 02:15 | |
| *** julim has quit IRC | 02:15 | |
| *** fallenpegasus has quit IRC | 02:17 | |
| *** fallenpegasus has joined #openstack-infra | 02:18 | |
| *** senk has quit IRC | 02:20 | |
| *** llu has left #openstack-infra | 02:20 | |
| *** DennyZhang has joined #openstack-infra | 02:20 | |
| *** jasondotstar has joined #openstack-infra | 02:21 | |
| *** senk has joined #openstack-infra | 02:22 | |
| *** jerryz has quit IRC | 02:23 | |
| *** dkehn_ is now known as dkehn | 02:23 | |
| *** ^d has joined #openstack-infra | 02:24 | |
| *** ^d has joined #openstack-infra | 02:24 | |
| *** ryanpetrello has joined #openstack-infra | 02:28 | |
| *** sandywalsh has quit IRC | 02:31 | |
| *** CaptTofu has quit IRC | 02:31 | |
| harlowja | wow, zuul is on fire, 70 active reviews | 02:34 |
| harlowja | *or jenkins is on fire, one of the above, lol | 02:34 |
| fungi | it's actually catching up. we were well over 100 earlier this morning north-american time | 02:36 |
| fungi | and it's not like people just stop approving things | 02:37 |
| jeblair | fungi: i disabled precise37 | 02:37 |
| *** mriedem has quit IRC | 02:37 | |
| fungi | i just noticed it myselg | 02:38 |
| fungi | myself | 02:38 |
| fungi | same error pattern | 02:38 |
| fungi | really looks to me exactly like the backtraces in https://issues.jenkins-ci.org/browse/JENKINS-19453 | 02:39 |
| fungi | except that we're also seeing it on a master which should have the fix | 02:39 |
| fungi | precise37 was rebooted and started working properly again a mere 4.5 hours ago, so it's either recurrent on the same machines (in which case this is the first one i've seen so soon after) or this is a coincidence that it was 37 again | 02:42 |
| jeblair | fungi: are we sure that's supposed to have the fix? | 02:42 |
| fungi | er, unless i'm misreading | 02:43 |
| fungi | i'm not entirely clear on the relationship between jenkins and the remoting lib | 02:43 |
| *** julim has joined #openstack-infra | 02:43 | |
| jeblair | i'll try to dig into that | 02:43 |
| fungi | but we're seeing it with slaves on jenkins02 as well and it's running a newer rev than the supposed fix-carrying version | 02:44 |
| jeblair | fungi: btw, jenkins02 has some 9h old jobs "running" on centos slaves | 02:44 |
| fungi | here's a supposition... perhaps the agent on the 02 slaves continued running after we updated it? | 02:44 |
| fungi | hadn't spotted the overdue centos jobs. that's unfortunate | 02:45 |
| fungi | ew | 02:45 |
| jeblair | fungi: https://github.com/jenkinsci/jenkins/blob/jenkins-1.543/pom.xml says 1.543 uses remoting 2.33 which bug report says has the fix | 02:45 |
| jeblair | so i guess that statement was correct afaict | 02:46 |
| *** llu has joined #openstack-infra | 02:46 | |
| fungi | got it. and jenkins{-dev,02,03,04} is on 1.543 | 02:46 |
| fungi | jenkins.o.o and 01 are still on 1.525 | 02:47 |
| *** hcc has joined #openstack-infra | 02:47 | |
| *** hcc is now known as hdd_ | 02:47 | |
| jeblair | switched to 2.33 in 1.540 | 02:47 |
| *** fallenpegasus has quit IRC | 02:48 | |
| *** sandywalsh has joined #openstack-infra | 02:48 | |
| *** fallenpegasus has joined #openstack-infra | 02:48 | |
| *** nati_ueno has quit IRC | 02:50 | |
| jeblair | huh | 02:52 |
| jeblair | ./maven/org.jenkins-ci.main/remoting/pom.xml: <tag>remoting-2.28</tag> | 02:52 |
| jeblair | (that's unpacked slave.jar from precise37 | 02:52 |
| fungi | so maybe that was a good guess | 02:52 |
| jeblair | but the timestamp on that file is Jan 9 00:55 | 02:53 |
| fungi | perhaps on master upgrades we should reinstall slave agents? | 02:53 |
| fungi | huh | 02:53 |
| clarkb | That is a plugin iirc maybe core and remoting differ? | 02:53 |
| jeblair | oh but precise37 is jenkins01 which is still old | 02:53 |
| fungi | on, precise37. that one's on jenkins01 which runs 1.525 | 02:53 |
| jeblair | what's a slave on 02 that died? | 02:54 |
| clarkb | 04 | 02:54 |
| fungi | though i've rebooted it, i haven't relaunched the agent on 04 yet | 02:54 |
| jeblair | cool, hold off on that for a bit | 02:54 |
| fungi | definitely | 02:54 |
| jeblair | Jan 9 02:31 slave.jar | 02:55 |
| fungi | huh, so it reinstalled on reboot | 02:55 |
| jeblair | did it reconnect? | 02:55 |
| fungi | according to jenkins02, "Ping response time is too long or timed out." | 02:56 |
| fungi | doesn't seem to be communicating, at any rate | 02:56 |
| jeblair | weird. it's connected and it did install a new slave.jar | 02:57 |
| *** changbl has joined #openstack-infra | 02:58 | |
| *** jasondotstar has quit IRC | 02:59 | |
| *** reed has quit IRC | 02:59 | |
| fifieldt | fungi, if you get a moment in the next couple days, could I get a look at a grep of that log? (my guess is it'll be prepended with "welcome_reviews" or something | 02:59 |
| jeblair | jenkins4 and jenkins6 have remoting 2.33 | 02:59 |
| *** prad has quit IRC | 02:59 | |
| jeblair | jenkins6 looks like it's been online for ~24 hrs | 02:59 |
| fungi | precise6? | 03:00 |
| jeblair | so this is looking like a dead end | 03:00 |
| jeblair | fungi: yes, s/jenkins/precise/ | 03:00 |
| fungi | k | 03:00 |
| fungi | if we could find an evenly-numbered jenkins slave which hasn't been restarted since the jenkins02 upgrade a couple weeks ago, that would be worth looking at | 03:01 |
| *** fallenpegasus has quit IRC | 03:01 | |
| lifeless | clarkb: q: why isn't https://review.openstack.org/#/c/65520/ on status.../zuul ? | 03:02 |
| fungi | fifieldt: getting late over here, but i'll try to check in a bit | 03:02 |
| jeblair | lifeless: "Queue lengths: 238 events, 46 results." | 03:02 |
| jog0 | I think we have a rabbit problem | 03:02 |
| jeblair | lifeless: it's probably in the queue | 03:02 |
| fifieldt | doesn't have to be today | 03:02 |
| fifieldt | just queuing the request :) | 03:02 |
| fungi | jog0: more lettuce | 03:02 |
| *** sarob has joined #openstack-infra | 03:02 | |
| fungi | jog0: s/lettuce/carrots/? | 03:02 |
| lifeless | jeblair: thanks | 03:03 |
| jog0 | fungi: http://logs.openstack.org/86/59686/1/gate/gate-tempest-dsvm-large-ops/f6f933b/logs/screen-n-cond.txt.gz?level=INFO | 03:03 |
| jog0 | we keep connecting to rabbit | 03:03 |
| jog0 | so we must keep getting disconnected too? | 03:04 |
| *** blamar has quit IRC | 03:05 | |
| jog0 | fungi: not sure how to confirm, don't know how to read rabbit logs | 03:05 |
| fungi | No distributions at all found for oslo.messaging>=1.2.0a11 in ./.tox/pep8/lib/python2.7/site-packages (from glance==2014.1.dev105.gd80aa3c) | 03:08 |
| fungi | seen in a glance pep8 job just now | 03:08 |
| fungi | in the gate | 03:08 |
| fungi | nevermind. looks like it was a change which should never have been approved | 03:09 |
| fungi | wow. we have bred a monster | 03:10 |
| fungi | https://review.openstack.org/60821 | 03:10 |
| fungi | some devs will look for *any* excuse to reverify their changes once they're approved | 03:11 |
| jog0 | fungi: whoa that is scray | 03:11 |
| jog0 | scary | 03:11 |
| jog0 | all those reverifies | 03:12 |
| fungi | if this were the only change it'd be one thing | 03:12 |
| jeblair | every now and then we're reminding why we have a gate | 03:13 |
| fungi | i've been playing whack-a-mole with havana patches all day asking devs to stop robo-reverifying them since grenade won't pass until the grizzly devstack exercises thing gets fixes | 03:13 |
| fungi | fixed | 03:13 |
| jeblair | fungi: so i think jenkins02 was restarted a few days ago, and it's likely that they all have that version | 03:14 |
| fungi | they just pick random bugs which have nothing at all to do with the same damn failure (in the case of the havana reverifies) | 03:14 |
| jog0 | fungi: sigh | 03:14 |
| jog0 | we can write a bot to tell us every time a reverify happens and spot checkthem and shame em | 03:14 |
| *** blamar has joined #openstack-infra | 03:14 | |
| jog0 | this is why we can't have nice things | 03:15 |
| fungi | jeblair: okay, that answers my question then. i wondered whether slave agents hanging around from pre-upgrade with old remoting versions explained the bug continuing | 03:15 |
| jeblair | i'm pretty sure jenkins copies it over each time it connects, so it should be all updated | 03:16 |
| fungi | jeblair: so we're left with two remaining likelihoods... 1) that jenkins bug isn't completely fixed or, 2) we have a different bug with the same backtrace | 03:16 |
| jeblair | and looking in the logs, the slave outputs a version number: 2.33, so i'm assuming that's in sync with the remoting lib | 03:16 |
| fungi | k | 03:17 |
| fungi | makes sense | 03:17 |
| fungi | fifieldt: bad news | 03:22 |
| fifieldt | damn | 03:22 |
| fungi | (is better than no news at all?) | 03:22 |
| fungi | [2014-01-09 03:22:08,632] INFO com.google.gerrit.common.ChangeHookRunner : hook[patchset-created] output: timeout: failed to run command `/usr/local/bin/welcome-message': No such file or directory | 03:22 |
| fifieldt | great :D | 03:23 |
| fungi | i can confirm that executable dne | 03:23 |
| fungi | probably missing an entrypoint for it in the jeepyb setup.cfg? | 03:24 |
| *** banix has quit IRC | 03:24 | |
| jeblair | i wonder if we should write a quick daemon that behaves like nodepool and we offline all nodes after completion, and then have this tool disconnect and reconnect the static slaves | 03:24 |
| fifieldt | that'd be it | 03:24 |
| jeblair | i guess the question is would that be faster, or would it be better to rush ahead with all-single-use slaves? | 03:25 |
| *** sarob has quit IRC | 03:25 | |
| fungi | jeblair: i vote for the latter. the pain is likely similar, but it gets us where we want to be sooner | 03:25 |
| *** sarob has joined #openstack-infra | 03:26 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/jeepyb: Add entrypoint for welcome_message https://review.openstack.org/65607 | 03:27 |
| *** fallenpegasus has joined #openstack-infra | 03:28 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/jeepyb: Add entrypoint for welcome_message https://review.openstack.org/65607 | 03:28 |
| fifieldt | there we go | 03:28 |
| fungi | fifieldt: second patchset lgtm! | 03:30 |
| fifieldt | cheers fungi, sorry for the sloppyness | 03:30 |
| fifieldt | I must have assumed jeepyb was just magic :D | 03:30 |
| fungi | fifieldt: no worries, that's an easy bit to forget | 03:30 |
| fungi | fifieldt: technically, the magic there is in pbr | 03:30 |
| fifieldt | :) | 03:31 |
| *** oubiwann has joined #openstack-infra | 03:34 | |
| *** julim has quit IRC | 03:35 | |
| *** sarob has quit IRC | 03:38 | |
| *** sarob has joined #openstack-infra | 03:39 | |
| *** ^d has quit IRC | 03:46 | |
| *** coolsvap has quit IRC | 03:46 | |
| *** yaguang has quit IRC | 03:47 | |
| *** yaguang has joined #openstack-infra | 03:47 | |
| *** yaguang has quit IRC | 03:50 | |
| *** sarob has quit IRC | 03:50 | |
| *** afazekas has joined #openstack-infra | 03:56 | |
| *** banix has joined #openstack-infra | 03:56 | |
| *** fallenpegasus has quit IRC | 03:56 | |
| *** DennyZhang has quit IRC | 03:56 | |
| *** sarob has joined #openstack-infra | 04:02 | |
| *** dcramer_ has quit IRC | 04:04 | |
| *** sarob has quit IRC | 04:07 | |
| *** amotoki has joined #openstack-infra | 04:07 | |
| *** harlowja is now known as harlowja_away | 04:07 | |
| *** vipul has quit IRC | 04:08 | |
| *** vipul has joined #openstack-infra | 04:08 | |
| *** rakhmerov has joined #openstack-infra | 04:11 | |
| *** harlowja_away is now known as harlowja | 04:14 | |
| *** yaguang has joined #openstack-infra | 04:15 | |
| *** cody-somerville has quit IRC | 04:17 | |
| *** fallenpegasus has joined #openstack-infra | 04:20 | |
| *** AaronGr is now known as AaronGr_Zzz | 04:25 | |
| *** praneshp has quit IRC | 04:28 | |
| *** fallenpegasus has quit IRC | 04:28 | |
| *** fallenpegasus has joined #openstack-infra | 04:29 | |
| *** cody-somerville has joined #openstack-infra | 04:31 | |
| *** prad has joined #openstack-infra | 04:32 | |
| *** fallenpegasus has quit IRC | 04:33 | |
| *** dstanek has joined #openstack-infra | 04:42 | |
| *** vogxn has joined #openstack-infra | 04:43 | |
| *** vogxn has quit IRC | 04:43 | |
| *** banix has quit IRC | 04:45 | |
| *** wenlock has joined #openstack-infra | 04:45 | |
| fungi | okay, knocking off early for the night... back in 7 or 8 hours | 04:46 |
| *** nosnos_ has joined #openstack-infra | 04:47 | |
| *** branen_ has joined #openstack-infra | 04:48 | |
| *** cody-somerville_ has joined #openstack-infra | 04:49 | |
| *** fifieldt_ has joined #openstack-infra | 04:49 | |
| *** sdake_ has joined #openstack-infra | 04:49 | |
| *** banix has joined #openstack-infra | 04:50 | |
| *** vipuls has joined #openstack-infra | 04:50 | |
| *** dstanek has quit IRC | 04:55 | |
| *** cody-somerville has quit IRC | 04:55 | |
| *** vipul has quit IRC | 04:55 | |
| *** changbl has quit IRC | 04:55 | |
| *** sandywalsh has quit IRC | 04:55 | |
| *** fifieldt has quit IRC | 04:55 | |
| *** nosnos has quit IRC | 04:55 | |
| *** krtaylor has quit IRC | 04:55 | |
| *** branen has quit IRC | 04:55 | |
| *** sdake has quit IRC | 04:55 | |
| *** sandywalsh has joined #openstack-infra | 04:57 | |
| *** changbl has joined #openstack-infra | 04:57 | |
| *** praneshp has joined #openstack-infra | 04:58 | |
| *** fallenpegasus has joined #openstack-infra | 05:03 | |
| *** krtaylor has joined #openstack-infra | 05:03 | |
| *** fallenpegasus has quit IRC | 05:03 | |
| *** fallenpegasus has joined #openstack-infra | 05:04 | |
| *** ryanpetrello has quit IRC | 05:05 | |
| *** fallenpegasus has quit IRC | 05:05 | |
| *** banix has quit IRC | 05:06 | |
| *** morganfainberg has quit IRC | 05:07 | |
| *** noorul has left #openstack-infra | 05:09 | |
| *** fallenpegasus has joined #openstack-infra | 05:10 | |
| clarkb | fungi: good night | 05:10 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Create more bare-precise nodes https://review.openstack.org/65619 | 05:13 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Allow nova to use bare-precise nodes https://review.openstack.org/65620 | 05:15 |
| openstackgerrit | A change was merged to openstack-infra/config: Use performance rax nodes in the gate https://review.openstack.org/65236 | 05:18 |
| *** chandankumar has joined #openstack-infra | 05:27 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Move gear import to a safe place https://review.openstack.org/65561 | 05:29 |
| *** praneshp_ has joined #openstack-infra | 05:32 | |
| *** praneshp has quit IRC | 05:35 | |
| *** praneshp_ is now known as praneshp | 05:35 | |
| *** prad has quit IRC | 05:36 | |
| *** fallenpegasus has quit IRC | 05:38 | |
| *** chandankumar_ has joined #openstack-infra | 05:39 | |
| *** gyee_ has quit IRC | 05:39 | |
| *** dstanek has joined #openstack-infra | 05:40 | |
| *** morganfainberg has joined #openstack-infra | 05:41 | |
| *** nati_ueno has joined #openstack-infra | 05:43 | |
| *** dstanek has quit IRC | 05:47 | |
| *** coolsvap has joined #openstack-infra | 05:53 | |
| *** dstanek has joined #openstack-infra | 05:56 | |
| *** fallenpegasus has joined #openstack-infra | 06:05 | |
| *** wenlock has quit IRC | 06:06 | |
| *** Ryan_Lane has joined #openstack-infra | 06:07 | |
| *** dstanek has quit IRC | 06:08 | |
| *** jamielennox is now known as jamielennox|away | 06:13 | |
| clarkb | pleia2: anteaya: were you guys having a day off? | 06:13 |
| StevenK | I did wonder that myself. | 06:15 |
| *** pelix has left #openstack-infra | 06:15 | |
| *** hdd_ has quit IRC | 06:21 | |
| *** talluri has joined #openstack-infra | 06:23 | |
| *** chandankumar has quit IRC | 06:25 | |
| *** harlowja is now known as harlowja_away | 06:25 | |
| pleia2 | clarkb: yeah, I seem to have a cold :( decided to take it easy | 06:27 |
| clarkb | get better | 06:27 |
| pleia2 | we're meeting in the lobby soon to play with some testing stuff though | 06:27 |
| clarkb | I feel like i am coming down with a cold too, going to skip the dinner thing tonight and try t oget a decent amount of sleep | 06:28 |
| pleia2 | good idea | 06:28 |
| openstackgerrit | A change was merged to openstack-infra/publications: Update sysadmin-codereview from Oct presentation https://review.openstack.org/64292 | 06:28 |
| pleia2 | \o/ | 06:29 |
| pleia2 | if someone feels like tagging that, I can get my latest changes in too: git tag -s -m "Bay Area LUG, 2013" 2013-balug-sysadmin-codereview | 06:31 |
| jeblair | pleia2: to confirm, i'll tag the commit that just merged with that ^ ? | 06:32 |
| pleia2 | jeblair: correct, october was my balug talk | 06:32 |
| *** fallenpegasus has quit IRC | 06:35 | |
| jeblair | pleia2: pushed | 06:35 |
| pleia2 | jeblair: thanks! | 06:35 |
| jeblair | no prob! | 06:36 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove devstack-precise-check rax images https://review.openstack.org/65237 | 06:36 |
| openstackgerrit | A change was merged to openstack-infra/config: Increase hpcloud ssh timeout to 180 https://review.openstack.org/65246 | 06:37 |
| openstackgerrit | A change was merged to openstack-infra/config: Create more bare-precise nodes https://review.openstack.org/65619 | 06:37 |
| *** dkliban has quit IRC | 06:43 | |
| *** pblaho has joined #openstack-infra | 06:48 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/publications: Add a couple services we manage and checks https://review.openstack.org/65625 | 06:48 |
| *** oubiwann has quit IRC | 06:53 | |
| *** morganfainberg has quit IRC | 07:00 | |
| *** fallenpegasus has joined #openstack-infra | 07:00 | |
| *** morganfainberg has joined #openstack-infra | 07:01 | |
| *** fallenpegasus2 has joined #openstack-infra | 07:06 | |
| *** fallenpegasus has quit IRC | 07:06 | |
| *** dstufft_ has joined #openstack-infra | 07:09 | |
| *** dstufft has quit IRC | 07:10 | |
| *** fallenpegasus2 has quit IRC | 07:10 | |
| *** fallenpegasus has joined #openstack-infra | 07:11 | |
| *** ilyashakhat_ has quit IRC | 07:12 | |
| *** morganfainberg has quit IRC | 07:13 | |
| *** morganfainberg has joined #openstack-infra | 07:14 | |
| *** mozawa has joined #openstack-infra | 07:14 | |
| *** bknudson has quit IRC | 07:15 | |
| *** zhiyan has joined #openstack-infra | 07:15 | |
| *** zhiyan has left #openstack-infra | 07:16 | |
| *** bknudson has joined #openstack-infra | 07:16 | |
| mozawa | Hi, can someone please restore this one ? https://review.openstack.org/#/c/63708/ | 07:18 |
| mozawa | It's not owned by me but by Jenkins. So I can not restore it. | 07:18 |
| *** SergeyLukjanov has joined #openstack-infra | 07:18 | |
| *** obondarev has joined #openstack-infra | 07:19 | |
| mozawa | Due to this issue is in abandoned state, I got the following error in git review (dependency error). | 07:19 |
| mozawa | [mozawa@mubuntu unit (bp/s3-multi-part-upload)]$ git review | 07:19 |
| mozawa | You have more than one commit that you are about to submit. | 07:19 |
| mozawa | The outstanding commits are: | 07:19 |
| mozawa | 56be241 (HEAD, bp/s3-multi-part-upload) Implemented S3 multi-part upload functionality | 07:19 |
| mozawa | 6dace29 Updated from global requirements | 07:19 |
| mozawa | Is this really what you meant to do? | 07:19 |
| mozawa | Type 'yes' to confirm: yes | 07:19 |
| mozawa | remote: Resolving deltas: 100% (13/13) | 07:19 |
| mozawa | remote: Processing changes: refs: 1, done | 07:19 |
| mozawa | To ssh://mozawa@review.openstack.org:29418/openstack/glance.git | 07:19 |
| mozawa | ! [remote rejected] HEAD -> refs/for/master/bp/s3-multi-part-upload (change 63708 closed) | 07:19 |
| mozawa | error: failed to push some refs to 'ssh://mozawa@review.openstack.org:29418/openstack/glance.git' | 07:19 |
| mozawa | [mozawa@mubuntu unit (bp/s3-multi-part-upload)]$ | 07:19 |
| *** fallenpegasus has quit IRC | 07:20 | |
| *** NikitaKonovalov has joined #openstack-infra | 07:21 | |
| *** jamielennox|away is now known as jamielennox | 07:23 | |
| *** yolanda has joined #openstack-infra | 07:23 | |
| *** yolanda has quit IRC | 07:26 | |
| *** yolanda has joined #openstack-infra | 07:31 | |
| *** fallenpegasus has joined #openstack-infra | 07:32 | |
| *** coolsvap has quit IRC | 07:33 | |
| *** katyafervent has joined #openstack-infra | 07:33 | |
| *** skraynev has joined #openstack-infra | 07:41 | |
| *** coolsvap has joined #openstack-infra | 07:44 | |
| *** andreaf has joined #openstack-infra | 07:45 | |
| *** SergeyLukjanov has quit IRC | 07:47 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:48 | |
| *** talluri_ has joined #openstack-infra | 07:49 | |
| *** jcoufal has joined #openstack-infra | 07:50 | |
| *** talluri has quit IRC | 07:51 | |
| *** fallenpegasus has quit IRC | 08:01 | |
| *** dpyzhov has joined #openstack-infra | 08:09 | |
| *** pblaho has quit IRC | 08:11 | |
| *** talluri has joined #openstack-infra | 08:18 | |
| *** talluri_ has quit IRC | 08:21 | |
| *** fallenpegasus has joined #openstack-infra | 08:27 | |
| *** fallenpegasus has quit IRC | 08:30 | |
| *** fallenpegasus has joined #openstack-infra | 08:31 | |
| ttx | when cell data has less packet loss than your ISP connection, things are not going well | 08:33 |
| *** fbo_away is now known as fbo | 08:33 | |
| *** senk has quit IRC | 08:36 | |
| *** fallenpegasus has quit IRC | 08:37 | |
| *** mancdaz_away is now known as mancdaz | 08:38 | |
| *** mancdaz is now known as mancdaz_away | 08:39 | |
| *** senk has joined #openstack-infra | 08:40 | |
| *** mancdaz_away is now known as mancdaz | 08:40 | |
| *** flaper87|afk is now known as flaper87 | 08:43 | |
| *** SergeyLukjanov has quit IRC | 08:44 | |
| *** nati_ueno has quit IRC | 08:47 | |
| *** rossella_s has joined #openstack-infra | 08:48 | |
| *** jpich has joined #openstack-infra | 08:49 | |
| *** Ryan_Lane has quit IRC | 08:50 | |
| *** fallenpegasus has joined #openstack-infra | 08:59 | |
| *** _ruhe is now known as ruhe | 09:04 | |
| *** saschpe has quit IRC | 09:05 | |
| *** che-arne has quit IRC | 09:07 | |
| *** saschpe has joined #openstack-infra | 09:13 | |
| *** dizquierdo has joined #openstack-infra | 09:15 | |
| *** yassine has joined #openstack-infra | 09:15 | |
| *** jooools has joined #openstack-infra | 09:18 | |
| *** jroovers has joined #openstack-infra | 09:25 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:25 | |
| *** yassine has quit IRC | 09:25 | |
| *** yassine has joined #openstack-infra | 09:25 | |
| anteaya | ttx nice | 09:26 |
| *** johnthetubaguy has joined #openstack-infra | 09:30 | |
| *** fallenpegasus has quit IRC | 09:32 | |
| *** fallenpegasus has joined #openstack-infra | 09:35 | |
| *** johnthetubaguy has quit IRC | 09:38 | |
| *** johnthetubaguy1 has joined #openstack-infra | 09:38 | |
| *** saschpe has quit IRC | 09:39 | |
| *** pblaho has joined #openstack-infra | 09:40 | |
| *** yassine has quit IRC | 09:40 | |
| *** yassine has joined #openstack-infra | 09:40 | |
| *** fallenpegasus has quit IRC | 09:43 | |
| *** fallenpegasus has joined #openstack-infra | 09:43 | |
| *** coolsvap has quit IRC | 09:46 | |
| *** yassine has quit IRC | 09:46 | |
| *** yassine has joined #openstack-infra | 09:46 | |
| *** praneshp has quit IRC | 09:47 | |
| *** ilyashakhat has joined #openstack-infra | 09:47 | |
| *** derekh has joined #openstack-infra | 09:52 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files https://review.openstack.org/48783 | 09:53 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Add tests for YamlParser and patch 2.6 minidom https://review.openstack.org/63579 | 09:53 |
| anteaya | when someone who can access the gerrit db is awake and able, can you check account id # 9832 username mayu and let me know what the db has as ssh keys? | 09:57 |
| anteaya | I am looking to confirm this public key: http://paste.openstack.org/show/60864/ for that account | 09:57 |
| *** nati_ueno has joined #openstack-infra | 09:58 | |
| *** jasondotstar has joined #openstack-infra | 10:00 | |
| *** plomakin_ has quit IRC | 10:01 | |
| *** plomakin has joined #openstack-infra | 10:01 | |
| *** nati_ueno has quit IRC | 10:02 | |
| *** mozawa has quit IRC | 10:02 | |
| *** kruskakli has quit IRC | 10:08 | |
| *** coolsvap has joined #openstack-infra | 10:10 | |
| clarkb | anteaya: can the user confirm it themselves? | 10:12 |
| ttx | sdague: did you propose a date for the gate bugs day ? | 10:17 |
| * ttx is slightly out of touch with crappy Internet connection | 10:18 | |
| ttx | tethering on cell data right now | 10:18 |
| * ttx hugs his cell data provider for not filtering anything | 10:18 | |
| *** fallenpegasus has quit IRC | 10:22 | |
| *** fallenpegasus2 has joined #openstack-infra | 10:22 | |
| *** SergeyLukjanov has quit IRC | 10:30 | |
| *** fallenpegasus2 has quit IRC | 10:35 | |
| *** fallenpegasus has joined #openstack-infra | 10:35 | |
| *** boris-42 has quit IRC | 10:39 | |
| *** ruhe is now known as _ruhe | 10:42 | |
| *** fallenpegasus has quit IRC | 10:44 | |
| *** jpeeler has quit IRC | 10:44 | |
| *** nosnos_ has quit IRC | 10:46 | |
| *** saschpe has joined #openstack-infra | 10:49 | |
| *** boris-42 has joined #openstack-infra | 10:50 | |
| *** yassine has quit IRC | 10:51 | |
| *** saschpe has quit IRC | 10:57 | |
| *** hashar has joined #openstack-infra | 11:01 | |
| *** rakhmerov has quit IRC | 11:05 | |
| *** _ruhe is now known as ruhe | 11:06 | |
| *** jcoufal has quit IRC | 11:14 | |
| *** jcoufal has joined #openstack-infra | 11:15 | |
| *** markmc has joined #openstack-infra | 11:16 | |
| *** kruskakli has joined #openstack-infra | 11:17 | |
| *** yaguang has quit IRC | 11:18 | |
| *** ruhe is now known as _ruhe | 11:18 | |
| *** _ruhe is now known as ruhe | 11:23 | |
| *** pasquier-s has joined #openstack-infra | 11:25 | |
| sdague | ttx: I have not yet, I was going to run it by the ptls on friday. Honestly, I was trying to get through some of this analysis first | 11:28 |
| *** boris-42 has quit IRC | 11:29 | |
| ttx | sdague: ack. Was about to suggest it as a response to jog0's call for help, but figured I should ask you first | 11:30 |
| *** pblaho has quit IRC | 11:34 | |
| *** che-arne has joined #openstack-infra | 11:37 | |
| *** boris-42 has joined #openstack-infra | 11:38 | |
| *** michchap_ has joined #openstack-infra | 11:41 | |
| *** michchap has quit IRC | 11:41 | |
| *** dstanek has joined #openstack-infra | 11:42 | |
| *** yamahata has quit IRC | 11:43 | |
| *** pblaho has joined #openstack-infra | 11:46 | |
| *** pblaho has quit IRC | 11:46 | |
| *** pblaho has joined #openstack-infra | 11:46 | |
| sdague | yeh, honestly, I'd like that thread to live on it's own. | 11:47 |
| *** dstanek has quit IRC | 11:48 | |
| *** rakhmerov has joined #openstack-infra | 11:51 | |
| *** rakhmerov1 has joined #openstack-infra | 11:52 | |
| *** rakhmerov has quit IRC | 11:52 | |
| *** cody-somerville_ has quit IRC | 11:52 | |
| *** rfolco has joined #openstack-infra | 11:57 | |
| *** weshay has joined #openstack-infra | 11:57 | |
| *** rfolco_ has joined #openstack-infra | 11:58 | |
| *** coolsvap has quit IRC | 12:02 | |
| *** tma996 has joined #openstack-infra | 12:03 | |
| *** ruhe is now known as ruhe_away | 12:05 | |
| *** cody-somerville_ has joined #openstack-infra | 12:06 | |
| *** ruhe_away is now known as ruhe | 12:08 | |
| *** dkranz has joined #openstack-infra | 12:09 | |
| *** jooools has quit IRC | 12:09 | |
| *** cody-somerville_ is now known as cody-somerville | 12:10 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Add jenkins slave name to the logstash records https://review.openstack.org/65593 | 12:10 |
| *** dims_ is now known as dims | 12:11 | |
| *** fifieldt_ has quit IRC | 12:13 | |
| *** dstufft_ is now known as dstufft | 12:15 | |
| *** pblaho has quit IRC | 12:15 | |
| *** derekh has quit IRC | 12:23 | |
| *** ruhe is now known as _ruhe | 12:23 | |
| *** rfolco_ has quit IRC | 12:23 | |
| *** hashar has quit IRC | 12:26 | |
| *** saschpe has joined #openstack-infra | 12:29 | |
| *** yassine has joined #openstack-infra | 12:32 | |
| *** dstanek has joined #openstack-infra | 12:35 | |
| *** crank has quit IRC | 12:35 | |
| *** crank has joined #openstack-infra | 12:36 | |
| *** smarcet has joined #openstack-infra | 12:38 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 12:40 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 12:41 |
| *** jroovers has quit IRC | 12:43 | |
| *** dstanek has quit IRC | 12:47 | |
| *** _ruhe is now known as ruhe | 12:50 | |
| *** fallenpegasus has joined #openstack-infra | 12:53 | |
| *** heyongli has joined #openstack-infra | 12:54 | |
| *** dkranz has quit IRC | 12:54 | |
| *** mozawa has joined #openstack-infra | 12:55 | |
| *** NikitaKonovalov has quit IRC | 12:57 | |
| *** derekh has joined #openstack-infra | 12:57 | |
| *** oubiwann has joined #openstack-infra | 12:58 | |
| *** pblaho has joined #openstack-infra | 12:59 | |
| fungi | mozawa: i've restored 63708 as you requested | 12:59 |
| *** johnthetubaguy1 is now known as johnthetubaguy | 13:00 | |
| fungi | anteaya: checking into mayu's account now | 13:00 |
| *** dkranz has joined #openstack-infra | 13:01 | |
| *** NikitaKonovalov has joined #openstack-infra | 13:02 | |
| fungi | anteaya: it looks like mayu pasted their ssh key into gerrit the same way it's pasted into that paste... incorrectly | 13:03 |
| fungi | anteaya: it should not have embedded newlines, but rather be all one line with no breaks | 13:04 |
| *** pblaho has left #openstack-infra | 13:04 | |
| *** pblaho has joined #openstack-infra | 13:04 | |
| mozawa | fungi: Thank you very much ! | 13:06 |
| fungi | looks like precise26 has gone rogue. marking offline | 13:06 |
| *** dkranz has quit IRC | 13:08 | |
| *** jcoufal has quit IRC | 13:11 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/config: Notify openstack-operators on UpgradeImpact https://review.openstack.org/65685 | 13:14 |
| *** rakhmerov1 has quit IRC | 13:15 | |
| *** sandywalsh has quit IRC | 13:16 | |
| *** oubiwann has quit IRC | 13:17 | |
| *** jroovers has joined #openstack-infra | 13:18 | |
| *** jroovers has quit IRC | 13:18 | |
| *** jcoufal has joined #openstack-infra | 13:18 | |
| *** jroovers has joined #openstack-infra | 13:18 | |
| *** oubiwann has joined #openstack-infra | 13:21 | |
| *** alexpilotti has joined #openstack-infra | 13:25 | |
| *** prad has joined #openstack-infra | 13:25 | |
| *** thomasem has joined #openstack-infra | 13:26 | |
| *** jcoufal_ has joined #openstack-infra | 13:28 | |
| *** jcoufal has quit IRC | 13:28 | |
| ruhe | i'm trying to setup zuul sever. and constantly get the following error on attempt to enqueue a job: | 13:28 |
| ruhe | Job <gear.Job 0x1f4df50 handle: None name: build:job_name unique: feb7436652964357bfeb1d1db8443478> is not registered with Gearman | 13:28 |
| ruhe | is it jenkins who has to register the job with gearman? | 13:28 |
| fungi | ruhe: yes, or more specifically, the jenkins-gearman plugin installed and configured on the jenkins master | 13:28 |
| fungi | that plugin connects to your gearman server (usually the one provided by the gear library as a child process of the zuul daemon) to register the jobs | 13:29 |
| ruhe | fungi: does it happen by itself when i create a new job? (assuming plugin is installed and enabled) | 13:30 |
| fungi | ruhe: it should, yes | 13:30 |
| fungi | assuming correct configuration and connectivity | 13:30 |
| ruhe | fungi: thank you. now i now where to dig | 13:30 |
| fungi | you're welcome. there's also an admin protocol for gearman, so if you want to debug there you can connect to the gearman service and interrogate it | 13:31 |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Added rbenv-env wrapper https://review.openstack.org/65352 | 13:31 |
| *** sandywalsh has joined #openstack-infra | 13:31 | |
| fungi | ruhe: with gearman admin commands you can do things like see connected workers and registered jobs | 13:31 |
| *** jasondotstar has quit IRC | 13:38 | |
| *** prad has quit IRC | 13:38 | |
| *** jasondotstar has joined #openstack-infra | 13:39 | |
| *** CaptTofu has joined #openstack-infra | 13:40 | |
| *** wenlock has joined #openstack-infra | 13:46 | |
| *** yamahata has joined #openstack-infra | 13:49 | |
| *** yamahata has quit IRC | 13:50 | |
| *** yamahata has joined #openstack-infra | 13:51 | |
| openstackgerrit | A change was merged to openstack-infra/config: Allow nova to use bare-precise nodes https://review.openstack.org/65620 | 13:56 |
| *** oubiwann has quit IRC | 13:57 | |
| *** mriedem has joined #openstack-infra | 14:00 | |
| *** dprince has joined #openstack-infra | 14:00 | |
| fungi | i'm keeping an eye on nova jobs and will revert that ^ if it starts to cause trouble | 14:01 |
| *** med_ has quit IRC | 14:01 | |
| *** jpeeler has joined #openstack-infra | 14:05 | |
| *** jpeeler has quit IRC | 14:05 | |
| *** jpeeler has joined #openstack-infra | 14:05 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: use pandas DataFrames for new check_success_pd https://review.openstack.org/65173 | 14:05 |
| *** talluri has quit IRC | 14:08 | |
| *** thuc has joined #openstack-infra | 14:08 | |
| *** amotoki has quit IRC | 14:09 | |
| *** talluri has joined #openstack-infra | 14:09 | |
| *** luqas has joined #openstack-infra | 14:10 | |
| *** heyongli has quit IRC | 14:11 | |
| *** coolsvap has joined #openstack-infra | 14:11 | |
| *** julim has joined #openstack-infra | 14:13 | |
| *** talluri has quit IRC | 14:13 | |
| fungi | i've taken precise34 offline now too, same problem | 14:14 |
| *** yaguang has joined #openstack-infra | 14:17 | |
| *** rakhmerov has joined #openstack-infra | 14:20 | |
| *** prad has joined #openstack-infra | 14:22 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:23 | |
| SergeyLukjanov | hey folks | 14:24 |
| SergeyLukjanov | are there any bugs for specifying them if jenkins slaves problems? | 14:25 |
| SergeyLukjanov | especially in case when this slave was already disabled | 14:26 |
| fungi | SergeyLukjanov: we think it may be https://issues.jenkins-ci.org/browse/JENKINS-19453 but some of the failing slaves are on jenkins02 which is supposed to have that fixed, so we're not yet sure | 14:26 |
| *** michchap_ has quit IRC | 14:26 | |
| *** michchap has joined #openstack-infra | 14:27 | |
| SergeyLukjanov | I see another error on precise34 | 14:27 |
| SergeyLukjanov | http://logs.openstack.org/98/65598/1/gate/gate-python-savannaclient-pep8/e036655/console.html | 14:27 |
| SergeyLukjanov | it was already disabled by you too | 14:27 |
| fungi | SergeyLukjanov: however, we've merged https://review.openstack.org/65620 as the latest tack in our progress toward not reusing any general-purpose slaves any longer, and i'm seeing what looks like success on the nova changes in the gate so far | 14:27 |
| fungi | so maybe this will be mostly behind us within the next day or so | 14:28 |
| fungi | python 2.6 and 3.3 unit tests are going to be a bit more work to knock out though | 14:28 |
| fungi | since we'll need additional images for them, and possibly additional scripting | 14:29 |
| SergeyLukjanov | in savanna I see mostly pep8 and py27 failed | 14:29 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add pandas infrastruce into elastic recheck https://review.openstack.org/65695 | 14:29 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: use pandas DataFrames for new check_success_pd https://review.openstack.org/65173 | 14:29 |
| fungi | SergeyLukjanov: yeah, we run a lot more tests on regular ubuntu precise, while the centos6 and precisepy3k nodes are pretty much only used for python 2.6 and 3.3, so we have many more regular precise slaves as a result and i so expect to see them fail (and have) with greater frequency | 14:30 |
| SergeyLukjanov | fungi, which bug is better to use to verify CRs failed on precise34? | 14:31 |
| fungi | SergeyLukjanov: there have been a few one-off openstack-ci bugs opened for the individual slaves which failed, but they're starting to happen with such regularity now that i'm not sure continuing to open new bugs for each of them is worthwhile this week. maybe just take over one of those and mark the others as a dupe | 14:32 |
| *** ryanpetrello has joined #openstack-infra | 14:32 | |
| SergeyLukjanov | fungi, that's why I'm asking | 14:32 |
| SergeyLukjanov | I'll take a look on them | 14:33 |
| *** prad has quit IRC | 14:33 | |
| fungi | the uptick in slave agent failures seems to correspond with the rise in test load, but also possibly even more sharply with the zuul upgrade. i suspect as we made zuul more efficient, it has put further strain on the jenkins masters | 14:33 |
| *** rakhmerov has quit IRC | 14:35 | |
| luqas | Hi guys, I'm seeking for a service account for 3rd party testing | 14:35 |
| luqas | i've written to open-infra mail list but got no answer so far | 14:35 |
| fungi | luqas: i was just in the process of handling that one, actually | 14:36 |
| luqas | oh perfect | 14:36 |
| fungi | luqas: i'll reply to your message here shortly | 14:36 |
| *** mrodden has joined #openstack-infra | 14:36 | |
| fungi | well, i'll reply to your message *there* shortly (i've already replied here) ;) | 14:36 |
| luqas | thanks a lot, we'll maybe changing the contact e-mail if possible | 14:36 |
| fungi | luqas: i can change it for you later if needed, but it's not something you'll be able to change directly yourself since it needs to happen in the database | 14:37 |
| SergeyLukjanov | fungi, yup, it looks like we can have extremely bigger load on jenkins masters in case if zuul will process events faster | 14:37 |
| luqas | fungi, ok, it's ok right now | 14:37 |
| *** markmcclain has quit IRC | 14:39 | |
| *** dkliban has joined #openstack-infra | 14:39 | |
| SergeyLukjanov | fungi, I see only one open bug about jenkins agent init failed | 14:41 |
| SergeyLukjanov | that was creted by me for precise37 | 14:41 |
| SergeyLukjanov | https://bugs.launchpad.net/openstack-ci/+bug/1267364 | 14:41 |
| *** ilyashakhat has quit IRC | 14:41 | |
| *** eharney has joined #openstack-infra | 14:41 | |
| fungi | SergeyLukjanov: okay, that works. we'd previously been marking them fix-released shortly after disabling/restoring the affected slave | 14:42 |
| fungi | which is probably why you don't see others with a normal search | 14:42 |
| SergeyLukjanov | fungi, yup, I mean that all other was already closed | 14:42 |
| SergeyLukjanov | fungi, I think that it'll be better to rename it to "Jenkins agent init failed" and use it for recheck/reverify | 14:42 |
| SergeyLukjanov | and probably track failed slaves list in description | 14:43 |
| SergeyLukjanov | to not create new bugs each time | 14:43 |
| *** GheRiver1 is now known as GheRivero | 14:43 | |
| *** thuc has quit IRC | 14:43 | |
| fungi | maybe even "recurrent jenkins slave agent failures" | 14:43 |
| *** thuc has joined #openstack-infra | 14:44 | |
| SergeyLukjanov | fungi, ok, I'll rename it, update description and send a bug it to os-dev to make guys able to recheck/reverify failed CRs | 14:45 |
| fungi | SergeyLukjanov: thanks! that's a hude help | 14:46 |
| fungi | er, huge too | 14:46 |
| fungi | SergeyLukjanov: in positive news, these nova jobs, which previously would have been run on long-running reusable slaves, ran successfully on nodepool bare nodes: http://paste.openstack.org/show/60881/ | 14:46 |
| fungi | so i think i may try turning it on for a few more projects here shortly | 14:47 |
| *** thuc has quit IRC | 14:48 | |
| *** CaptTofu has quit IRC | 14:48 | |
| SergeyLukjanov | fungi, oh, that's cool | 14:48 |
| SergeyLukjanov | bare nodes are single-use? | 14:49 |
| fungi | SergeyLukjanov: for now at least, all nodepool-managed nodes are single-use | 14:49 |
| *** dcramer_ has joined #openstack-infra | 14:50 | |
| SergeyLukjanov | fungi, oh, that's surprinsing ;) | 14:50 |
| SergeyLukjanov | I was thinking that long-running nodes are creating but nodepool | 14:50 |
| fungi | we may decide to instead set an upper-bound on reuse for them so we don't incur as much build/beat/burn overhead, but we'll approach that once we see it's an actual problem | 14:51 |
| SergeyLukjanov | btw we're now using nodepool to build non single-use nodes for savanna-ci | 14:51 |
| fungi | SergeyLukjanov: awesome! | 14:51 |
| SergeyLukjanov | fungi, zuul/nodepool/gearman on trunk OpenStack with Neutron are used now | 14:52 |
| fungi | oh wow, good work | 14:52 |
| *** banix has joined #openstack-infra | 14:53 | |
| SergeyLukjanov | there are very small changes to support Neutron | 14:53 |
| SergeyLukjanov | like adding network id AFAIK | 14:53 |
| SergeyLukjanov | we'll return back with patches when it'll be fully checked | 14:53 |
| *** malini_afk is now known as malini | 14:54 | |
| fungi | so just to confirm, you have a continuously-deployed openstack-neutron cloud, and you have nodepool adding/deleting slaves within it? definitely appreciate the patches | 14:55 |
| *** oubiwann has joined #openstack-infra | 14:56 | |
| fungi | that'll put us ahead of the curve for when our cloud providers catch up to that | 14:57 |
| *** marun has quit IRC | 14:57 | |
| *** thuc has joined #openstack-infra | 14:57 | |
| *** prad has joined #openstack-infra | 14:57 | |
| *** herndon_ has joined #openstack-infra | 14:58 | |
| *** CaptTofu has joined #openstack-infra | 14:58 | |
| *** rakhmerov has joined #openstack-infra | 14:59 | |
| *** rakhmerov has quit IRC | 14:59 | |
| ruhe | fungi: what do you mean by "continuously-deployed" ? our (savanna-ci) cloud is based on stable/havana with neutron | 15:00 |
| *** oubiwann has quit IRC | 15:00 | |
| fungi | ruhe: aha, i misread. when SergeyLukjanov said "on trunk" he meant trunk zuul, nodepool... not trunk openstack | 15:01 |
| SergeyLukjanov | fungi, yup, we have an OpenStack cluster with Neutron manually installed (using devstack) | 15:01 |
| SergeyLukjanov | fungi, and nodepool creates slaves in this cluster | 15:01 |
| fungi | still sounds good, and definitely excited to get patches to support that | 15:02 |
| *** burt has joined #openstack-infra | 15:02 | |
| *** thuc has quit IRC | 15:02 | |
| *** CaptTofu has quit IRC | 15:02 | |
| *** markmcclain has joined #openstack-infra | 15:02 | |
| *** CaptTofu has joined #openstack-infra | 15:02 | |
| *** thuc has joined #openstack-infra | 15:02 | |
| ruhe | fungi: we're also working towards deploying it all with openstack-infra/config puppet scripts (that's where i've got stuck with zuul not being able to find jobs) | 15:03 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: make a dedicated page for gate status https://review.openstack.org/65700 | 15:04 |
| fungi | ruhe: makes sense. if you find the puppetry or documentation is missing something significant, please open bugs and/or submit patches for that. we're always looking to improve and make it easier | 15:04 |
| ruhe | sure | 15:05 |
| SergeyLukjanov | fungi, I hope that we'll eventually adopt os-infra/config to install all our savanna-ci related infra | 15:06 |
| *** rcarrillocruz1 has joined #openstack-infra | 15:07 | |
| fungi | SergeyLukjanov: i hope so too! | 15:07 |
| *** oubiwann has joined #openstack-infra | 15:07 | |
| *** thuc has quit IRC | 15:07 | |
| SergeyLukjanov | fungi, here is the bug for agent failures https://bugs.launchpad.net/openstack-ci/+bug/1267364 | 15:07 |
| *** CaptTofu has quit IRC | 15:07 | |
| SergeyLukjanov | fungi, could you please take a look on it and I'll send an email to dev | 15:07 |
| *** rcarrillocruz has quit IRC | 15:07 | |
| fungi | SergeyLukjanov: definitely, having a look now | 15:07 |
| SergeyLukjanov | fungi, thx | 15:08 |
| *** marun has joined #openstack-infra | 15:12 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add in the optional ; everywhere https://review.openstack.org/65703 | 15:15 |
| sdague | fungi: can you give https://review.openstack.org/65700 a look. That will let me drop that piece off the ER webpage | 15:15 |
| fungi | SergeyLukjanov: okay, bug looks good. i added some additional status detail on what we're doing to solve it | 15:16 |
| fungi | sdague: sure | 15:16 |
| SergeyLukjanov | fungi, thank you, so, I'll send it to the -dev | 15:16 |
| *** senk has quit IRC | 15:17 | |
| *** CaptTofu has joined #openstack-infra | 15:19 | |
| *** senk has joined #openstack-infra | 15:22 | |
| *** krotscheck has joined #openstack-infra | 15:23 | |
| *** rwsu has joined #openstack-infra | 15:23 | |
| *** dims has quit IRC | 15:23 | |
| *** dims has joined #openstack-infra | 15:24 | |
| *** senk1 has joined #openstack-infra | 15:25 | |
| *** senk has quit IRC | 15:26 | |
| *** dkranz has joined #openstack-infra | 15:27 | |
| *** kraman has joined #openstack-infra | 15:27 | |
| *** mfink has joined #openstack-infra | 15:27 | |
| *** mfer has joined #openstack-infra | 15:29 | |
| *** rakhmerov has joined #openstack-infra | 15:30 | |
| openstackgerrit | Cyril Roelandt proposed a change to openstack-infra/config: pycadf: enable python33 in the gates https://review.openstack.org/65706 | 15:30 |
| *** jorisroovers has joined #openstack-infra | 15:34 | |
| shardy | Hi all, seeing this failure which seems to be a network outage or something: | 15:34 |
| shardy | http://logs.openstack.org/03/65203/3/gate/gate-heat-python27/c5f8207/console.html | 15:34 |
| shardy | Is it safe to reverify no bug, or should I raise one? | 15:35 |
| *** rnirmal has joined #openstack-infra | 15:35 | |
| *** rakhmerov has quit IRC | 15:35 | |
| *** mfink has left #openstack-infra | 15:36 | |
| *** jroovers has quit IRC | 15:37 | |
| *** rakhmerov has joined #openstack-infra | 15:38 | |
| fungi | shardy: reverify bug 1267364 | 15:39 |
| shardy | fungi: Thanks, hadn't spotted that one | 15:40 |
| *** dpyzhov has quit IRC | 15:40 | |
| fungi | shardy: i'm starting to suspect that the increased approval volume following everyone's return from the holidays, plus our recent upgrade of zuul to a beefier server, has started to strain jenkins in ways we hadn't previously seen with this regularity | 15:41 |
| *** dpyzhov has joined #openstack-infra | 15:41 | |
| pasquier-s | hi, I've got a review that's been approved but the gate jobs seem to be lost: | 15:41 |
| pasquier-s | https://review.openstack.org/#/c/65269/ | 15:42 |
| pasquier-s | any hint? | 15:42 |
| fungi | pasquier-s: looking | 15:42 |
| pasquier-s | fungi, thanks! | 15:42 |
| *** mfink has joined #openstack-infra | 15:42 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Reformat heat.json https://review.openstack.org/65558 | 15:43 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add Bartosz Gorski to heat-core https://review.openstack.org/65534 | 15:43 |
| *** jgrimm has joined #openstack-infra | 15:44 | |
| *** mfink has quit IRC | 15:44 | |
| fungi | pasquier-s: yeah, it looks like that may have happened right when we were restarting zuul yesterday, possibly between when jeblair dumped the queue and stopped the service, so it didn't get reenqueued with the others. i'll add it back to the gate for you in just a moment | 15:44 |
| *** mfink has joined #openstack-infra | 15:44 | |
| *** mfink has quit IRC | 15:45 | |
| *** mfink has joined #openstack-infra | 15:46 | |
| *** rcleere has joined #openstack-infra | 15:48 | |
| fungi | pasquier-s: openstack/python-heatclient 65269,2 has been enqueued into the gate pipeline and appears on http://status.openstack.org/zuul/ now | 15:51 |
| *** pblaho has quit IRC | 15:52 | |
| *** roaet has left #openstack-infra | 15:52 | |
| *** chandankumar_ has quit IRC | 15:54 | |
| *** AJaeger has quit IRC | 15:56 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Update Trove Core https://review.openstack.org/65587 | 15:59 |
| *** markmcclain has quit IRC | 16:00 | |
| *** rakhmerov has quit IRC | 16:01 | |
| *** gothicmindfood has joined #openstack-infra | 16:02 | |
| *** gothicmindfood has quit IRC | 16:02 | |
| *** gothicmindfood has joined #openstack-infra | 16:02 | |
| *** gothicmindfood has quit IRC | 16:02 | |
| *** pblaho has joined #openstack-infra | 16:03 | |
| *** rakhmerov has joined #openstack-infra | 16:03 | |
| *** pblaho has quit IRC | 16:03 | |
| *** gothicmindfood has joined #openstack-infra | 16:04 | |
| *** rakhmerov has quit IRC | 16:07 | |
| *** markmcclain has joined #openstack-infra | 16:10 | |
| *** medberry has joined #openstack-infra | 16:12 | |
| *** medberry has joined #openstack-infra | 16:12 | |
| *** rcarrillocruz has joined #openstack-infra | 16:12 | |
| *** rcarrillocruz1 has quit IRC | 16:13 | |
| *** markmcclain has quit IRC | 16:16 | |
| *** krotscheck has quit IRC | 16:20 | |
| *** alexpilotti has quit IRC | 16:20 | |
| *** yaguang has quit IRC | 16:20 | |
| *** tma996 has quit IRC | 16:22 | |
| *** gyee_ has joined #openstack-infra | 16:25 | |
| *** pcrews has joined #openstack-infra | 16:26 | |
| *** thuc has joined #openstack-infra | 16:28 | |
| *** thuc_ has joined #openstack-infra | 16:30 | |
| *** thuc has quit IRC | 16:30 | |
| *** jcoufal_ has quit IRC | 16:31 | |
| *** rakhmerov has joined #openstack-infra | 16:31 | |
| *** mozawa has quit IRC | 16:34 | |
| *** rakhmerov has quit IRC | 16:36 | |
| *** mozawa has joined #openstack-infra | 16:39 | |
| *** bogdando has quit IRC | 16:42 | |
| *** CaptTofu has quit IRC | 16:44 | |
| *** UtahDave has joined #openstack-infra | 16:45 | |
| *** medberry is now known as med_ | 16:47 | |
| openstackgerrit | João Vale proposed a change to openstack-infra/jenkins-job-builder: Add support for parameters in pipeline publisher. https://review.openstack.org/65719 | 16:49 |
| *** che-arne has quit IRC | 16:49 | |
| *** che-arne has joined #openstack-infra | 16:50 | |
| *** senk1 has quit IRC | 16:50 | |
| *** ^d has joined #openstack-infra | 16:57 | |
| *** senk has joined #openstack-infra | 16:57 | |
| *** sparkycollier has joined #openstack-infra | 16:59 | |
| fungi | i've taken precise39 offline | 17:00 |
| *** derekh has quit IRC | 17:02 | |
| SergeyLukjanov | the same problems? | 17:02 |
| fungi | yep | 17:03 |
| SergeyLukjanov | :( | 17:03 |
| SergeyLukjanov | jenkins01 again | 17:03 |
| fungi | it seems to be fairly evenly distributed between jenkins01 and 02 | 17:04 |
| *** markmc has quit IRC | 17:04 | |
| fungi | i need to add an account in their jira and comment on that bug that we're seeing the same backtraces in 1.543 (possible regression? different issue?) | 17:04 |
| SergeyLukjanov | fungi, yup | 17:04 |
| *** NikitaKonovalov has quit IRC | 17:04 | |
| *** AaronGr_Zzz is now known as AaronGr | 17:06 | |
| *** praneshp has joined #openstack-infra | 17:07 | |
| *** alexpilotti has joined #openstack-infra | 17:09 | |
| *** jorisroovers has quit IRC | 17:10 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow cinder to use bare-precise nodes https://review.openstack.org/65732 | 17:16 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow glance to use bare-precise nodes https://review.openstack.org/65733 | 17:16 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow keystone to use bare-precise nodes https://review.openstack.org/65734 | 17:16 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow heat to use bare-precise nodes https://review.openstack.org/65735 | 17:16 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow horizon to use bare-precise nodes https://review.openstack.org/65736 | 17:16 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow ceilometer to use bare-precise nodes https://review.openstack.org/65737 | 17:16 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow swift to use bare-precise nodes https://review.openstack.org/65738 | 17:16 |
| fungi | those ^ should cover typical precise-based jobs for ~80-90% of the changes i currently see in the gate | 17:16 |
| fungi | done as individual changes to make them easier to roll in or revert as we see issues | 17:17 |
| *** rcarrillocruz has quit IRC | 17:17 | |
| SergeyLukjanov | fungi, are you planning to move all projects to use bare nodes? | 17:21 |
| *** markmcclain has joined #openstack-infra | 17:21 | |
| fungi | SergeyLukjanov: yeah, but for now i'm just looking to get the bulk of them covered. i think we can probably follow up with one more or a handful of changes to move the remaining precise-using jobs in bulk once we're comfortable | 17:22 |
| fungi | and i want to wait until there are other infra cores around to confirm this is a sane direction | 17:23 |
| SergeyLukjanov | fungi, I"m proposing to make savanna able to use bare nodes to test it, what do you think about it? | 17:24 |
| fungi | sounds great! | 17:24 |
| fungi | base it on the tip of master though, so we can merge it without needing to wait on my stack | 17:25 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow swift to use bare-precise nodes https://review.openstack.org/65738 | 17:26 |
| fungi | that one ^ i had the wrong project on (accidentally put it on the client when i meant to do the server) | 17:26 |
| *** ruhe is now known as _ruhe | 17:27 | |
| SergeyLukjanov | fungi, I'll add CR for using bare nodes for savanna too | 17:27 |
| fungi | awesome | 17:27 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Allow savanna to use bare-precise nodes https://review.openstack.org/65739 | 17:28 |
| *** yassine has quit IRC | 17:30 | |
| *** praneshp has quit IRC | 17:31 | |
| fungi | SergeyLukjanov: we can merge that if you're feeling fairly confident it won't break for you (or if you don't mind rechecking once we revert) | 17:32 |
| *** rakhmerov has joined #openstack-infra | 17:32 | |
| fungi | also, thanks for volunteering to be a guinea pig! | 17:32 |
| *** reed has joined #openstack-infra | 17:33 | |
| SergeyLukjanov | fungi, I don't think that it break anything in savanna, we're not using any specific jobs | 17:33 |
| SergeyLukjanov | fungi, let's try tomerge that and wait for a day to collect results | 17:34 |
| SergeyLukjanov | fungi, is it possible to search in console logs using logstash? | 17:34 |
| fungi | yes, just no wildcard searches | 17:35 |
| *** praneshp has joined #openstack-infra | 17:35 | |
| *** mancdaz is now known as mancdaz_away | 17:35 | |
| SergeyLukjanov | fungi, k, thx | 17:36 |
| *** rakhmerov has quit IRC | 17:37 | |
| *** NikitaKonovalov has joined #openstack-infra | 17:37 | |
| fungi | so far, all of the nova jobs in the gate which ran on nodepool bare-precise nodes (and which were allowed to complete without being cancelled/aborted) have succeeded. also, we haven't previously seen any impact from the infra jobs we moved to that node type either | 17:41 |
| SergeyLukjanov | fungi, great | 17:42 |
| fungi | i just jinxed myself... https://jenkins01.openstack.org/job/gate-config-layout/1845/console | 17:42 |
| SergeyLukjanov | fungi, I'll monitor savanna jobs for any issues | 17:42 |
| *** nati_ueno has joined #openstack-infra | 17:42 | |
| SergeyLukjanov | fungi, oh, the same error at bare node slave @ jenkins01 | 17:43 |
| fungi | though that's an expected behavior. jenkins will still have occasional broken slave agents regardless, because of whatever the jenkins bug is | 17:43 |
| fungi | the up side is that the node will get thrown away and not reused over and over in a rapid-fire loop failing every job it's given | 17:44 |
| SergeyLukjanov | fungi, yup and it's much better I think than manually disable slaves each time | 17:44 |
| SergeyLukjanov | at least... | 17:44 |
| fungi | and as intended, if you follow the link to https://jenkins01.openstack.org/computer/bare-precise-rax-ord-1032590 where it ran, that slave has already been deregistered and deleted | 17:44 |
| fungi | so the broken slave killed a job, but just *that one* job | 17:45 |
| SergeyLukjanov | fungi, yup, see it | 17:45 |
| SergeyLukjanov | we've used 534 jenkins for a long time for savanna-ci w/o such problems | 17:46 |
| zaro | fungi: is there a way to make one review dependent on another when both reviews have already been pushed as independent changes to gerrit? | 17:46 |
| SergeyLukjanov | j02 is 543 and we have another agent error on it | 17:46 |
| SergeyLukjanov | zaro, you need to rebase one to another | 17:47 |
| fungi | zaro: easy solution, git review -d change1 && git review -x change2 && git review -x change3 && ... | 17:47 |
| SergeyLukjanov | fungi, j03 and j04 are 543 too | 17:48 |
| SergeyLukjanov | fungi, are there any problems in updating jenkins? | 17:48 |
| fungi | SergeyLukjanov: we upgraded -dev from 1.525 to 1.543, ran it through some tests, then uprgaded 02 to 1.543, ran into a bug in one of our plugins so rolled that back tracked down/fixed and upgraded again | 17:49 |
| SergeyLukjanov | fungi, oh, got it | 17:50 |
| fungi | about the time we determined it was at least as stable as 1.525 we rushed into building 03 and 04 with 1.543 before we got around to upgrading 01 | 17:50 |
| fungi | jenkins.o.o is similarly still on 1.525 | 17:50 |
| zaro | fungi, SergeyLukjanov : both solutions would mean a new patch getting pushed to gerrit. I guess i wanted to avoid that and just make the 2 existing patches on gerrit dependent. | 17:50 |
| mgagne | could someone explain to me the fundamental differences between precise and bare-precise ? | 17:51 |
| SergeyLukjanov | zaro, you can't do it w/o pushing a new patch | 17:51 |
| fungi | mgagne: "precise" nodes are long-running nodes which get reused over and over | 17:51 |
| openstackgerrit | João Vale proposed a change to openstack-infra/jenkins-job-builder: Add support to specify GitLab version. https://review.openstack.org/65745 | 17:51 |
| fungi | mgagne: bare-precise are our new nodepool-managed single-use slaves | 17:51 |
| mgagne | fungi: bare-precise are throw away managed by nodepool? | 17:51 |
| zaro | SergeyLukjanov: ok. thanks. | 17:51 |
| mgagne | fungi: alright, thanks | 17:51 |
| fungi | mgagne: "bare" in this case meaning "not dsvm" | 17:52 |
| mgagne | fungi: oh | 17:52 |
| *** dmsimard has joined #openstack-infra | 17:52 | |
| zaro | fungi: seeems very overloaded term :) | 17:52 |
| dmsimard | fungi: Thanks for deactivating https://jenkins01.openstack.org/computer/precise39/? .. was confused about the build failures :) | 17:52 |
| fungi | mgagne: the expectation being that we may wind up with more types of precise nodes in nodepool (for example, maybe py3k-precise) | 17:53 |
| fungi | dmsimard: i should start linking https://launchpad.net/bugs/1267364 in the deactivation messages | 17:54 |
| *** luqas has quit IRC | 17:54 | |
| mgagne | fungi: sure. I would have suggested renaming precise to dsvm-precise then :P | 17:54 |
| dmsimard | fungi: Is there a way to ask jenkins another run without submitting another patch set ? | 17:55 |
| fungi | dmsimard: yes, was the patch already approved or just being checked? | 17:55 |
| mgagne | dmsimard: recheck bug #### | 17:56 |
| dmsimard | fungi: Just being checked, not approved yet | 17:56 |
| mgagne | leave a comment: recheck bug 1267364 | 17:56 |
| *** NikitaKonovalov has quit IRC | 17:56 | |
| dmsimard | the bug is the id of the review, right %? | 17:56 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: make scm test as the examples https://review.openstack.org/65186 | 17:57 |
| SergeyLukjanov | dmsimard, you can find more info about it here - https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures | 17:57 |
| SergeyLukjanov | dmsimard, the bug is the launchpad bug i | 17:57 |
| SergeyLukjanov | id* | 17:57 |
| *** afazekas has quit IRC | 18:00 | |
| fungi | dmsimard: in this case, bug 1267364 is a bug about failing jenkins slave agents | 18:00 |
| *** BobBall is now known as BobBallAway | 18:00 | |
| fungi | dmsimard: the recheck bug ###### syntax is so that we can try to keep track of what bugs are causing devs to need to re-test their changes | 18:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow savanna to use bare-precise nodes https://review.openstack.org/65739 | 18:01 |
| *** CaptTofu has joined #openstack-infra | 18:01 | |
| *** sparkycollier has quit IRC | 18:01 | |
| *** jroovers has joined #openstack-infra | 18:01 | |
| fungi | SergeyLukjanov: in about 30 minutes you can recheck an open change and see whether that ^ worked | 18:01 |
| dmsimard | fungi: Yeah, I caught on - thanks | 18:01 |
| *** NikitaKonovalov has joined #openstack-infra | 18:01 | |
| *** jorisroovers has joined #openstack-infra | 18:01 | |
| *** moted has quit IRC | 18:01 | |
| fungi | mgagne: we already have dsvm-precise nodes (those are created by nodepool to run "devstack vm" jobs) | 18:02 |
| fungi | mgagne: the "precise" nodes are just the ones which aren't managed by nodepool at all | 18:02 |
| mgagne | fungi: oh, now that's getting confusing =) | 18:02 |
| mgagne | fungi: thanks for the info | 18:03 |
| fungi | mgagne: i don't think it should be confusing... "precise" is legacy long-running general-purpose workers. "precise-bare" and "precise-dsvm" are two types of precise nodes built and managed by nodepool | 18:03 |
| *** moted has joined #openstack-infra | 18:04 | |
| fungi | er, "bare-precise" and "dsvm-precise" i mean | 18:04 |
| mgagne | fungi: understood | 18:04 |
| fungi | we'll likely soon also grow some "py3k-precise" and "bare-centos6" node types too | 18:04 |
| *** jroovers has quit IRC | 18:06 | |
| *** praneshp has quit IRC | 18:07 | |
| *** harlowja_away is now known as harlowja | 18:07 | |
| *** jpich has quit IRC | 18:08 | |
| *** sparkycollier has joined #openstack-infra | 18:08 | |
| *** thuc_ has quit IRC | 18:12 | |
| *** karpukhina has joined #openstack-infra | 18:12 | |
| *** sdake_ has quit IRC | 18:12 | |
| SergeyLukjanov | fungi, I'll check it | 18:12 |
| *** thuc has joined #openstack-infra | 18:13 | |
| *** karpukhina has quit IRC | 18:13 | |
| *** sparkycollier has quit IRC | 18:14 | |
| *** Ajaeger has joined #openstack-infra | 18:15 | |
| *** johnthetubaguy has quit IRC | 18:15 | |
| openstackgerrit | Yanis Guenane proposed a change to openstack-infra/config: New project request: eDeploy https://review.openstack.org/62377 | 18:17 |
| *** thuc has quit IRC | 18:17 | |
| *** morganfainberg is now known as needscoffee | 18:17 | |
| *** thuc has joined #openstack-infra | 18:20 | |
| SergeyLukjanov | fungi, heh, I need to update my zuul changes to easily determine the time when zuul config updated :) | 18:21 |
| *** thuc has quit IRC | 18:21 | |
| *** thuc has joined #openstack-infra | 18:21 | |
| *** thuc has quit IRC | 18:22 | |
| *** thuc has joined #openstack-infra | 18:22 | |
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/config: Add compass project to stackforge https://review.openstack.org/65178 | 18:27 |
| SergeyLukjanov | fungi, do you now anything about new project creation? | 18:27 |
| SergeyLukjanov | fungi, I mean about the how it created | 18:28 |
| SergeyLukjanov | will it be the empty repo or the python project now? | 18:28 |
| fungi | SergeyLukjanov: which new project, specifically? | 18:29 |
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/config: Add compass project to stackforge https://review.openstack.org/65178 | 18:29 |
| fungi | SergeyLukjanov: it can be either, depending on your configuration | 18:30 |
| *** dizquierdo has quit IRC | 18:30 | |
| SergeyLukjanov | fungi, for example, if i'd like to add new project to the stackforge | 18:31 |
| SergeyLukjanov | fungi, w/o upstream | 18:31 |
| fungi | SergeyLukjanov: what determines that behavior is whether you provide an "upstream" (poorly named, we should probably eventually change that) | 18:31 |
| SergeyLukjanov | fungi, I remember some work on using cookiecutter for new projects creation | 18:31 |
| SergeyLukjanov | but can't find it atm | 18:31 |
| fungi | without "upstream" specified, you get an empty repo with a solitary commit adding a correct .gitreview file | 18:32 |
| mgagne | SergeyLukjanov: like this one: https://github.com/stackforge/puppet-savanna/commit/cba54c4c6a5e7cf17b91f70b54c76336b28dda28 | 18:32 |
| jog0 | so I don't think the rax high perf are fast enough | 18:32 |
| jog0 | http://logs.openstack.org/28/62328/6/gate/gate-tempest-dsvm-large-ops/e140eb7/console.html | 18:32 |
| fungi | SergeyLukjanov: which, if you don't have much code to put into the repo yet, us probably a good bet | 18:32 |
| SergeyLukjanov | mgagne, yup, I know, but I was thinking that this behaviour was changed | 18:33 |
| *** rakhmerov has joined #openstack-infra | 18:33 | |
| mgagne | SergeyLukjanov: oh, I don't know. All I know is there was issues with project creation at one time, don't know if it got fixed ^^' | 18:33 |
| fungi | SergeyLukjanov: you can check out and use openstack-dev/cookiecutter to create a templated project and add that as your next commit fairly easily if you don't already have a repo you want to import | 18:34 |
| fungi | jog0: that may explain why large-ops has been causing so many resets recently | 18:34 |
| jog0 | fungi: yup | 18:34 |
| jog0 | I545a4ac4ce3d63191d2c9da10cfbc968ce5522d1 | 18:34 |
| jog0 | timeline fits | 18:34 |
| SergeyLukjanov | fungi, yup, I know about it, just was confused about possible default behavior change, thank you | 18:34 |
| jog0 | jeblair: ^ | 18:36 |
| jog0 | fungi: how should we handle this? | 18:36 |
| SergeyLukjanov | fungi, the question was because I'm thinking about moving savanna-ci jjb/zuul configs to the stackforge to be able to review/manage them, what do you think about this? | 18:36 |
| jog0 | I think this means rax will have more timeouts on other tests too | 18:37 |
| fungi | gah, a bare-precise node ran into a java io exception and then continued grabbing several jobs before it was deregistered... https://jenkins04.openstack.org/job/gate-nova-docs/84/console https://jenkins04.openstack.org/job/gate-nova-pep8/91/console https://jenkins04.openstack.org/job/gate-nova-python27/78/console | 18:37 |
| dims | jog0, https://review.openstack.org/#/c/65593/ should help figure out problems with specific slaves | 18:37 |
| SergeyLukjanov | jog0, do we have timeouts only on rax nodes now? | 18:38 |
| jog0 | SergeyLukjanov: I don't know how to confirm that | 18:38 |
| *** rakhmerov has quit IRC | 18:38 | |
| jog0 | we don't store that data in logstash | 18:38 |
| dims | jog0, see above :) | 18:38 |
| fungi | jog0: the review dims just linked | 18:38 |
| fungi | (would add that) | 18:38 |
| jog0 | dims: \o/ | 18:38 |
| dims | jog0, what else will help? | 18:39 |
| *** thuc has quit IRC | 18:39 | |
| dims | what else should we log that will help? | 18:39 |
| mfer | SergeyLukjanov mgagne the issue with new project creation is still open... https://bugs.launchpad.net/openstack-ci/+bug/1242569 | 18:39 |
| fungi | zaro: would you mind looking at https://review.openstack.org/65593 ? it's a one-liner addition to the zmq publisher plugin which would help with job failure diagnostics | 18:40 |
| *** thuc has joined #openstack-infra | 18:40 | |
| jog0 | dims: that was the big thing missing | 18:40 |
| jog0 | so spot check of large-ops failures points to rax | 18:40 |
| fungi | mfer: SergeyLukjanov: yes, i'm cycling back around on new project requests to get one in shape to test the current assumed fix for new project creation... i'll try to make another pass here in a bit | 18:40 |
| mfer | fungi i'll be around if there is anything i can do to help | 18:41 |
| jog0 | fungi: can we revert the rax high perf patch for now? or is that a bad idea | 18:41 |
| *** thuc_ has joined #openstack-infra | 18:42 | |
| fungi | jog0: i need to look back and see whether there's just the one change to revert, or whether we have to go through a cycle of transitioning back to non-performance images et cetera | 18:43 |
| *** thuc_ has quit IRC | 18:43 | |
| *** thuc_ has joined #openstack-infra | 18:44 | |
| jog0 | fungi: thanks. the other option is to have a seperate large_ops number for rax nodes | 18:44 |
| *** thuc has quit IRC | 18:44 | |
| jog0 | its 100 on hp | 18:44 |
| jog0 | down from 150 to avoid these issues | 18:44 |
| jog0 | turns out rax is just hella slow | 18:44 |
| *** thuc_ has quit IRC | 18:45 | |
| *** thuc has joined #openstack-infra | 18:45 | |
| *** che-arne has quit IRC | 18:45 | |
| *** dstanek has joined #openstack-infra | 18:45 | |
| fungi | jog0: yeah, so looking at the patch series, we'd need to take 65237, 65246 and 65619 into account if we're going to revert 65236 | 18:46 |
| *** herndon_ has quit IRC | 18:46 | |
| *** ^d has quit IRC | 18:47 | |
| *** dmsimard has left #openstack-infra | 18:47 | |
| fungi | i think it's mainly 65246 which would need to be undone first, then get the images back, then undo 65237 (but we'd need to make sure the other two i mentioned don't have any implications on that) | 18:47 |
| fungi | jog0: the main difference 65236 brings is that it's now using them for gate jobs as well as check jobs, but these failures should have been apparent in the weeks that we ran check jobs on them if so | 18:49 |
| *** sarob has joined #openstack-infra | 18:49 | |
| jog0 | large-ops only runs on gate nodes | 18:50 |
| jog0 | even in cehck queue | 18:50 |
| jog0 | because of this issue | 18:50 |
| fungi | ahhh | 18:50 |
| jog0 | the perf aspect to it | 18:50 |
| jog0 | which is something we don't like | 18:50 |
| *** dstanek has quit IRC | 18:50 | |
| fungi | also i mis-pasted above. it's 65237 which removed the images, and which we'd need to undo and wait on before reverting 65236 | 18:51 |
| *** _ruhe is now known as ruhe | 18:52 | |
| *** jerryz has joined #openstack-infra | 18:52 | |
| fungi | jog0: how much lower would the quantity need to go, do you think, to work on rackspace performance nodes? | 18:52 |
| fungi | jog0: or should we think about upping the job timeout? | 18:53 |
| jog0 | fungi: tiemout is in nova | 18:53 |
| jog0 | and not sure how much worse rax hi perf are | 18:53 |
| jog0 | would have to experiment | 18:53 |
| openstackgerrit | João Vale proposed a change to openstack-infra/jenkins-job-builder: Add support for parameters in pipeline publisher. https://review.openstack.org/65719 | 18:53 |
| fungi | oh, got it, so the timeout we're hitting on those isn't the overall d-g timeout, right | 18:54 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Add new javamelody-plugin project https://review.openstack.org/65162 | 18:56 |
| *** beagles has quit IRC | 18:56 | |
| fungi | jog0: well, i can add the devstack-precise-check images back, get those building, then tear the devstack-precise images back out of rax, but it'll be a little while to complete and will reduce our capacity again | 18:57 |
| fungi | jog0: or i can see about adding a new label just to the hpcloud nodes and switch the large-ops job to that (but again, that's a sort of ugly workaround) | 18:58 |
| *** dstanek has joined #openstack-infra | 18:59 | |
| jog0 | fungi: correct, nova is timing out. | 19:00 |
| jog0 | fungi: and solution wise: its your call | 19:00 |
| *** b3nt_pin has joined #openstack-infra | 19:01 | |
| *** b3nt_pin is now known as beagles | 19:01 | |
| *** NikitaKonovalov has quit IRC | 19:02 | |
| fungi | jeblair: clarkb: mordred: if you're near an internet, opinions would be appreciated. basic summary, we were never running large-ops jobs on rackspace, even in check. now that we got rid of the check-specific nodes, large-ops jobs are running on rackspace and timing out | 19:02 |
| sdague | mriedem: did you start in on adding other job support into er? if not I was going to work on that, because I need to step away from the data analysis bits for a while or I'm going to break my computer | 19:03 |
| *** dstanek has quit IRC | 19:04 | |
| mriedem | sdague: which other job support? i thought there was a tempest one that wasn't getting hit for large ops yesterday but turned out that it was | 19:04 |
| mriedem | sometimes... | 19:04 |
| fungi | jeblair: clarkb: mordred: long-term, i think jobs like large-ops whose success or failure is determined by the performance of the underlying provider (global job timeouts aside) need to do some sort of benchmarking prior to starting the job so they know how far its safe to scale. short term, our options are somewhat more limited | 19:04 |
| mriedem | sdague: so no | 19:04 |
| mriedem | sdague: this is a bug i started looking at though, some notes in there: https://bugs.launchpad.net/openstack-ci/+bug/1265906 | 19:05 |
| sdague | mriedem: grenade still needs to be added, as well as the unit tests | 19:05 |
| *** dstanek has joined #openstack-infra | 19:05 | |
| fungi | jeblair: clarkb: mordred: options i see are to bring back the check nodes and stop putting normal nodes in rackspace again, or try to find some way to designate the hpcloud nodes and switch the large-ops job to only run on those, or scale down the large-ops job to the point where it's probably ineffective on hpcloud | 19:05 |
| mriedem | sdague: ah, no, didn't dig into that. got some stuff i needed to work on before being out next week | 19:06 |
| mriedem | i spent about 2 days doing reviews and infra only stuff this week so got a bit sidetracked | 19:06 |
| *** alexpilotti has quit IRC | 19:06 | |
| sdague | yep, no worries, I just didn't want to dive into it if you had. | 19:06 |
| *** ^d has joined #openstack-infra | 19:08 | |
| *** alexpilotti has joined #openstack-infra | 19:09 | |
| *** praneshp has joined #openstack-infra | 19:10 | |
| *** ^d has quit IRC | 19:12 | |
| *** _david_ has joined #openstack-infra | 19:12 | |
| *** SergeyLukjanov has quit IRC | 19:13 | |
| *** rcarrillocruz has joined #openstack-infra | 19:14 | |
| *** rossella_s has quit IRC | 19:14 | |
| jerryz | fungi: is it that large-ops jobs end up timeout waiting for instance to be Active? | 19:14 |
| fungi | jeblair: clarkb: mordred: i'm going to work on bringing back the devstack-precise-check nodes for now, and then stop building devstack-precise nodes on rackspace | 19:15 |
| *** rossella_s has joined #openstack-infra | 19:15 | |
| fungi | jerryz: i don't have the error details handy. it's the one which tries to spin up 100 instances at once | 19:15 |
| *** hogepodge has joined #openstack-infra | 19:16 | |
| jerryz | fungi: i used to run into that kind of error on my own test cloud provider. remember me asking you about flavor choice on your cloud providers? i ended up having to raise the number of cpus for devstack slaves | 19:17 |
| sdague | jog0: so is there a way, like in the turbo hipster case, where we can isolate large-ops? Because I'm concerned we're going to run into another issue where it's actually a performance test, and 2 std deviations is not enough | 19:18 |
| *** melwitt has joined #openstack-infra | 19:19 | |
| fungi | jerryz: yes, i believe this is probably similar to what you were encountering in your cloud | 19:19 |
| jog0 | sdague: not sure what you mean? | 19:20 |
| *** rcleere has quit IRC | 19:22 | |
| jerryz | fungi: i found with poor cpu performance, the iowait during the large-ops test was big. Having to reduce it under 20% at its peak then large-ops passed. before upping the flavor, iowait hit over 40% | 19:22 |
| sdague | jog0: large-ops is basically a performance test | 19:24 |
| sdague | that's got a large-ops value set to completely within a timeout, otherwise fail | 19:25 |
| sdague | but the variability in the cloud envs mean performance tests are hard | 19:25 |
| sdague | because the timing is all over the map | 19:25 |
| jog0 | the variance without a single cloud hasn't been much of an issue | 19:26 |
| fungi | jog0: depends on what you call a single cloud | 19:26 |
| fungi | variance between hpcloud east and west is huge | 19:27 |
| jog0 | and while this is a perforamance test its just to make sure things aren't terrible so dropping the number is fine globallty is fine with me | 19:27 |
| jog0 | we used to not abe able to boot 30 instances at once | 19:27 |
| jog0 | fungi: true | 19:27 |
| jog0 | I am fine with dropping large-ops to 50 | 19:27 |
| jog0 | that still enough of a test to catch things like rootwrap regressions etc | 19:28 |
| fungi | jog0: if you want to try that first, i can promote it to the head of the gate asap for some quick relief | 19:28 |
| *** thuc has quit IRC | 19:29 | |
| *** thuc has joined #openstack-infra | 19:30 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Drop large-ops test down to 50 instances from 100 https://review.openstack.org/65760 | 19:30 |
| jog0 | fungi: ^ | 19:30 |
| fungi | rolling back the changes which shifted more of our load onto rackspace is going to be a more involved transition, so i'd like to confer with the rest of infra before we do that (but i can get the changes and a basic plan of attack drafted up to save us some time in case we decide we should) | 19:30 |
| jog0 | fungi: sounds like a plan to me | 19:31 |
| jog0 | hopefully this will work instead though | 19:31 |
| *** ^d has joined #openstack-infra | 19:32 | |
| *** rnirmal has quit IRC | 19:33 | |
| Ajaeger | fungi: did you read http://lists.openstack.org/pipermail/openstack-infra/2014-January/000630.html ? | 19:34 |
| Ajaeger | Monty Taylor's laptop was stolen and he asks to have his ssh-keys disabled... | 19:34 |
| *** rakhmerov has joined #openstack-infra | 19:34 | |
| fungi | Ajaeger: haven't seen it yet. irc has kept me away from -email | 19:34 |
| fungi | will do | 19:34 |
| *** thuc has quit IRC | 19:34 | |
| Ajaeger | fungi: that's why I asked ;) Thanks for taking care. | 19:35 |
| *** thuc has joined #openstack-infra | 19:35 | |
| fungi | jog0: okay, it's promoted to the head of teh gate | 19:36 |
| *** rakhmerov1 has joined #openstack-infra | 19:36 | |
| *** rakhmerov has quit IRC | 19:36 | |
| jog0 | fungi: thanks | 19:37 |
| *** rcleere has joined #openstack-infra | 19:38 | |
| *** oubiwann has quit IRC | 19:39 | |
| *** oubiwann has joined #openstack-infra | 19:39 | |
| *** rnirmal has joined #openstack-infra | 19:39 | |
| *** _david_ has left #openstack-infra | 19:40 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove SSH key for Monty Taylor (mordred) https://review.openstack.org/65764 | 19:41 |
| *** rakhmerov1 has quit IRC | 19:41 | |
| *** sparkycollier has joined #openstack-infra | 19:42 | |
| *** markmcclain has quit IRC | 19:42 | |
| *** _david_ has joined #openstack-infra | 19:43 | |
| fungi | approved that ^ | 19:43 |
| fungi | thanks Ajaeger! | 19:43 |
| fungi | also removing his key from the gerrit database | 19:44 |
| fungi | i've disabled precise 12. it just decided to go on a rampage and fail a ton of jobs | 19:44 |
| *** ruhe is now known as _ruhe | 19:44 | |
| *** dstanek has quit IRC | 19:46 | |
| fungi | jog0: 65760 won the coin toss and is self-gating on a rax node... https://jenkins02.openstack.org/job/gate-tempest-dsvm-large-ops/10221/console | 19:47 |
| _david_ | fungi, zaro have you tried that guy on gerrit-dev.o.o: https://review.openstack.org/#/c/64978 | 19:47 |
| Ajaeger | fungi, thank you for fast action, I suggest you followup via email and tell mordred about it. | 19:49 |
| _david_ | s/gerrit-dev.o.o/review-dev.o.o | 19:49 |
| *** pliszka has joined #openstack-infra | 19:49 | |
| fungi | Ajaeger: no need--i can reply | 19:49 |
| fungi | Ajaeger: thanks a ton for bringing it to my attention quickly | 19:50 |
| *** sarob has quit IRC | 19:50 | |
| *** sarob has joined #openstack-infra | 19:51 | |
| *** sarob has quit IRC | 19:53 | |
| Ajaeger | fungi: thanks a lot for holding the infrastructure together ;) | 19:53 |
| *** sarob has joined #openstack-infra | 19:53 | |
| Shrews | fungi: he just wanted an excuse for the new x240 | 19:53 |
| *** needscoffee is now known as morganfainberg | 19:54 | |
| *** blamar has quit IRC | 19:54 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove SSH key for Monty Taylor (mordred) https://review.openstack.org/65764 | 19:54 |
| Shrews | jokes on mordred, though, b/c the HD display version isn't out yet | 19:55 |
| *** david-lyle_ has joined #openstack-infra | 19:55 | |
| fungi | Shrews: maybe it's an excuse to not have to work while he;s out | 19:56 |
| *** markmcclain has joined #openstack-infra | 19:56 | |
| Shrews | also possible | 19:57 |
| fungi | Shrews: i used to take the battery out of my pager when i got really tired of work beeping me | 19:57 |
| fungi | "battery must have ran dry, you called it so many times!" | 19:57 |
| *** CaptTofu has quit IRC | 19:59 | |
| fungi | _david_: i saw the commit title but haven's made it that far through my review queue yet. that's awesome that they integrated it. one more item for the list of code we can stop maintaining | 20:03 |
| *** mrmartin has joined #openstack-infra | 20:05 | |
| _david_ | fungi, exact | 20:05 |
| fungi | it's in my starred list of patches, looks good at first glance but we definitely don't want to merge that into production of course until we upgrade | 20:05 |
| *** vipuls is now known as vipuls-away | 20:05 | |
| *** vipuls-away is now known as vipuls | 20:05 | |
| mrmartin | re | 20:05 |
| _david_ | sure, sure, i just wonder what would be the best way to integrate it in config site. Is this solely manual step (what i assume)? | 20:06 |
| fungi | mrmartin: hi there! re what? | 20:06 |
| mrmartin | hi fungi | 20:07 |
| mrmartin | I have a quick question. I want to add some check / gating scripts for the community portal. | 20:07 |
| fungi | _david_: i think what we'd want to do is wrap the command in the hook with a conditional on the role id and then only set it empty on review-dev | 20:07 |
| mrmartin | What do you think, what is the shortest way to support php platform somehow? | 20:07 |
| fungi | _david_: that way we could merge it as is and not break 2.4.4 in production | 20:08 |
| *** boris-42 has quit IRC | 20:08 | |
| _david_ | fungi, make sense | 20:08 |
| fungi | mrmartin: do you have any example code for your tests? usually you would put them in your repository, probably in a tests subdirectory, then we can run them automatically to test proposed changes | 20:09 |
| *** SergeyLukjanov has joined #openstack-infra | 20:09 | |
| mrmartin | fungi: ok, but for running the test you need to deploy some php environment, right? | 20:09 |
| *** ^d has quit IRC | 20:10 | |
| fungi | mrmartin: mainly what we'd need to know is what additional tools are required to run the test, and if possible we'd like to be able to run them on ubuntu 12.04 lts while downloading the least amount of third-party tools as possible | 20:10 |
| *** pliszka has left #openstack-infra | 20:11 | |
| mrmartin | ok 12.04 lts is supported, so it won't be a problem. And one additional thing, finally I want to build a snapshot tarball from the output of a drush make command. | 20:11 |
| fungi | mrmartin: could the tests be run directly under a php interpreter (with no separate webserver process)? if so, that would probably be pretty easy to implement in a job | 20:11 |
| *** alexpilotti has quit IRC | 20:11 | |
| mrmartin | fungi, yes the tests can run without a browser, we are not yet using any selenium type testing. | 20:11 |
| *** SergeyLukjanov has quit IRC | 20:11 | |
| fungi | mrmartin: hashar, when he's around, also may have some suggestions. at wikimedia they test a *ton* of php using mostly the same tools we do | 20:12 |
| mrmartin | ok, so if I prepare a test and write some notice about the required environment, and execution you could help me to integrate it into ci process. | 20:13 |
| fungi | mrmartin: as for the custom tarball job, that's probably easy to add as well. our normal tarball jobs are specific to python project packaging/tooling (setup.py sdist stuff) but we have other custom tarball jobs. storyboard-webclient has a change proposed for something similar | 20:13 |
| fungi | mrmartin: sure thing, i'd love to help | 20:13 |
| mrmartin | What I want to achieve is to run the tests first, create a tarball as a part of commit. From other part I want to upgrade the staging scripts and create prod puppet manifests to use those tarballs for site deploy / upgrade. | 20:14 |
| mrmartin | cool. | 20:14 |
| fungi | mrmartin: here's how storyboard-webclient is thinking about doing their tarballs... https://review.openstack.org/#/c/64761/6/modules/openstack_project/files/jenkins_job_builder/config/storyboard.yaml | 20:14 |
| mrmartin | thanks | 20:14 |
| fungi | mrmartin: i'm going to guess that what you're referring to as a snapshot tarball is going to be more of a milestone/release and not something you're going to want to install in production for every single approved commit to the git repository, righth? | 20:16 |
| *** sdake has joined #openstack-infra | 20:16 | |
| *** sdake has quit IRC | 20:16 | |
| *** sdake has joined #openstack-infra | 20:16 | |
| mrmartin | how difficult could be to deploy a zuul / gerrit environment for local testing/development? | 20:16 |
| mrmartin | fungi: yes, snapshot can go to staging anytime, but prod must be linked for some git tags. | 20:17 |
| *** boris-42 has joined #openstack-infra | 20:17 | |
| fungi | mrmartin: others have done it and documented it fairly thoroughly... http://ci.openstack.org/running-your-own.html | 20:17 |
| fungi | mrmartin: so for the tarballs, our usual workflow on other projects is similar. we have per-branch tarballs that get replaced each time a commit is merged, so that they always reflect the tip of their respective branches, and then individual tarballs built from tagged commits which get kept around for ever | 20:18 |
| mrmartin | oh great, thank you, I'll try it, and tell you when I have a question, or something is ready. | 20:18 |
| fungi | mrmartin: we can also distinguish between pre-release and release version numbers in tags, and take different steps accordingly | 20:19 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add query for bug 1261182 https://review.openstack.org/65768 | 20:19 |
| *** hogepodge has quit IRC | 20:19 | |
| mrmartin | ok, first I want to do gating scripts and tarball creation. | 20:20 |
| fungi | mrmartin: sounds great | 20:21 |
| *** vipuls is now known as vipuls-away | 20:21 | |
| fungi | jog0: i re-promoted 65760,1 to reset the gate, since precise12's shooting spree was going to take out the next half dozen changes if it merged | 20:23 |
| *** yolanda has quit IRC | 20:25 | |
| *** briancli1e is now known as briancline | 20:25 | |
| *** jecarey has joined #openstack-infra | 20:26 | |
| fungi | jog0: http://logs.openstack.org/60/65760/1/check/gate-tempest-dsvm-neutron-large-ops/abcaa61/console.html and http://logs.openstack.org/60/65760/1/gate/gate-tempest-dsvm-large-ops/d20cfd6/console.html both ran on rax and succeeded, if you want to do any early evaluation on those | 20:26 |
| *** ryanpetrello has quit IRC | 20:27 | |
| *** alexpilotti has joined #openstack-infra | 20:27 | |
| *** rfolco has quit IRC | 20:28 | |
| *** malini has left #openstack-infra | 20:29 | |
| *** gothicmindfood has quit IRC | 20:30 | |
| *** eharney has quit IRC | 20:30 | |
| *** _david_ has quit IRC | 20:31 | |
| *** _david_ has joined #openstack-infra | 20:31 | |
| *** ryanpetrello has joined #openstack-infra | 20:33 | |
| *** ^d has joined #openstack-infra | 20:36 | |
| *** rakhmerov has joined #openstack-infra | 20:37 | |
| zaro | _david_: i haven't set that up yet. we will try it out though. | 20:38 |
| _david_ | zaro, that would be great, but it would surprise me, if t wouldn't work, because it works on gerrit-review ;-) | 20:39 |
| *** mrmartin has quit IRC | 20:40 | |
| zaro | _david_: hey, why is this invalid? | 20:40 |
| zaro | _david_: https://code.google.com/p/gerrit/issues/detail?id=2387 | 20:40 |
| _david_ | zaro, you claimed, that you can only change groups from UI and that this prevent you from automating project configuration. I explained in the description why i think it's not true. | 20:41 |
| *** rakhmerov has quit IRC | 20:42 | |
| zaro | _david_: yeah i see that. but my point to that bug is that you should be able do what i was describing without having to manually adding change owner to groups file. | 20:43 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add some documentation on wildcard limitations in queries https://review.openstack.org/65326 | 20:43 |
| zaro | _david_: i think it's a valid bug. | 20:43 |
| *** ^d has quit IRC | 20:45 | |
| _david_ | zaro, anyway i can change it to enhancement, or something but with current situation you can achieve what you want: set up and configure new gerrit project from a python script or something by editing two files | 20:46 |
| zaro | _david_: sounds good to me. | 20:49 |
| _david_ | zaro, Done: you can may be clarify what you want there | 20:51 |
| *** denis_makogon_ has joined #openstack-infra | 20:52 | |
| *** denis_makogon_ is now known as denis_makogon | 20:52 | |
| *** dprince has quit IRC | 20:53 | |
| *** sandywalsh has quit IRC | 20:54 | |
| *** ^d has joined #openstack-infra | 20:54 | |
| *** ^d has joined #openstack-infra | 20:54 | |
| fungi | zaro: _david_: if that project.config gets pushed through gerrit's git interface, it will create the groups file reference automatically, right? (it does the same for any other group you mention in an acl) | 20:55 |
| _david_ | zaro, "if that project.config gets pushed through gerrit's git interface" you mean through UI? | 20:56 |
| zaro | _david_: yes, it works when using the UI. | 20:56 |
| fungi | _david_: through git+ssh (or whatever other protocols new gerrit supports) | 20:56 |
| fungi | _david_: right now, when we push project.config via gerrit's git service, gerrit is smart enough to create groups and groups file references | 20:57 |
| fungi | and it also syntax-checks the project.config and rejects the push if it's non-parseable | 20:58 |
| fungi | zaro: were you pushing locally on the filesystem instead? | 20:58 |
| _david_ | OK, i din't check the old version, only the master. | 20:58 |
| _david_ | on mater git push doesn't create reference in groups file, but the UI does, at least for system groups | 20:59 |
| zaro | fungi: no i have All-Projects cloned to my laptop. then pushing to review-dev.oo | 20:59 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add query for bug 1264755 https://review.openstack.org/65776 | 20:59 |
| fungi | zaro: so your git remote is ssh://zaro@review-dev.openstack.org:2xxx/blablah | 21:00 |
| fungi | and you're pushing to that? | 21:00 |
| zaro | fungi: yes, corret | 21:00 |
| fungi | zaro: yeah, then that does definitely do the right thing under 2.4 | 21:00 |
| *** dstanek has joined #openstack-infra | 21:01 | |
| fungi | we frequently add acl entries specifying new groups, or referring to existing groups not previously referenced in the acl, and gerrit adds the line to the groups file automatically | 21:01 |
| * fungi has to step away for just a moment... will brb | 21:01 | |
| *** hogepodge has joined #openstack-infra | 21:01 | |
| zaro | yeah. it would be really bad for upgrade situations where there are already lots of existing groups. | 21:01 |
| *** herndon has joined #openstack-infra | 21:02 | |
| *** SEJeff_work has quit IRC | 21:02 | |
| zaro | _david_: not to be too anal about this, but since it worked in 2.4 should work same in 2.8 so definately a bug and not an enhancement. | 21:03 |
| _david_ | zaro, how about to describe it this way? It sounds may be differently then ;-) | 21:04 |
| *** eharney has joined #openstack-infra | 21:05 | |
| anteaya | fungi: thanks for checking the ssh key, I will advise them | 21:06 |
| zaro | _david_: were you suggesting a new bug or just update to existing on? | 21:06 |
| _david_ | zaro i would suggest a new one. | 21:07 |
| _david_ | zaro, like fungi mentioned; can you verify that it is regression? | 21:07 |
| *** sandywalsh has joined #openstack-infra | 21:07 | |
| _david_ | and that not only related to the system groups? | 21:08 |
| anteaya | clarkb: the user in question is the motivation for http://lists.openstack.org/pipermail/openstack-dev/2014-January/023818.html | 21:08 |
| fungi | zaro: _david_: right, we should see whether adding an existing system group to an acl in 2.4.4 automatically adds an entry in that project's groups file | 21:08 |
| *** vipuls-away is now known as vipuls | 21:08 | |
| zaro | fungi: were you asking me to review the config change, https://review.openstack.org/65593, or the actual zmq change https://review.openstack.org/#/c/41814 ? | 21:08 |
| zaro | looks like 41814 already merged. | 21:09 |
| fungi | zaro: oh, right, you're right. i linked the wrong one, and it's already merged | 21:09 |
| fungi | zaro: good review! ;) | 21:09 |
| zaro | fungi: the commit message should probably link to review.o.o, no? | 21:10 |
| zaro | _david_, fungi : i can verify behavior on 2.4 | 21:10 |
| fungi | zaro: the commit message could just mention the change id in fact (I1cf2aee446c1e51c8eb15f7d84c3e828f3716cce) | 21:11 |
| zaro | _david_: i've already verified that in 2.8 you cannot assign any groups to permissions that are not in the groups file. | 21:11 |
| *** fbo is now known as fbo_away | 21:12 | |
| _david_ | zaro, to be more precisely, you can't do this through command line, OK | 21:12 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Drop large-ops test down to 50 instances from 100 https://review.openstack.org/65760 | 21:13 |
| fungi | jog0: ^ | 21:13 |
| zaro | _david_: that is correct, you cannot do it from cmd line unless you manually add the group to the groups file first. | 21:14 |
| fungi | jog0: (and a massive string of green behind it) | 21:14 |
| fungi | numerous passing large-ops tests | 21:14 |
| _david_ | zaro, are you on Master or on 2.8? | 21:14 |
| zaro | _david_: on 2.8 | 21:15 |
| _david_ | zaro, because /me is on master: https://gerrit-review.googlesource.com/#/c/52433/ | 21:16 |
| _david_ | this is the only big change on group system that i am aware of | 21:16 |
| jog0 | fungi: woot | 21:17 |
| *** jamielennox is now known as jamielennox|away | 21:17 | |
| zaro | _david_: ahh. that is a big change. | 21:18 |
| _david_ | zaro, that's the change dborowitz mentioned, as you've asked | 21:18 |
| zaro | _david_: however this probably won't fix it for upgrade situations. on upgrade existing groups are still in the db correct? | 21:18 |
| zaro | _david_: that would mean you still will not be able to assign non-system groups to permissions? | 21:19 |
| _david_ | zaro, you mean without changing the group manually? | 21:19 |
| zaro | _david_: yes | 21:20 |
| _david_ | zaro, let me check it | 21:20 |
| *** julim has quit IRC | 21:22 | |
| * zaro steps away, will brb | 21:23 | |
| *** sandywalsh has quit IRC | 21:27 | |
| *** praneshp has quit IRC | 21:28 | |
| *** praneshp has joined #openstack-infra | 21:30 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Add jenkins slave name to the logstash records https://review.openstack.org/65593 | 21:33 |
| *** dstanek has quit IRC | 21:35 | |
| *** sandywalsh has joined #openstack-infra | 21:41 | |
| *** thuc has quit IRC | 21:44 | |
| *** thomasem has quit IRC | 21:45 | |
| *** masayukig has joined #openstack-infra | 21:45 | |
| *** thuc has joined #openstack-infra | 21:45 | |
| *** DennyZhang has joined #openstack-infra | 21:47 | |
| *** herndon has quit IRC | 21:48 | |
| *** banix has quit IRC | 21:48 | |
| *** banix has joined #openstack-infra | 21:49 | |
| *** thuc has quit IRC | 21:49 | |
| _david_ | zaro, Have you verified against 2.4.2? | 21:50 |
| *** sarob has quit IRC | 21:50 | |
| zaro | _david_: not yet. will start in a few moments. | 21:50 |
| *** sarob has joined #openstack-infra | 21:51 | |
| *** DennyZhang has quit IRC | 21:53 | |
| *** sarob_ has joined #openstack-infra | 21:53 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: parse the failed jobs in stream https://review.openstack.org/65789 | 21:53 |
| _david_ | zaro, checked | 21:54 |
| *** Ajaeger has quit IRC | 21:54 | |
| _david_ | zaro, it always worked this way, and it is even documented: | 21:54 |
| _david_ | In order to reference a group in +project.config+, it must be listed in | 21:54 |
| _david_ | the +groups+ file. When editing permissions through the web UI this | 21:54 |
| _david_ | file is maintained automatically, but when pushing updates to | 21:54 |
| _david_ | +refs/meta/config+ this must be dealt with by hand. Gerrit will refuse | 21:54 |
| _david_ | +project.config+ files that refer to groups not listed in +groups+. | 21:54 |
| *** david_lyle has joined #openstack-infra | 21:55 | |
| *** lcestari has quit IRC | 21:55 | |
| *** sarob has quit IRC | 21:56 | |
| _david_ | zaro, https://gerrit-review.googlesource.com/Documentation/config-project-config.html#file-groups | 21:56 |
| *** nati_ueno has quit IRC | 21:56 | |
| *** dklyle has joined #openstack-infra | 21:57 | |
| _david_ | zaro, the code that refuses it: com.google.gerrit.server.git.validators.CommitValidators$ConfigValidator.onCommitReceived(CommitValidators.java:327) | 21:57 |
| *** nati_ueno has joined #openstack-infra | 21:59 | |
| *** david-lyle_ has quit IRC | 21:59 | |
| *** dklyle has quit IRC | 22:00 | |
| *** thuc has joined #openstack-infra | 22:00 | |
| *** david_lyle has quit IRC | 22:01 | |
| *** nati_ueno has quit IRC | 22:03 | |
| *** mfink has quit IRC | 22:04 | |
| *** nati_ueno has joined #openstack-infra | 22:04 | |
| fungi | _david_: zaro: i stand corrected! we automated that apparently... http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/manage_projects.py#n200 | 22:05 |
| zaro | _david_: thanks for checking. | 22:05 |
| * fungi apologizes profusely for the confusion | 22:05 | |
| _david_ | fungi, zaro n. p. | 22:06 |
| fungi | for some reason i really thought gerrit had magicked that into existence, but it looks like we just brute force it in with db queries and writes to the fs | 22:06 |
| _david_ | zaro, after thinking about it more, i think your enhancement request makes sense: harmonize the behavior between UI & git push | 22:08 |
| zaro | _david_: ++ | 22:08 |
| fungi | _david_: that's part of why i was confused. gerrit already does some spooky magic to syntax-check the projects.config when you git push it to refs/meta | 22:09 |
| fungi | so it clearly has somewhere this could get hooked in | 22:09 |
| _david_ | zaro, so may be opening a feature request is really a good idea, but as always with open source it would be even better to contribute a patch ;-) | 22:09 |
| _david_ | fungi, jepp | 22:10 |
| *** smarcet has quit IRC | 22:11 | |
| *** praneshp has quit IRC | 22:12 | |
| *** dims has quit IRC | 22:12 | |
| *** jgrimm has quit IRC | 22:13 | |
| *** CaptTofu has joined #openstack-infra | 22:14 | |
| zaro | _david_: of course :)) | 22:16 |
| _david_ | ;-) | 22:16 |
| *** mestery has quit IRC | 22:16 | |
| zaro | _david_: will review javamelody today. | 22:16 |
| _david_ | zaro, Have you noticed? It was merged | 22:17 |
| *** alexpilotti has quit IRC | 22:17 | |
| zaro | _david_: ohh wow, that was fast. awesome! | 22:17 |
| _david_ | zaro, so basically, forget github.com. The preference is gerrit-review | 22:18 |
| _david_ | ohh wow, that was fast.: that because /me have +2 ;-) | 22:18 |
| *** dims has joined #openstack-infra | 22:18 | |
| *** mestery has joined #openstack-infra | 22:19 | |
| zaro | haha! cores are powerful over there. | 22:21 |
| *** ^d has quit IRC | 22:21 | |
| *** gokrokve has joined #openstack-infra | 22:23 | |
| zaro | _david_: ohh i was confused by new gerrit ui. i was expecting a comment from gerrit that change change merged, but i guess that doesn't happen on new UI? or is it an option that can be turned off? | 22:25 |
| jeblair | fungi: morning; how's the large-ops issue? | 22:25 |
| fungi | jeblair: i just watched half a dozen changes in series pass the gate (including a couple of nova changes running pep8/docs/py27 jobs on nodepool bare-precise nodes), so i think we're in *much* better shape again | 22:27 |
| _david_ | zaro, don't quite understand you question, sorry | 22:27 |
| zaro | _david_: the h | 22:27 |
| fungi | jeblair: i need to step away for a moment to cook dinner, but have a series of changes to incrementally move more projects to bare-precise nodes if you think it's a good idea... see https://launchpad.net/bugs/1267364 | 22:28 |
| jeblair | fungi: cool, sorry about that. i do think you found the right solution though -- i think with the upcoming hpcloud changes we're going to need the full flexibility we have with multiple providers. | 22:28 |
| zaro | _david_: the "history" used to say ..changed was merged successfully. but not gerrit-review.o.o doesn't do that. | 22:28 |
| fungi | jeblair: i've been offlining precise nodes which go into rapid-fire fail and not bringing them back online (there have been 4 or five so far today, i lost count) | 22:28 |
| jeblair | fungi: i just realized my change to do that for nova is wrong -- it neglects consideration of OFFLINE_NODE_WHEN_COMPLETE | 22:28 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/devstack-gate: Allow concurrency to be tweaked for tempest https://review.openstack.org/65804 | 22:29 |
| jeblair | fungi: (so basically, i think we can't do 'precise || bare-precise' and have to completely switch a project at a time, along with a zuul change to set that value) | 22:29 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/devstack-gate: Cut tempest concurrency in half https://review.openstack.org/65805 | 22:29 |
| *** weshay has quit IRC | 22:29 | |
| jeblair | russellb: what's that for ^? | 22:29 |
| fungi | jeblair: i did see a couple of bare-precise nodes rapid-fire fail a handful of jobs within 3-5 seconds before they got deregistered... would that be the cause? | 22:29 |
| jeblair | fungi: yes | 22:30 |
| fungi | jeblair: okay, i'll rework my changes and fix nova after dinner | 22:30 |
| _david_ | zaro, yes, never bothered me | 22:30 |
| *** flaper87 is now known as flaper87|afk | 22:30 | |
| jeblair | fungi: i'll go ahead and propose the nova fi x | 22:30 |
| jeblair | fungi: enjoy dinner | 22:30 |
| zaro | _david_: not a problem. just that i've been traided to look at history for the merge. got me confused on new UI. | 22:31 |
| russellb | jeblair: yeah so ... a group of us have been diving deep into failures, and at least half of them are performance problems that we theorize would improve if we just toned down the test load | 22:31 |
| russellb | testing theories right now, no need to worry too much yet | 22:31 |
| jeblair | russellb: roger, thx! | 22:31 |
| fungi | thanks jeblair! | 22:33 |
| _david_ | zaro, i think its' not really old/new change UI related, only merge firehouse was shut down | 22:34 |
| *** mfer has quit IRC | 22:35 | |
| _david_ | ha gerrit-review seems to removed old change screen? | 22:37 |
| *** sparkycollier has quit IRC | 22:37 | |
| *** mriedem has quit IRC | 22:39 | |
| *** mfink has joined #openstack-infra | 22:40 | |
| *** ^d has joined #openstack-infra | 22:43 | |
| zaro | _david_: gone for good? | 22:43 |
| _david_ | zaro, that the question ;-) | 22:43 |
| zaro | _david_: did you ever push a change to secure monitoring? | 22:44 |
| _david_ | zaro, i've spent months writing this change: https://gerrit-review.googlesource.com/#/c/49601 | 22:44 |
| _david_ | it wasn't merged and the old change screen is disappeared ;-) | 22:44 |
| _david_ | zaro, nope, it's open | 22:44 |
| _david_ | zaro, i didn't do it for a reason: do you really think it is necessary? | 22:45 |
| *** rwsu has quit IRC | 22:45 | |
| *** thuc has quit IRC | 22:47 | |
| *** thuc has joined #openstack-infra | 22:47 | |
| zaro | _david_: i'm not real familiar with javamelody, but clarkb seems to think it's absolutely necessary. | 22:47 |
| *** kraman has quit IRC | 22:48 | |
| _david_ | zaro, OK, then we should definitely do that | 22:48 |
| zaro | _david_: apparently you can do stuff if it's left open. | 22:48 |
| *** kraman has joined #openstack-infra | 22:48 | |
| *** mfink has quit IRC | 22:48 | |
| _david_ | zaro what we are missing beside that? | 22:48 |
| _david_ | I think we are waiting for Gerrit 2.9? Right? | 22:48 |
| clarkb | zaro you can kill threads and stuff | 22:49 |
| clarkb | could be used to dos us or whatever | 22:49 |
| jeblair | or create a heap dump | 22:49 |
| zaro | _david_: i think what we absolutely needed was core plugins+monitoring. rquirements are in the etherpad: https://etherpad.openstack.org/p/gerrit-2.8-upgrade | 22:50 |
| *** rockyg has joined #openstack-infra | 22:50 | |
| zaro | _david_: the current thought is to upgrade to 2.8 first. | 22:51 |
| *** ryanpetrello has quit IRC | 22:51 | |
| *** praneshp has joined #openstack-infra | 22:52 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Move nova/savanna to only use bare-precise nodes https://review.openstack.org/65807 | 22:52 |
| *** kraman has quit IRC | 22:52 | |
| _david_ | zaro, I don' understand what you mean with plugins+monitoring | 22:53 |
| _david_ | and reading the link i don't understand that commit message: | 22:53 |
| _david_ | https://review.openstack.org/#/c/65368 | 22:53 |
| _david_ | zaro, first line of this document states: | 22:54 |
| _david_ | Gerrit 2.8 ships 4 core plugins, that must be installed to be full functional. Particularly replication and download-commands plugin are vital. There are number of ways how those plugins can be installed. Unattended mode is supported with --batch and --install-plugin foo option. This option must be provided multiply time to install all core plugins | 22:54 |
| *** nati_ueno has quit IRC | 22:54 | |
| zaro | _david_: yeah, i misunderstood that. i have abandoned the patch. | 22:54 |
| *** markmcclain has quit IRC | 22:54 | |
| zaro | _david_: try refreshing. | 22:55 |
| _david_ | zaro, ah, missed it, because it was listed... | 22:55 |
| _david_ | how? | 22:55 |
| zaro | _david_: because the current documentation states that it's --install-plugin is not supported. | 22:55 |
| _david_ | zaro where? | 22:56 |
| *** nati_ueno has joined #openstack-infra | 22:56 | |
| *** dcramer_ has quit IRC | 22:57 | |
| *** nati_ueno has quit IRC | 22:57 | |
| *** nati_ueno has joined #openstack-infra | 22:57 | |
| *** jorisroovers has quit IRC | 22:57 | |
| zaro | _david_: https://gerrit-review.googlesource.com/Documentation/config-auto-site-initialization.html | 22:58 |
| _david_ | zaro, it's the wrong link | 22:58 |
| _david_ | zaro, this is the right one: <zaro> _david_: i think what we absolutely needed was core plugins+monitoring. | 22:58 |
| _david_ | upps | 22:58 |
| _david_ | https://gerrit-review.googlesource.com/Documentation/pgm-init.html | 22:58 |
| _david_ | zaro, check --install-plugin option | 22:58 |
| *** DennyZhang has joined #openstack-infra | 22:58 | |
| *** thuc has quit IRC | 22:59 | |
| zaro | _david_: yeah. i got it now that's why i abandoned the change. | 22:59 |
| _david_ | anyway, what do you mean with "i think what we absolutely needed was core plugins+monitoring"? | 22:59 |
| sdague | fungi: https://review.openstack.org/#/c/63365/ | 22:59 |
| *** burt has quit IRC | 22:59 | |
| *** thuc has joined #openstack-infra | 22:59 | |
| sdague | that fixes a gate fail | 22:59 |
| *** UtahDave has quit IRC | 22:59 | |
| sdague | so next time there is a reset, popping it to the top of the list would be cool | 23:00 |
| jeblair | sdague: i will do it now | 23:00 |
| jeblair | sdague: the current head is a dead change walking | 23:00 |
| sdague | jeblair: thanks | 23:01 |
| *** sarob_ has quit IRC | 23:01 | |
| *** UtahDave has joined #openstack-infra | 23:01 | |
| zaro | _david_: i assume you mean content on the doc is incorrect. is there a patch to fix that? | 23:01 |
| *** sarob has joined #openstack-infra | 23:01 | |
| zaro | _david_: were you asking me about what else is needed for javamelody specifically? i thought you mean what's needed for us to upgrade. | 23:02 |
| jeblair | sdague: gah, it was a race with my command and jenkins result; i don't know which won. | 23:02 |
| _david_ | zaro, nope, this documentation you pointed to is correct, but its for something different! That why it called automatic: | 23:03 |
| *** DennyZhang has quit IRC | 23:03 | |
| jeblair | jenkins did, it seems. oh well. | 23:03 |
| _david_ | zaro, use case: you have nothing: no database, nothing. and you want to set up in one run. | 23:03 |
| _david_ | zaro, Has nothing to do with what are doing: we have everything | 23:04 |
| *** thuc has quit IRC | 23:04 | |
| _david_ | zaro i was asking what is needed fir us to upgrade ? | 23:04 |
| _david_ | not javamelody specific, but generally Also what? | 23:04 |
| *** sarob has quit IRC | 23:06 | |
| zaro | _david_: the gerrit 2.8 doc says.. "Installation of plugins during the site creation/initialization is not yet supported". you can do this right? | 23:06 |
| *** denis_makogon has quit IRC | 23:07 | |
| _david_ | zaro wrong, can you please totally ignore this document: https://gerrit-review.googlesource.com/Documentation/config-auto-site-initialization.html | 23:08 |
| zaro | _david_: so to upgrade, we need to answer all of the outstanding questions in the etherpad and then do additional testing. | 23:08 |
| *** jecarey has quit IRC | 23:08 | |
| zaro | _david_: we would definately like the core plugins to work and would also like to use the monitoring plugin as well. | 23:09 |
| _david_ | zaro, found it: https://gerrit-review.googlesource.com/#/c/46171/ | 23:10 |
| *** wenlock has quit IRC | 23:10 | |
| _david_ | zaro, this was the change that introduced this feature that you get confused, again has nothing to do with what we are doing | 23:11 |
| jeblair | sdague, fungi: turns out i haven't done it yet as it hasn't quite made it into the queue | 23:11 |
| *** jamielennox|away is now known as jamielennox | 23:13 | |
| * zaro wipes out memory of that doc. | 23:13 | |
| _david_ | zaro: Gerrit upgrade + Plugin installation must be done unattended, right? | 23:13 |
| sdague | jeblair: ok, well I'm about to step out for a bit. When you can, getting it in would be good | 23:13 |
| zaro | _david_: ohh, yeah that too. but i don't think that can be done for 1st upgrade. | 23:14 |
| jeblair | sdague: yep. i'll keep checking back. should be there in a few minutes | 23:14 |
| *** yamahata has quit IRC | 23:14 | |
| zaro | _david_: due to CLA configuration thing mentioned in the etherpad | 23:15 |
| _david_ | zaro, basically one command would do that: java -jar <path-to-gerrit.war> init -d <path-to-gerrit-site> --batch --install-plugin foo --install-plugin bar --install-plugin baz | 23:15 |
| _david_ | zaro, Exception that CAL problem that sounds like a bug to me | 23:16 |
| *** dstufft has quit IRC | 23:16 | |
| zaro | _david_: https://code.google.com/p/gerrit/issues/detail?id=2377 | 23:16 |
| *** dstufft has joined #openstack-infra | 23:17 | |
| *** rcleere has quit IRC | 23:19 | |
| *** senk has quit IRC | 23:19 | |
| _david_ | zaro, how that? it is adding some groups and some group not? | 23:19 |
| _david_ | to the database? | 23:19 |
| jeblair | sdague: fungi: 63365,5 is at the head | 23:20 |
| zaro | _david_: the upgrade does not add the "CLA Accepted - ICLA" to the db | 23:20 |
| sdague | jeblair: thanks | 23:20 |
| fungi | jeblair: in other news, mordred needs to get familiar with luks/dm-crypt | 23:20 |
| sdague | fungi: yeh, he wasn't luksed? | 23:21 |
| zaro | _david_: that is the problem. | 23:21 |
| jeblair | fungi: yeah, i saw that. :( | 23:21 |
| _david_ | zaro, may be i have misread your bug description | 23:21 |
| _david_ | you have three CLA sections right? | 23:21 |
| _david_ | with 3 different groups, right? | 23:22 |
| zaro | _david_: yes. | 23:22 |
| _david_ | now you are saying that only one groups was mssing in the database? | 23:22 |
| _david_ | and other two were correct inserted? | 23:22 |
| zaro | _david_: sorry i still can't get that doc out of my mind. since https://code.google.com/p/gerrit/issues/detail?id=2377 was merged why is that doc still referencable? | 23:23 |
| fungi | _david_: the other groups already existed. we have three clas, two of which are group-based and one of which was not. during upgrade the new group was added to the groups file in all-projects but not to the db | 23:23 |
| _david_ | fungi, zaro so i i would like to reproduce the problem with missing group: | 23:25 |
| _david_ | 1. set up 2.4.2 with CLA not group based. | 23:26 |
| _david_ | 2. upgrade to 2.8 | 23:26 |
| zaro | fungi, _david_ : what _david_ is saying is correct. 1 missing, other 2 were inserted correctly. | 23:26 |
| fungi | _david_: the utopian expectation is that the new group which got automatically added to the groups file in all-projects would also be added to the various related groups tables in the db, and the new group members would be populated based on a query against the account_agreements table | 23:26 |
| _david_ | 3. add new group to the all-projects | 23:26 |
| fungi | zaro: the other two groups were already in the database. those were group-based clas to begin with | 23:27 |
| _david_ | fungi so you have added a new CLA during migration? i still don't get it, sorry | 23:27 |
| zaro | fungi: ohh i see. | 23:28 |
| *** slong has joined #openstack-infra | 23:28 | |
| _david_ | if you only had two group based CLA in 2.4.2 then the migration was successful and all is fine? | 23:28 |
| zaro | _david_: this is what's in the db after migration: http://paste.openstack.org/show/60920/ | 23:28 |
| fungi | _david_: one of our clas, the problem one, was not a group-based cla. it was the sort which was enforced with autoverify which, in older gerrit, added an entry for each account_id to an account_agreements table in the db. in new gerrit this is replaced by a mechanism wherein autoverify adds accepted cla accounts to the indicated group | 23:29 |
| *** senk has joined #openstack-infra | 23:29 | |
| fungi | so it *seems* like there's a missing migration step there in the upgrade. we can script it as a follow-on, but it's not a seamless upgrade | 23:30 |
| zaro | fungi: probably would not be in puppet right? since it's just a 1 time deal. | 23:31 |
| _david_ | fungi, sorry, this is a wrong wrong way to treat the problem. The right was it to fix it upstream | 23:31 |
| _david_ | apply the patch or wait for the fix and make it seamless upgrade | 23:31 |
| fungi | _david_: i agree. that's why it's open as a bug | 23:32 |
| _david_ | fungi, i can not start to fix the problem based on this bug description | 23:32 |
| _david_ | what i need to fix it: | 23:32 |
| _david_ | Exact 10 steps to reproduce it | 23:32 |
| _david_ | 1.set up 2.4.2 Gerrit | 23:32 |
| _david_ | 2. set up non group based CLA: | 23:33 |
| _david_ | 2.1, 2.n | 23:33 |
| _david_ | 9 upgrade | 23:33 |
| _david_ | 10 Error | 23:33 |
| *** dkranz has quit IRC | 23:33 | |
| fungi | _david_: sure, the bug report could definitely use a more reproducible test case if the gerrit devs don't have intimate familiarity with changes they've made to the cla bits, no question | 23:33 |
| _david_ | fungi, the problem is: the dev that did that change not would be the same that is going to fix that problem. | 23:34 |
| fungi | _david_: i'm hoping one of us gets time to dig into gerrit source and identify the cause | 23:34 |
| zaro | sorry about that. i didn't know how we setup the CLA. | 23:34 |
| _david_ | the bug described working parts: CLAs that were correctly migrated | 23:34 |
| _david_ | zaro, i don't know it either. let us find it out and fix it | 23:35 |
| fungi | i saw that bug report as a placeholder, pending a more detailed report | 23:35 |
| *** UtahDave has quit IRC | 23:35 | |
| *** alexpilotti has joined #openstack-infra | 23:35 | |
| fungi | (and also in hopes someone would just pipe up with "oh, yeah, we fixed that on trunk" or something) | 23:36 |
| *** prad has quit IRC | 23:36 | |
| *** _david_ has quit IRC | 23:38 | |
| fungi | :( i ran him off | 23:38 |
| zaro | _david_: while your here. the Change Owner owner_group_id is set to 0 in the DB. that seems wrong to me. | 23:38 |
| *** mozawa has quit IRC | 23:39 | |
| zaro | darn! just when i was about to ask another question. | 23:39 |
| fungi | zaro: that doesn't seem wrong to me without more context, at least. in older gerrit, "administrators" is group_id 0 and owns most system groups | 23:41 |
| zaro | fungi: ohh, well now it makes total sense why the code assigned 0. but that may not work since admin group_id looks like it's 1 now. | 23:42 |
| *** CaptTofu has quit IRC | 23:43 | |
| fungi | oh, maybe it was admins=1 i was thinking of. not sure what 0 is in that case (maybe undefined? change owners wouldn't have a static list of accounts like a normal group anyway, right?) | 23:44 |
| *** CaptTofu has joined #openstack-infra | 23:45 | |
| zaro | fungi: nope. | 23:46 |
| zaro | fungi: not sure if i need to patch that up or not. | 23:46 |
| fungi | yeah, i'd be curious to hear the meaning of group_id=0 then, since the owner_group_id always seemed to correspond to another group_id in the table | 23:48 |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/devstack-gate: Enable Firewall as a Service plugin https://review.openstack.org/65822 | 23:49 |
| *** thuc has joined #openstack-infra | 23:54 | |
| *** vipuls is now known as vipuls-away | 23:56 | |
| *** jasondotstar has quit IRC | 23:56 | |
| *** vipuls-away is now known as vipuls | 23:58 | |
| jeblair | fungi: is 0 self-owned? | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!