| bknudson | keystone stable/grizzly gets "Installed distribution iso8601 0.1.4 conflicts with requirement iso8601>=0.1.8" ... | 00:00 |
|---|---|---|
| bknudson | update requirements in stable/grizzly? | 00:00 |
| bknudson | https://review.openstack.org/#/c/51973/ | 00:00 |
| jog0 | what just happened to gate neutron? | 00:01 |
| jog0 | clarkb: ^ | 00:01 |
| *** adarazs has joined #openstack-infra | 00:02 | |
| jog0 | its failure rate just spiked | 00:02 |
| *** jhesketh has quit IRC | 00:02 | |
| *** jhesketh has joined #openstack-infra | 00:02 | |
| harlowja | jeblair so some updates | 00:03 |
| harlowja | But the recipients will not be able to tell if the changes are essentially superficial (such as I have described) or substantial (the whole content has been replaced by something completely different such as a phishing email or similar). Cryptographic signatures are very fragile — they succeed or fail. | 00:03 |
| harlowja | The DKIM signature cannot be patched up by the mailing list (because it does not have access to the signing keys held by example.com), but of course the mailing list could apply its own signature. | 00:03 |
| *** jhesketh has quit IRC | 00:03 | |
| harlowja | However, what really matters is the From: header field — because DMARC (the method by which domain owners announce their cryptographic signature application policies and recommend how "forged" email should be dealt with) must be changed so it no longer says From: …@example.com because it's the From: line that says whose policy should be applied, and so the From: has to have the mailing list's domain. | 00:03 |
| harlowja | Once the mailing list has changed the From: then it's their choice whether to cryptographically sign or not (since the email is now From the mailing list and they can choose the policy). | 00:03 |
| harlowja | Turning the above generic description into practicality: | 00:03 |
| harlowja | If the mailing list rewrites email so that it says | 00:03 |
| harlowja | From: …@mailinglistexample.com | 00:03 |
| harlowja | Then everything will be fine. | 00:03 |
| harlowja | There is a patch for Mailman that does exactly this (soon a config option I believe). | 00:03 |
| jeblair | harlowja: that's a shame because that changes the meaning of the From: header. | 00:03 |
| harlowja | *seeing if i can get the location of said patch | 00:03 |
| *** julim has quit IRC | 00:04 | |
| harlowja | jeblair ya, it seems painful | 00:04 |
| *** MarkAtwood has quit IRC | 00:05 | |
| *** senk has joined #openstack-infra | 00:07 | |
| harlowja | jeblair https://mail.python.org/pipermail/mailman-announce/2013-July/000180.html (apparently see first new feature) | 00:10 |
| *** paul-- has joined #openstack-infra | 00:10 | |
| *** wenlock has quit IRC | 00:11 | |
| *** wenlock has joined #openstack-infra | 00:12 | |
| *** sdake_ has joined #openstack-infra | 00:14 | |
| *** sdake_ has quit IRC | 00:14 | |
| *** sdake_ has joined #openstack-infra | 00:14 | |
| *** jerryz has joined #openstack-infra | 00:14 | |
| zaro | pleia2: i want to make contribution to a project hosted on launchpad. can you help me with that? | 00:17 |
| zaro | pleia2: code is bzr | 00:17 |
| pleia2 | zaro: sure | 00:18 |
| *** branen_ has quit IRC | 00:21 | |
| fungi | harlowja: awesome... the next ubuntu lts looks like it will have 2.1.16 (2.1.16~rc2 is staged in it presently), so if we upgrade lists.o.o from ubuntu 12.04 lts to ubuntu 14.04 lts, we'll have support for author_is_list | 00:21 |
| fungi | ...sometime in the spring | 00:22 |
| pleia2 | zaro: this is a good place to start http://doc.bazaar.canonical.com/bzr.dev/en/mini-tutorial/ do "Introducing yourself" (much like git) | 00:22 |
| *** nati_uen_ has joined #openstack-infra | 00:23 | |
| pleia2 | zaro: then you `bzr branch lp:whatever-project` and make your changes in the directory, then: bzr commit -m 'Commit message' | 00:23 |
| pleia2 | zaro: and then you upload to your own spot on launchpad, so something like: bzr push ~zaro0508/project/fixes-bug-foo | 00:24 |
| pleia2 | zaro: last step is proposing it for merging in the master branch with: bzr lp-propose | 00:24 |
| *** markwash has quit IRC | 00:25 | |
| pleia2 | oh, that should be: bzr push lp:~zaro0508/project/fixes-bug-foo | 00:25 |
| *** nati_uen_ has quit IRC | 00:25 | |
| *** nati_uen_ has joined #openstack-infra | 00:26 | |
| *** nati_ueno has quit IRC | 00:27 | |
| pleia2 | workflow is a lot like github in that you fork (branch) and then do a pull request (merge proposal) | 00:27 |
| * pleia2 still thinks "merge proposal" makes more sense | 00:28 | |
| fungi | harlowja: jeblair: i have to say, i tend to agree with https://mail.python.org/pipermail/mailman-developers/2013-September/023302.html on the new from rewriting options though | 00:32 |
| *** matsuhashi has joined #openstack-infra | 00:32 | |
| fungi | to some extent, the 500 lb gorilla e-mail service providers will implement whatever dictates save them the most money, and will continue to edge out minorities (for example, people who have any interest in actually following ietf rfcs) unless enough of their clients (and probably only their paying clients) complain | 00:34 |
| fungi | though i would be in favor of setting whatever options prevent mailman from munging messages by way of footers et cetera, and only add informational headers instead | 00:36 |
| zaro | pleia2: do you need to be in a group that can lp-propose? | 00:37 |
| *** oubiwann has joined #openstack-infra | 00:38 | |
| zaro | pleia2: i mean with permissions to lp-propose? | 00:38 |
| pleia2 | zaro: nope | 00:38 |
| harlowja | fungi ya, i wonder how other ML are dealing with this (and what has worked out for them) | 00:38 |
| pleia2 | zaro: folks outside the group can just merge in, proposals are for folks on the outside | 00:38 |
| pleia2 | err | 00:38 |
| pleia2 | zaro: folks INside the group can just merge in, proposals are for folks on the outside | 00:38 |
| fungi | harlowja: i gather apache.org already configures their system not to alter messages, and to pass dkim headers through (so they should validate successfully?) | 00:39 |
| pleia2 | zaro: though I have seen some teams ask for proposals from everyone so there is some review (launchpad has poor man's code review :)) | 00:39 |
| zaro | pleia2: will proposal appear on my page or the project i'm proposing to? | 00:40 |
| pleia2 | zaro: both | 00:40 |
| zaro | pleia2: ok, cool thanks. | 00:40 |
| pleia2 | here's an example, I contributed some docs to bitlbee: https://code.launchpad.net/~lyz/bitlbee/quickstart-edits and you'll also see my MP on https://code.launchpad.net/bitlbee (it's still there because they actually host their code elsewhere, oops) | 00:41 |
| pleia2 | (in this case they actually manually moved my patch from lp over to their real code repo because they do accept via launchpad even though they actually host elsewhere) | 00:43 |
| pleia2 | crazies :) | 00:43 |
| fifieldt | anyone able to comment on my email to the openstack-infra list from a couple weeks ago? (http://permalink.gmane.org/gmane.comp.cloud.openstack.infrastructure/422) | 00:43 |
| *** pete5 has quit IRC | 00:44 | |
| *** gyee_ has joined #openstack-infra | 00:45 | |
| bknudson | we need a Tao of OpenStack ... http://www.ietf.org/tao.html | 00:45 |
| harlowja | fungi unsure, but in a few days i'll be DKIM enabled, so can be a guinea pig, lol | 00:45 |
| ekarlso | harlowja: dkim ? | 00:48 |
| *** yamahata_ has joined #openstack-infra | 00:48 | |
| harlowja | ekarlso http://www.dmarc.org/ | 00:48 |
| harlowja | apparently y! corp is turning this on (and many others) | 00:48 |
| harlowja | and it affects mailing lists afaik | 00:49 |
| mikal | jog0 / clarkb: is http://logstash.openstack.org/#eyJzZWFyY2giOiJmaWxlbmFtZTpjb25zb2xlLmh0bWwgQU5EIG1lc3NhZ2U6XCJub3QgYXR0YWNoZWQgdG8gZXgtdm9sLWluc3RcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4NTQyNzAyNTkyOX0= too infrequent to report as a tempest bug? | 00:51 |
| ekarlso | harlowja: k | 00:53 |
| jog0 | mikal: nothing is to infrequent | 00:54 |
| jog0 | mikal: except we haven't tought e-r about grenade yet | 00:55 |
| jog0 | so it will only show up on status.o.o/e-r and not the bot | 00:55 |
| *** llu has joined #openstack-infra | 00:55 | |
| jog0 | mikal: but I have some other bugs for you if you want | 00:56 |
| jog0 | another libvirt error | 00:56 |
| mikal | jog0: I am on vacation this week, so I should probably minimize the number of bugs I take | 00:57 |
| mikal | jog0: setting expectations and all | 00:57 |
| mikal | jog0: I've reported that grenade failure, including the logstash search | 00:57 |
| mikal | jog0: and then did a recheck | 00:57 |
| mikal | jog0: (this is me trying to never do a "recheck no bug") | 00:57 |
| jog0 | mikal: mikal https://bugs.launchpad.net/nova/+bug/1254872 | 00:58 |
| uvirtbot | Launchpad bug 1254872 in nova "libvirtError: Timed out during operation: cannot acquire state change lock" [Critical,Triaged] | 00:58 |
| jog0 | mikal: awesome souce | 00:58 |
| jog0 | your a model dev | 00:58 |
| mikal | jog0: its true, but that's mostly because of my fantastic hair | 00:58 |
| mikal | :P | 00:58 |
| jog0 | sauce* | 00:58 |
| jog0 | :P | 00:58 |
| mikal | jog0: is it just me or is the merge queue getting out of hand again? | 01:00 |
| mikal | jog0: 60 something changes at the moment | 01:00 |
| mikal | jog0: oh, nodepool is going bonkers too? | 01:00 |
| *** atiwari has quit IRC | 01:01 | |
| *** senk has quit IRC | 01:02 | |
| *** guohliu has quit IRC | 01:03 | |
| fungi | mikal: the change at the head of the main integrated queue is on;y about 5.5 hours since approval. not *too* bad. nodepool churn like that, as we've discovered, starts to become common when the queue gets that long, since any reset high up the queue cancels enough running jobs that it's pretty much turning over the entire pool and spinning up hundreds of new nodes in one go | 01:03 |
| mikal | fungi: five hours seems like a long time to me, but perhaps its because I am impatient | 01:04 |
| mikal | fungi: so we don't believe that we're doomed again? | 01:04 |
| jeblair | fungi: 5.5 hours sounds like forever. also the sparkline is going up, not down. | 01:04 |
| lifeless | fungi: wouldn't more nodes help? | 01:04 |
| jeblair | it would help us use more nodes, not help us merge changes faster | 01:04 |
| fungi | i take it as a sign weve still got nondeterministic failures on a notable percentage of the changes in the gate | 01:05 |
| jog0 | mikal: its nuetron | 01:05 |
| lifeless | jeblair: we only speculate about one side of each branch in the pipeline? | 01:05 |
| mikal | jog0: this is unsurprising to me | 01:05 |
| jog0 | mikal: see http://paste.openstack.org/show/53765/ | 01:06 |
| jog0 | neutron broke | 01:06 |
| jog0 | or so it seems | 01:07 |
| lifeless | thats certainly a spike | 01:08 |
| fungi | certainly seems to have spiked, agreed | 01:08 |
| *** mrodden has quit IRC | 01:09 | |
| *** nosnos has joined #openstack-infra | 01:09 | |
| jog0 | there are some enturon fixes in the queue so hopefully that will help | 01:09 |
| fungi | it might be interesting to statistically model gate throughput vs job failure percentage, to have an idea of how many changes an hour we can anticipate merging at a given nondeterministic failure rate | 01:10 |
| jog0 | fungi: yeah | 01:10 |
| mikal | I wonder if we could do some alerting off of that data as well? | 01:10 |
| mikal | Perhaps not a pager, but an automated heads up that things are grim | 01:10 |
| jog0 | mikal: consider yourself alerted | 01:10 |
| mikal | jog0: you are a carbon based meat monitoring machine | 01:10 |
| fungi | though available pool exhaustion makes that more complex to calcutale after a certain threshold, i suspect | 01:10 |
| *** wenlock_ has quit IRC | 01:11 | |
| fungi | around about the point where the length of the queue times the average nodepool-slave-consuming jobs per change is roughly equat to the overall pool capacity | 01:12 |
| jog0 | fungi: there is also the question of how far ahead to tset gate jobs | 01:12 |
| jog0 | it may make more sense to put workers in check queue | 01:12 |
| mikal | Ok, so I think my delayed flight will board soon, I should go | 01:12 |
| lifeless | mikal: bai | 01:12 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Run df after gate https://review.openstack.org/56298 | 01:13 |
| *** jorisroovers has quit IRC | 01:14 | |
| jog0 | we are up to 11:30AM | 01:14 |
| *** nati_ueno has joined #openstack-infra | 01:14 | |
| jog0 | PST time | 01:14 |
| jog0 | for merges | 01:14 |
| fungi | i worry that starting to decide not to test more than x number of jobs at a time because it exceeds the anticipated failure chance takes this from being an optimistic scheduler to a pessimistic scheduler | 01:14 |
| jog0 | fungi: heh yeah | 01:14 |
| jog0 | just with check jobs pending | 01:15 |
| jog0 | and running test for 20+deep gate jobs | 01:15 |
| jog0 | lifeless: https://review.openstack.org/#/c/51735/ woot | 01:15 |
| *** jorisroovers has joined #openstack-infra | 01:17 | |
| *** nati_uen_ has quit IRC | 01:17 | |
| lifeless | jog0: cool! https://review.openstack.org/#/c/52704/ :( | 01:17 |
| jog0 | lifeless: yeah just noticed too | 01:17 |
| lifeless | jog0: + the one above that which we need too :) | 01:18 |
| *** mriedem has joined #openstack-infra | 01:18 | |
| jog0 | lifeless: https://review.openstack.org/#/c/57347/ | 01:18 |
| jog0 | you may want to chime in | 01:18 |
| openstackgerrit | Christopher Yeoh proposed a change to openstack-infra/devstack-gate: TEMPEST_LARGE_OPS_NUMBER tuning https://review.openstack.org/58422 | 01:19 |
| *** rongze has joined #openstack-infra | 01:21 | |
| fungi | clarkb: are you satisfied with mordred's answer on https://review.openstack.org/#/c/50400/1 (i'm okay with it, but didn't want to approve without checking with you) | 01:22 |
| *** gyee_ has quit IRC | 01:23 | |
| lifeless | jog0: chimed in. | 01:24 |
| lifeless | jog0: I agree with sdague (but you knew that already :)) | 01:24 |
| lifeless | however, it won't hurt, as long as we're not squeezed on capacity. | 01:24 |
| *** herndon has quit IRC | 01:24 | |
| lifeless | what I'd really like is to harvest all the failing checks to find common aspects and turn those into gate fail signal quickly | 01:25 |
| *** mrodden has joined #openstack-infra | 01:25 | |
| *** rongze has quit IRC | 01:26 | |
| jog0 | lifeless: I agree with that assessment. I don' tthink it will helpin any significant way, but as long as we are not resource constrained it can't hurt | 01:26 |
| *** thomasem has joined #openstack-infra | 01:27 | |
| jog0 | lifeless: we already classify all failures or at least try to | 01:27 |
| jog0 | not sure what you mean | 01:27 |
| *** mrodden1 has joined #openstack-infra | 01:27 | |
| lifeless | jog0: there are many more checks run than gates (except when the gate is bust) | 01:27 |
| jog0 | lifeless: we track both check and gate in e-r | 01:27 |
| lifeless | jog0: assume that there were no rechecks - that we had found them all | 01:27 |
| lifeless | jog0: and then a new glitch entered the system. | 01:28 |
| *** sjing has joined #openstack-infra | 01:28 | |
| lifeless | jog0: whats the shortest time between it entering the system and us detecting it | 01:28 |
| jog0 | lifeless: agreed we want to improve that | 01:28 |
| jog0 | right now we just post to openstack-qa every time aunclassfied failure happens (in check or gate) | 01:28 |
| jog0 | we want to do better then that though | 01:29 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Update projects.yaml for new upstream tracking https://review.openstack.org/49957 | 01:29 |
| lifeless | jog0: so check failures are interesting because they may be a) spurious or b) a patch failure | 01:29 |
| lifeless | jog0: how do you decide which is which? | 01:29 |
| jog0 | lifeless: so two parts | 01:30 |
| *** mrodden has quit IRC | 01:30 | |
| jog0 | if unit tests fail, we assume its a patch failure | 01:30 |
| jog0 | if not, we check by hand. a spot check takes about 30 seconds | 01:30 |
| lifeless | so I'm proposing an automated thing | 01:31 |
| *** jorisroovers has quit IRC | 01:31 | |
| lifeless | that diffs two failed checks | 01:31 |
| lifeless | + a good check | 01:31 |
| lifeless | and generates a pattern that would pick up the two failed but not the good | 01:31 |
| lifeless | and then we review that | 01:31 |
| lifeless | pick two checks from different trees, e.g. nova change + neutron change | 01:32 |
| jog0 | go on ... not sure if I follow yet but I like anything that automates this more | 01:33 |
| lifeless | so if the two failures have the same symptom | 01:33 |
| jog0 | fungi: http://status.openstack.org/zuul/ makes no sense | 01:33 |
| lifeless | the pattern will be narrow | 01:33 |
| jog0 | gate has two pending jobs at the top | 01:33 |
| lifeless | if they don't, the pattern will be wide | 01:33 |
| jog0 | err queued | 01:33 |
| lifeless | pair off new failures against all known patterns, -> unique narrow pattern | 01:34 |
| fungi | jog0: makes no sense how? | 01:34 |
| jog0 | lifeless: ohh I see | 01:34 |
| fungi | jog0: oh, that | 01:34 |
| lifeless | jog0: or variations on a theme | 01:35 |
| jog0 | lifeless: so to make sure I Fully grok: | 01:35 |
| lifeless | jog0: basically, diff good, lets use. | 01:35 |
| fungi | jog0: not sure, but i think a large swath of changes merged and you're now seeing some that have newly arrived at the head of the queue while some of their jobs haven't been assigned available nodes yet | 01:35 |
| portante | clarkb: where would I file suggestions for the zuul status page? | 01:35 |
| fungi | there is a dip in the sparkline for the gate pipeline | 01:35 |
| jog0 | when a new failure in check comes in its hard to tell if valid or not. If we see the same failure again on a different patch its safer to say its a valid bug | 01:35 |
| jog0 | so while I really like this, the issue at the moment is not: too many unknown bugs | 01:37 |
| jog0 | its too many known bugs | 01:37 |
| jeblair | fungi, jog0: it's possible there were worker failures on those jobs which is whey they ended up at the back of the queue (but at least it didn't fail the change) | 01:38 |
| jog0 | which is to say I like the idea and it would help but its only part | 01:38 |
| *** rongze has joined #openstack-infra | 01:39 | |
| fungi | jeblair: worker failures meaning the specific impacted job doesn't report and gets restarted? maybe we're overdriving hpcloud again? | 01:39 |
| lifeless | jog0: okie | 01:39 |
| lifeless | jog0: so, mark em all critical :> | 01:39 |
| jeblair | fungi: er, how does overdriving hpcloud cause that to happen? | 01:39 |
| fungi | jeblair: no idea... what is a worker failure? terminology i haven't encountered yet in this context | 01:40 |
| jog0 | lifeless: ++ | 01:41 |
| jog0 | I am trying that | 01:41 |
| jeblair | fungi: the jenkins node. | 01:41 |
| jog0 | lifeless: can you file a bug or something with your idea | 01:41 |
| jog0 | so we have a record of it | 01:41 |
| jog0 | I like it | 01:41 |
| jeblair | fungi: usually it's jenkins being jenkins or hpcloud being hpcloud. not load related as far as i've seen, except just by frequency of occurence. | 01:41 |
| *** thomasem has quit IRC | 01:41 | |
| lifeless | jog0: list post isn't enough :) | 01:42 |
| lifeless | jog0: this isn't a project bug, its a social consensus. | 01:42 |
| *** datsun180b has quit IRC | 01:42 | |
| fungi | jeblair: oh, for example a slave gets marked offline according to the jenkins master in the middle of running an assigned job? | 01:42 |
| lifeless | jog0: -> TC meeting | 01:42 |
| jog0 | lifeless: a bug for your check failure idea | 01:43 |
| lifeless | jog0: oh, ack. Where? | 01:43 |
| jog0 | lifeless: not sure really | 01:43 |
| jog0 | maybe on ifra? | 01:43 |
| jog0 | and assign to me | 01:43 |
| jeblair | fungi: yeah | 01:44 |
| jog0 | lifeless: speaking of which https://bugs.launchpad.net/tempest/+bug/1253896 | 01:45 |
| uvirtbot | Launchpad bug 1253896 in tempest "test_minimum_basic_scenario fails with SSHException: Error reading SSH protocol banner" [Critical,Confirmed] | 01:45 |
| jog0 | thats a big one | 01:45 |
| *** mrodden has joined #openstack-infra | 01:45 | |
| fungi | i've wondered if we're not impacting performance/stability in hpcloud with the heavy create/delete churn on all these devstack nodes, but agreed we don't have any evidence to actually support that. and i agree it's more likely jenkins being jenkins given history | 01:45 |
| *** rongze has quit IRC | 01:46 | |
| *** jhesketh has joined #openstack-infra | 01:47 | |
| *** mrodden1 has quit IRC | 01:47 | |
| *** sandywalsh has quit IRC | 01:49 | |
| lifeless | jog0: you can self-assign | 01:50 |
| lifeless | https://bugs.launchpad.net/openstack-ci/+bug/1254939 | 01:50 |
| uvirtbot | Launchpad bug 1254939 in openstack-ci "assess checks for new failure patterns automatically" [Undecided,New] | 01:50 |
| fungi | nothing to jarring on cacti graphs for jenkins01 or 02, though 02 is running a little heavier than 01 | 01:52 |
| fungi | er, nothing too jarring | 01:52 |
| jog0 | sdague: https://bugs.launchpad.net/openstack-ci/+bug/1254939 | 01:52 |
| uvirtbot | Launchpad bug 1254939 in openstack-ci "assess checks for new failure patterns automatically" [Undecided,New] | 01:52 |
| jog0 | lifeless: thanks | 01:52 |
| fungi | interesting that they're pumping around 40mbps out and 20mbps in each. i suppose that's artifact collection and publication primarily | 01:53 |
| *** rnirmal has quit IRC | 01:55 | |
| *** yaguang has joined #openstack-infra | 01:56 | |
| *** sarob has joined #openstack-infra | 01:56 | |
| jog0 | http://graphite.openstack.org/render/?from=-30days&width=1000&height=360&margin=0&target=color(stats.gauges.zuul.pipeline.gate.current_changes,%20'6b8182')&_t=0.9943263556342572#1385430729465 | 01:57 |
| jog0 | 30 day zuul gate queue length | 01:57 |
| jog0 | 90 day is even more interesting | 01:57 |
| jog0 | queues over 50 are not unheard of | 01:58 |
| lifeless | mm | 02:01 |
| lifeless | /rechecks could benefit by being sorted | 02:01 |
| portante | jog0, fungi: is there a place in the logs where all the checked out commit ids are stored for a given gate job? | 02:01 |
| lifeless | on either diminishing frequency or last-seen | 02:01 |
| *** rongze has joined #openstack-infra | 02:01 | |
| fungi | portante: i think you can see them mentioned in the setup log... confirming now | 02:02 |
| *** nosnos_ has joined #openstack-infra | 02:02 | |
| *** yamahata_ has quit IRC | 02:03 | |
| *** ljjjustin has joined #openstack-infra | 02:03 | |
| *** nati_uen_ has joined #openstack-infra | 02:03 | |
| fungi | portante: so it's not distilled, but you can see it about halfway down http://logs.openstack.org/99/53199/14/check/check-tempest-devstack-vm-full/0cf044f/logs/devstack-gate-setup-workspace-new.txt.gz for example | 02:04 |
| *** melwitt has quit IRC | 02:05 | |
| *** nosnos has quit IRC | 02:05 | |
| *** nati_ueno has quit IRC | 02:06 | |
| fungi | portante: here i assume you mean for diagnosing a failure (if you're interested to see what corresponding commits across projects a successfully merged commit represents, the openstack/openstack meta project tracks submodules for all the integrated project repos) | 02:06 |
| portante | okay | 02:06 |
| jeblair | lifeless: rechecks are sorted. they used to be sorted by frecency. sdague changed them to, i think, age. | 02:07 |
| portante | fungi: yes, I see a failure here: http://logs.openstack.org/84/58184/1/gate/gate-tempest-devstack-vm-full/2b21d48/logs/syslog.txt | 02:07 |
| portante | 02:07 | |
| portante | but the traceback leads me to believe this is old code involved, since it does not match the patch submitted on Nov. 22nd, even though the timestamp is 11/26 for this hit (see logstash search for 'message:"database is locked"') | 02:09 |
| *** nati_uen_ has quit IRC | 02:09 | |
| * fungi looks | 02:09 | |
| *** mriedem has quit IRC | 02:09 | |
| *** nati_ueno has joined #openstack-infra | 02:10 | |
| lifeless | jeblair: so if it's sorted by last seen | 02:10 |
| openstackgerrit | A change was merged to openstack-infra/config: Update projects.yaml for new upstream tracking https://review.openstack.org/49957 | 02:10 |
| lifeless | jeblair: we haven't seen any of them for a month | 02:10 |
| lifeless | jog0: ^ | 02:10 |
| fungi | portante: old swift code? HEAD is now at 7accddf Merge "CHANGELOG and AUTHORS updates for 1.10.0 release" | 02:11 |
| fungi | portante: that's according to the devstack-gate-setup-workspace-new.txt.gz log | 02:11 |
| fungi | portante: the change being tested was horizon https://review.openstack.org/58184 (a backport to their stable/havana branch) | 02:13 |
| fungi | portante: is it possible the "old code" to which you refer is expected in swift's stable/havana branch? | 02:14 |
| portante | so /opt/stack/new is stable/havana? | 02:14 |
| portante | hmmm | 02:14 |
| fungi | portante: looks like it to me | 02:14 |
| notmyname | fungi: portante: that would be the commit message for havana (ie swift 1.10.0) | 02:14 |
| portante | then i am lost as to what is being tested with this job | 02:15 |
| fungi | notmyname: sounds right then | 02:15 |
| *** dcramer_ has quit IRC | 02:18 | |
| *** bingbu has joined #openstack-infra | 02:18 | |
| *** nati_ueno has quit IRC | 02:21 | |
| portante | fungi: notmyname gave me the decoder ring, but now another question, how can I construct a logstash query for when build_ref contains "refs/zuul/master/" as a prefix? | 02:23 |
| *** reed has quit IRC | 02:29 | |
| *** guohliu has joined #openstack-infra | 02:30 | |
| *** ryanpetrello has joined #openstack-infra | 02:31 | |
| *** markwash has joined #openstack-infra | 02:32 | |
| fungi | portante: i would have thought build_ref:"refs/zuul/master/" or build_ref:"refs/zuul/master/.*" but i'm failing to figure out the wildcard syntax | 02:32 |
| portante | k thanks | 02:33 |
| fungi | looking for examples online but not sure if i should be looking up general lucene query syntax or something specific to kibana or what | 02:33 |
| fungi | maybe elastic-recheck repo has some examples | 02:33 |
| fungi | nevermind... i'm not finding any obvious wildcard use there either | 02:36 |
| portante | no problem, I'll find out another time, then | 02:36 |
| *** dims has quit IRC | 02:38 | |
| *** sarob has quit IRC | 02:40 | |
| *** sarob has joined #openstack-infra | 02:40 | |
| *** fifieldt has quit IRC | 02:41 | |
| *** sarob has quit IRC | 02:45 | |
| *** sarob has joined #openstack-infra | 02:45 | |
| fungi | portante: clarkb or jesusaurus might know... i can't tell if http://git.openstack.org/cgit/openstack-infra/config/tree/modules/kibana/templates/config.rb.erb#n133 means wildcards in specific fields should work or not | 02:45 |
| *** dolphm has joined #openstack-infra | 02:48 | |
| portante | fungi: thanks, though not encouraging. :) | 02:49 |
| jesusaurus | portante: fungi: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax | 02:51 |
| jesusaurus | tl;dr you need to escape the * with a \ | 02:51 |
| fungi | oho! | 02:52 |
| portante | cool | 02:52 |
| portante | 500 internal server error :) | 02:53 |
| jog0 | lifeless: the recheck server may have crashed | 02:54 |
| fungi | jesusaurus: trying build_ref:"refs/zuul/master/\*" (as well as several other combinations) in the search field on http://logstash.openstack.org/ with a long enough time range that it should match lots of entries, but i get a "no events matched" response. any ideas there? | 02:55 |
| jog0 | also we generally don't use that page, status.openstack.org/elastic-recheck/ is a lot more useful partially because we make sure everything added to it is areal bug | 02:55 |
| jesusaurus | also, queries should be substring matches if all you want is a trailing .\* | 02:55 |
| fungi | jesusaurus: is there maybe some special field type for build_ref which is causing it not to analyze as regular string content? | 02:56 |
| fungi | i see a lot of discussion on line about customizing field analyzers, which seems to affect whether substring searches work on them | 02:56 |
| lifeless | jog0: lol, so we need recheckrecheck? | 02:56 |
| *** ryanpetrello has quit IRC | 02:56 | |
| jog0 | lifeless: *nod* thats my guess | 02:57 |
| jog0 | an infra core can verify | 02:57 |
| fungi | jog0: theorizing that recheckwatch has died? i can take a look and possibly restart it | 02:57 |
| jesusaurus | fungi: maybe? is there an /etc/elasticsearch/default-mapping.json causing the string fields to not be analyzed? | 02:57 |
| jog0 | fungi: thanks | 02:58 |
| fungi | jesusaurus: it appears we install this there... https://git.openstack.org/cgit/openstack-infra/config/tree/modules/elasticsearch/files/elasticsearch.mapping.json | 03:01 |
| *** sarob has quit IRC | 03:01 | |
| jesusaurus | yeah, i just found that. | 03:02 |
| fungi | so it looks like @message/message are analyzed and otherwise no? | 03:02 |
| jesusaurus | its likely my fault that we aren't analyzing anything other than @message | 03:02 |
| *** sarob has joined #openstack-infra | 03:02 | |
| jesusaurus | analyzing the strings increases the time it takes to index documents into elasticsearch | 03:03 |
| *** slong has quit IRC | 03:04 | |
| fungi | then that makes sense that we aren't doing it. thanks jesusaurus! | 03:06 |
| jesusaurus | you're welcome | 03:07 |
| *** sarob has quit IRC | 03:07 | |
| fungi | jog0: so, the recheckwatch process on zuul has been running since october 23, but that day was the last time the html file was updated. stracing now to see if it's receiving any events at all | 03:07 |
| fungi | so far, nothing | 03:07 |
| *** matsuhashi has quit IRC | 03:11 | |
| *** gyee has quit IRC | 03:12 | |
| fungi | i wonder if its socket was dead. it doesn't seem to do any logging at all, so not much else there to diagnose. i stopped it and started it again, and its html and pickle files have current timestamps on them now | 03:13 |
| jog0 | fungi: now lets see if anything ages out soon | 03:13 |
| jog0 | I don't remember if the age out logic happens on restart or just new hit | 03:14 |
| jog0 | lifeless: http://status.openstack.org/rechecks/ | 03:23 |
| *** dolphm has quit IRC | 03:25 | |
| *** nati_ueno has joined #openstack-infra | 03:28 | |
| *** jerryz has quit IRC | 03:28 | |
| *** wenlock_ has joined #openstack-infra | 03:29 | |
| *** ^d has quit IRC | 03:32 | |
| *** dolphm has joined #openstack-infra | 03:33 | |
| *** nati_ueno has quit IRC | 03:35 | |
| *** nati_ueno has joined #openstack-infra | 03:35 | |
| *** sarob has joined #openstack-infra | 03:37 | |
| *** dolphm has quit IRC | 03:40 | |
| openstackgerrit | Liang Chen proposed a change to openstack-dev/pbr: Make sphinx builders configurable in LocalBuildDoc https://review.openstack.org/57892 | 03:55 |
| *** jorisroovers has joined #openstack-infra | 03:59 | |
| *** jroovers has joined #openstack-infra | 04:01 | |
| *** jorisroovers has quit IRC | 04:03 | |
| *** chandankumar has joined #openstack-infra | 04:05 | |
| *** yamahata_ has joined #openstack-infra | 04:14 | |
| *** matsuhashi has joined #openstack-infra | 04:19 | |
| *** senk has joined #openstack-infra | 04:20 | |
| lifeless | dhellmann-afk: dunno if you konw this but you can just move bug tasks around without marking them invalid | 04:22 |
| lifeless | dhellmann-afk: click on the edit icon beside the context and type in pbr, for instance. | 04:23 |
| dhellmann-afk | lifeless: didn't know that, thanks! | 04:24 |
| *** vipul-away is now known as vipul | 04:29 | |
| *** nati_uen_ has joined #openstack-infra | 04:29 | |
| *** nosnos_ has quit IRC | 04:32 | |
| *** nosnos has joined #openstack-infra | 04:32 | |
| *** nati_ueno has quit IRC | 04:33 | |
| *** nicedice has quit IRC | 04:42 | |
| *** yamahata_ has quit IRC | 04:47 | |
| *** rongze has quit IRC | 04:48 | |
| *** ArxCruz has quit IRC | 04:50 | |
| *** CaptTofu has quit IRC | 05:00 | |
| *** CaptTofu has joined #openstack-infra | 05:00 | |
| *** sarob has quit IRC | 05:02 | |
| *** zigo_ has quit IRC | 05:10 | |
| *** zigo has joined #openstack-infra | 05:11 | |
| *** pcrews has quit IRC | 05:14 | |
| *** rongze has joined #openstack-infra | 05:19 | |
| *** llu has left #openstack-infra | 05:20 | |
| *** oubiwann has quit IRC | 05:27 | |
| *** rongze has quit IRC | 05:27 | |
| *** sarob has joined #openstack-infra | 05:28 | |
| *** senk has quit IRC | 05:38 | |
| *** sarob has quit IRC | 05:39 | |
| *** matsuhashi has quit IRC | 05:50 | |
| *** matsuhashi has joined #openstack-infra | 05:51 | |
| *** matsuhas_ has joined #openstack-infra | 05:53 | |
| *** matsuhashi has quit IRC | 05:53 | |
| *** sarob has joined #openstack-infra | 06:07 | |
| *** paul-- has quit IRC | 06:07 | |
| *** michchap has quit IRC | 06:16 | |
| *** michchap has joined #openstack-infra | 06:16 | |
| *** yonglihe_ has joined #openstack-infra | 06:17 | |
| *** matsuhas_ has quit IRC | 06:20 | |
| *** yonglihe_ has quit IRC | 06:20 | |
| *** denis_makogon has joined #openstack-infra | 06:23 | |
| *** rongze has joined #openstack-infra | 06:24 | |
| *** rongze has quit IRC | 06:28 | |
| *** matsuhashi has joined #openstack-infra | 06:29 | |
| *** boris-42 has joined #openstack-infra | 06:29 | |
| *** yongli has quit IRC | 06:34 | |
| *** yongli has joined #openstack-infra | 06:35 | |
| *** rongze has joined #openstack-infra | 06:38 | |
| lifeless | mikal: btw I meant to ask | 06:41 |
| mikal | Why do you keep changing channels!?! | 06:41 |
| lifeless | mikal: I have in my head for some idea the idea that you have some lp bug scrounging script | 06:41 |
| mikal | Is this performance art? | 06:41 |
| Alex_Gaynor | The result queue has 2.4k results in it right now, that seems abnormally high | 06:41 |
| lifeless | mikal: because of context. | 06:41 |
| lifeless | mikal: I like to discuss things in the matching channel; its a thing. | 06:41 |
| lifeless | mikal: but I can do it to Bach, if you like! | 06:42 |
| mikal | We should create new channels for every possible conversation then | 06:42 |
| mikal | #snark | 06:42 |
| mikal | Nah, its fine | 06:42 |
| mikal | Just funny | 06:42 |
| lifeless | mikal: since I'm up to 110 or so windows now, why not? | 06:42 |
| lifeless | WCPGW | 06:42 |
| lifeless | mikal: so, do you? | 06:43 |
| mikal | Oh, sorry | 06:43 |
| mikal | I missed the question in all that | 06:43 |
| mikal | Yes, I have a script which does shitty LP bug scraping and draws graphs | 06:43 |
| lifeless | See, performance. | 06:43 |
| lifeless | mikal: vere is das kode? | 06:43 |
| mikal | http://openstack.stillhq.com/reviews/closesummary.html?reviewers=g:nova-core | 06:44 |
| mikal | The flat line may indicate it isn't working at the moment | 06:44 |
| mikal | In git I think | 06:44 |
| * mikal goes to look | 06:44 | |
| lifeless | indeed! | 06:44 |
| mikal | https://github.com/mikalstill/openstack-review-tools/blob/master/scrapebugs.py | 06:45 |
| *** mihgen has joined #openstack-infra | 06:47 | |
| lifeless | mikal: thanks | 06:47 |
| lifeless | sadly I think I still need to roll me own. STAB. | 06:47 |
| mikal | Oh, I am sure that code is horrible | 06:47 |
| mikal | The LP API doesn't help | 06:47 |
| Alex_Gaynor | Something is broken with all of the py26 builds | 06:48 |
| *** yongli has quit IRC | 06:48 | |
| Alex_Gaynor | mordred, fungi, clarkb: ^ anyone around? | 06:48 |
| *** yongli has joined #openstack-infra | 06:48 | |
| *** yongli has quit IRC | 06:50 | |
| *** yongli has joined #openstack-infra | 06:50 | |
| *** matsuhashi has quit IRC | 06:53 | |
| *** sarob has quit IRC | 06:54 | |
| *** matsuhashi has joined #openstack-infra | 06:54 | |
| *** nati_uen_ has quit IRC | 06:56 | |
| *** wenlock_ has quit IRC | 06:57 | |
| *** matsuhashi has quit IRC | 06:58 | |
| *** matsuhashi has joined #openstack-infra | 06:59 | |
| *** xeyed4good has joined #openstack-infra | 06:59 | |
| *** Ryan_Lane has joined #openstack-infra | 07:02 | |
| *** sjing has quit IRC | 07:04 | |
| *** xeyed4good has quit IRC | 07:04 | |
| *** sjing has joined #openstack-infra | 07:06 | |
| *** yongli has quit IRC | 07:06 | |
| *** Ryan_Lane has quit IRC | 07:07 | |
| *** yongli has joined #openstack-infra | 07:07 | |
| *** dstanek has quit IRC | 07:09 | |
| *** nati_ueno has joined #openstack-infra | 07:12 | |
| *** afazekas has joined #openstack-infra | 07:13 | |
| *** matsuhashi has quit IRC | 07:26 | |
| *** matsuhashi has joined #openstack-infra | 07:27 | |
| *** yongli has quit IRC | 07:35 | |
| *** yongli has joined #openstack-infra | 07:35 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:38 | |
| *** matsuhas_ has joined #openstack-infra | 07:39 | |
| *** matsuhashi has quit IRC | 07:42 | |
| *** boris-42 has quit IRC | 07:42 | |
| *** Grishkin has quit IRC | 07:43 | |
| *** evgeniyl has quit IRC | 07:45 | |
| *** Grishkin has joined #openstack-infra | 07:45 | |
| *** evgeniyl has joined #openstack-infra | 07:45 | |
| *** che-arne has quit IRC | 07:45 | |
| *** boris-42 has joined #openstack-infra | 07:45 | |
| *** katyafervent has quit IRC | 07:46 | |
| *** katyafervent has joined #openstack-infra | 07:47 | |
| *** yolanda has joined #openstack-infra | 07:49 | |
| *** DinaBelova has joined #openstack-infra | 07:53 | |
| *** pblaho has joined #openstack-infra | 07:59 | |
| *** matsuhas_ has quit IRC | 08:05 | |
| *** matsuhashi has joined #openstack-infra | 08:05 | |
| *** SergeyLukjanov has quit IRC | 08:06 | |
| *** jcoufal has joined #openstack-infra | 08:07 | |
| *** DinaBelova has quit IRC | 08:09 | |
| *** flaper87|afk is now known as flaper87 | 08:10 | |
| *** mihgen has quit IRC | 08:11 | |
| *** dizquierdo has joined #openstack-infra | 08:14 | |
| *** sarob has joined #openstack-infra | 08:20 | |
| *** llu has joined #openstack-infra | 08:23 | |
| *** sarob has quit IRC | 08:24 | |
| *** nati_ueno has quit IRC | 08:29 | |
| llu | anyone knows why recheck doesn't work? | 08:32 |
| llu | https://review.openstack.org/58211 | 08:33 |
| *** paul-- has joined #openstack-infra | 08:33 | |
| *** hashar has joined #openstack-infra | 08:38 | |
| *** denis_makogon has quit IRC | 08:39 | |
| llu | ok, it works again. maybe web UI delay? | 08:44 |
| *** llu has left #openstack-infra | 08:45 | |
| *** osanchez has joined #openstack-infra | 08:45 | |
| *** yongli has quit IRC | 08:46 | |
| *** yongli has joined #openstack-infra | 08:47 | |
| *** sarob has joined #openstack-infra | 08:50 | |
| *** sarob has quit IRC | 08:52 | |
| *** sarob has joined #openstack-infra | 08:52 | |
| *** paul-- has quit IRC | 08:52 | |
| *** sjing has quit IRC | 08:53 | |
| *** derekh has joined #openstack-infra | 08:54 | |
| *** mihgen has joined #openstack-infra | 08:55 | |
| *** sarob has quit IRC | 08:57 | |
| *** mihgen has quit IRC | 09:00 | |
| *** mihgen has joined #openstack-infra | 09:00 | |
| *** Ryan_Lane has joined #openstack-infra | 09:03 | |
| *** yonglihe_ has joined #openstack-infra | 09:05 | |
| *** fbo_away is now known as fbo | 09:06 | |
| *** yonglihe_ has quit IRC | 09:06 | |
| *** yonglihe_ has joined #openstack-infra | 09:07 | |
| *** yonglihe_ has quit IRC | 09:07 | |
| *** yongli has quit IRC | 09:07 | |
| *** yongli has joined #openstack-infra | 09:07 | |
| *** Ryan_Lane has quit IRC | 09:08 | |
| *** paul-- has joined #openstack-infra | 09:09 | |
| yaguang | what's wrong with gate server ? a lot of gate python2.6 tests fails eg. http://logs.openstack.org/81/58281/1/gate/gate-nova-python26/bbf88b0/console.html | 09:09 |
| *** nati_ueno has joined #openstack-infra | 09:09 | |
| *** yassine has joined #openstack-infra | 09:10 | |
| lifeless | yaguang: yes, it seems to have died at the time when we have no ops available to fix | 09:11 |
| *** jpich has joined #openstack-infra | 09:11 | |
| *** yongli has quit IRC | 09:13 | |
| *** yongli has joined #openstack-infra | 09:13 | |
| *** bingbu has quit IRC | 09:16 | |
| yongli | : | 09:17 |
| *** paul-- has quit IRC | 09:23 | |
| *** ruhe has joined #openstack-infra | 09:25 | |
| *** paul-- has joined #openstack-infra | 09:27 | |
| *** yamahata_ has joined #openstack-infra | 09:30 | |
| *** ljjjustin has quit IRC | 09:33 | |
| *** jhesketh has quit IRC | 09:40 | |
| *** nati_ueno has quit IRC | 09:40 | |
| *** sarob has joined #openstack-infra | 09:50 | |
| *** DinaBelova has joined #openstack-infra | 09:53 | |
| *** paul-- has quit IRC | 09:55 | |
| *** paul-- has joined #openstack-infra | 09:57 | |
| *** matsuhas_ has joined #openstack-infra | 09:58 | |
| *** matsuhas_ has quit IRC | 09:58 | |
| *** matsuhashi has quit IRC | 10:02 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:02 | |
| *** Ryan_Lane has joined #openstack-infra | 10:03 | |
| *** Ryan_Lane has quit IRC | 10:08 | |
| *** jcoufal has quit IRC | 10:13 | |
| *** DinaBelova has quit IRC | 10:18 | |
| *** paul-- has quit IRC | 10:19 | |
| *** ken1ohmichi has joined #openstack-infra | 10:21 | |
| *** sarob has quit IRC | 10:22 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Add Savanna testing support https://review.openstack.org/57325 | 10:24 |
| *** DinaBelova has joined #openstack-infra | 10:25 | |
| *** che-arne has joined #openstack-infra | 10:27 | |
| *** rongze has quit IRC | 10:29 | |
| *** bauzas has joined #openstack-infra | 10:34 | |
| bauzas | https://bugs.launchpad.net/openstack-ci/+bug/1255041 open | 10:36 |
| uvirtbot | Launchpad bug 1255041 in openstack-ci "Python 2.6 gate broken due to no workspace found" [Undecided,New] | 10:36 |
| bauzas | for py26 checks | 10:36 |
| *** yamahata_ has quit IRC | 10:36 | |
| *** paul-- has joined #openstack-infra | 10:42 | |
| *** markmc has joined #openstack-infra | 10:46 | |
| *** lcestari has joined #openstack-infra | 10:48 | |
| *** sarob has joined #openstack-infra | 10:50 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Setup devstack-gate tests for Savanna https://review.openstack.org/57317 | 10:52 |
| *** jroovers has quit IRC | 10:54 | |
| *** sarob has quit IRC | 10:54 | |
| *** paul-- has quit IRC | 10:58 | |
| *** paul-- has joined #openstack-infra | 11:02 | |
| *** paul-- has quit IRC | 11:02 | |
| *** Ryan_Lane has joined #openstack-infra | 11:03 | |
| *** yaguang has quit IRC | 11:05 | |
| *** guohliu has quit IRC | 11:07 | |
| *** ruhe has quit IRC | 11:07 | |
| *** Ryan_Lane has quit IRC | 11:08 | |
| *** paul-- has joined #openstack-infra | 11:11 | |
| *** rfolco has joined #openstack-infra | 11:11 | |
| *** paul-- has quit IRC | 11:12 | |
| *** wenlock has quit IRC | 11:14 | |
| *** ken1ohmichi has quit IRC | 11:19 | |
| *** paul-- has joined #openstack-infra | 11:20 | |
| *** paul-- has quit IRC | 11:22 | |
| *** paul-- has joined #openstack-infra | 11:24 | |
| *** paul-- has quit IRC | 11:26 | |
| *** jorisroovers has joined #openstack-infra | 11:26 | |
| *** jorisroovers has quit IRC | 11:27 | |
| *** jorisroovers has joined #openstack-infra | 11:27 | |
| ttx | #status notice Gate wedged - Most Py26 jobs fail currently (https://bugs.launchpad.net/openstack-ci/+bug/1255041) | 11:29 |
| openstackstatus | NOTICE: Gate wedged - Most Py26 jobs fail currently (https://bugs.launchpad.net/openstack-ci/+bug/1255041) | 11:29 |
| uvirtbot | Launchpad bug 1255041 in openstack-ci "Python 2.6 gate broken due to no workspace found" [Undecided,New] | 11:29 |
| ttx | fungi, mordred: ^for when you get up | 11:29 |
| *** ArxCruz has joined #openstack-infra | 11:30 | |
| *** mkerrin has quit IRC | 11:35 | |
| *** pcm_ has joined #openstack-infra | 11:41 | |
| *** jroovers has joined #openstack-infra | 11:44 | |
| *** michchap has quit IRC | 11:46 | |
| *** jorisroovers has quit IRC | 11:46 | |
| *** CaptTofu has quit IRC | 11:47 | |
| *** CaptTofu has joined #openstack-infra | 11:47 | |
| *** michchap has joined #openstack-infra | 11:48 | |
| *** saschpe_ has joined #openstack-infra | 11:49 | |
| *** saschpe has quit IRC | 11:50 | |
| *** sarob has joined #openstack-infra | 11:50 | |
| *** zoresvit has quit IRC | 11:50 | |
| *** DinaBelova has quit IRC | 11:51 | |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Print help message and exit if no config file by default https://review.openstack.org/51501 | 11:51 |
| *** jcoufal has joined #openstack-infra | 11:53 | |
| *** pcm_ has quit IRC | 11:55 | |
| *** pcm_ has joined #openstack-infra | 11:56 | |
| *** Ryan_Lane has joined #openstack-infra | 12:03 | |
| *** mkerrin has joined #openstack-infra | 12:04 | |
| *** DinaBelova has joined #openstack-infra | 12:05 | |
| *** Ryan_Lane has quit IRC | 12:08 | |
| *** flaper87 is now known as flaper87|afk | 12:11 | |
| *** nosnos has quit IRC | 12:11 | |
| *** sarob has quit IRC | 12:12 | |
| *** michchap has quit IRC | 12:13 | |
| *** yolanda has quit IRC | 12:14 | |
| *** yolanda has joined #openstack-infra | 12:14 | |
| *** rongze has joined #openstack-infra | 12:15 | |
| *** Guest46324 is now known as med_ | 12:15 | |
| *** med_ is now known as medberry | 12:15 | |
| *** flaper87|afk is now known as flaper87 | 12:16 | |
| *** yongli has quit IRC | 12:17 | |
| *** hashar has quit IRC | 12:17 | |
| *** mihgen has quit IRC | 12:26 | |
| *** CaptTofu has quit IRC | 12:26 | |
| *** CaptTofu has joined #openstack-infra | 12:27 | |
| *** paul-- has joined #openstack-infra | 12:27 | |
| *** hashar has joined #openstack-infra | 12:29 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Ensure jobparams and group_jobparams are dict https://review.openstack.org/57525 | 12:31 |
| *** CaptTofu has quit IRC | 12:31 | |
| *** sarob has joined #openstack-infra | 12:32 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Added support for Stash Notifier https://review.openstack.org/56337 | 12:33 |
| *** mihgen has joined #openstack-infra | 12:33 | |
| *** sarob has quit IRC | 12:37 | |
| *** CaptTofu has joined #openstack-infra | 12:39 | |
| openstackgerrit | QianLin proposed a change to openstack-infra/nodepool: Fix the issue which raise FloatingIpPoolNotFound https://review.openstack.org/54331 | 12:40 |
| *** boris-42 has quit IRC | 12:41 | |
| *** smarcet has joined #openstack-infra | 12:41 | |
| *** michchap has joined #openstack-infra | 12:43 | |
| *** yamahata_ has joined #openstack-infra | 12:47 | |
| hashar | boooh | 12:49 |
| hashar | SEVERE ERROR occurs: java.io.IOException: Remote call on centos6-4 failed | 12:49 |
| hashar | looks like a slave died somehow | 12:50 |
| *** sarob has joined #openstack-infra | 12:50 | |
| *** michchap has quit IRC | 12:51 | |
| *** paul-- has quit IRC | 12:53 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Raise exceptions when no entrypoint, macro or template found https://review.openstack.org/49102 | 12:53 |
| *** sarob has quit IRC | 12:54 | |
| *** dims has joined #openstack-infra | 12:55 | |
| *** markmcclain has joined #openstack-infra | 12:56 | |
| *** paul-- has joined #openstack-infra | 12:57 | |
| *** michchap has joined #openstack-infra | 12:57 | |
| *** osanchez has quit IRC | 12:58 | |
| ekarlso | how can I get devstack to push notifications ? | 13:01 |
| *** dkranz has joined #openstack-infra | 13:01 | |
| *** michchap has quit IRC | 13:01 | |
| *** Ryan_Lane has joined #openstack-infra | 13:03 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Communicate to user when we skip a requirement https://review.openstack.org/54628 | 13:04 |
| *** sandywalsh has joined #openstack-infra | 13:07 | |
| *** bingbu has joined #openstack-infra | 13:07 | |
| *** Ryan_Lane has quit IRC | 13:08 | |
| *** paul-- has quit IRC | 13:10 | |
| *** yaguang has joined #openstack-infra | 13:11 | |
| *** dhellmann-afk is now known as dhellmann | 13:15 | |
| *** rongze has quit IRC | 13:17 | |
| *** weshay has joined #openstack-infra | 13:21 | |
| markmcclain | looks like we're getting random py26 failures | 13:24 |
| markmcclain | https://jenkins02.openstack.org/job/gate-neutron-python26/3722/console | 13:24 |
| *** adarshdeep has joined #openstack-infra | 13:26 | |
| *** dstanek has joined #openstack-infra | 13:26 | |
| *** rahmu has quit IRC | 13:26 | |
| *** sandywalsh has quit IRC | 13:27 | |
| adarshdeep | Hi there!.. | 13:27 |
| *** rahmu has joined #openstack-infra | 13:28 | |
| *** michchap has joined #openstack-infra | 13:28 | |
| *** rongze has joined #openstack-infra | 13:29 | |
| markmcclain | yep.. definitely a problem with a machine | 13:30 |
| markmcclain | http://logs.openstack.org/86/58286/1/gate/gate-neutron-python26/4670b5c/console.html | 13:31 |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: Review and create Puppet configuration for dev server https://review.openstack.org/53644 | 13:31 |
| markmcclain | http://logs.openstack.org/46/58446/1/check/gate-neutron-python26/d4befef/console.html | 13:31 |
| *** dhellmann is now known as dhellmann-afk | 13:31 | |
| *** zoresvit has joined #openstack-infra | 13:36 | |
| *** michchap has quit IRC | 13:37 | |
| *** amotoki has joined #openstack-infra | 13:37 | |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: Review and create Puppet configuration for dev server https://review.openstack.org/53644 | 13:37 |
| *** vkmc has joined #openstack-infra | 13:37 | |
| *** thomasem has joined #openstack-infra | 13:40 | |
| *** sandywalsh has joined #openstack-infra | 13:42 | |
| bauzas | markmcclain: https://code.launchpad.net/bugs/1255041 | 13:42 |
| uvirtbot | Launchpad bug 1255041 in openstack-ci "Python 2.6 gate broken due to no workspace found" [Undecided,New] | 13:42 |
| markmcclain | jeblair, clarkb, fungi, mordred: py26 slave problem is messing with the gate | 13:42 |
| bauzas | seems like a Jenkins slave is down | 13:43 |
| markmcclain | bauzas: thanks for the link | 13:43 |
| bauzas | happening since 0900 UTC | 13:44 |
| bauzas | some checks are passing thou | 13:44 |
| bauzas | so there is probably one slave down | 13:44 |
| markmcclain | probably depends on where the work gets assigned | 13:44 |
| bauzas | yup | 13:44 |
| hashar | most probably centos6-4 is foobar | 13:45 |
| *** rahmu has quit IRC | 13:47 | |
| *** rahmu has joined #openstack-infra | 13:49 | |
| *** sarob has joined #openstack-infra | 13:50 | |
| *** markmcclain has quit IRC | 13:50 | |
| *** sandywalsh has quit IRC | 13:51 | |
| *** bpokorny has joined #openstack-infra | 13:51 | |
| *** dprince has joined #openstack-infra | 13:51 | |
| *** pblaho has quit IRC | 13:53 | |
| *** sarob has quit IRC | 13:54 | |
| *** michchap has joined #openstack-infra | 13:58 | |
| *** ruhe has joined #openstack-infra | 13:58 | |
| *** colinmcnamara has joined #openstack-infra | 13:59 | |
| *** rcleere has joined #openstack-infra | 14:00 | |
| Mithrandir | hmm, configparser doesn't have a useful .get method.. GRR | 14:02 |
| *** wenlock has joined #openstack-infra | 14:02 | |
| *** michchap has quit IRC | 14:03 | |
| *** sandywalsh has joined #openstack-infra | 14:04 | |
| *** wenlock has quit IRC | 14:04 | |
| *** wenlock has joined #openstack-infra | 14:05 | |
| *** bingbu has quit IRC | 14:08 | |
| *** jd__` has joined #openstack-infra | 14:09 | |
| *** jd__ has quit IRC | 14:10 | |
| *** jd__` is now known as jd__ | 14:10 | |
| *** mriedem has joined #openstack-infra | 14:11 | |
| *** dhellmann-afk is now known as dhellmann | 14:11 | |
| *** julim has joined #openstack-infra | 14:12 | |
| *** mkerrin has quit IRC | 14:12 | |
| *** salv-orlando has quit IRC | 14:15 | |
| *** xeyed4good has joined #openstack-infra | 14:15 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add merge-release-tags job to Savanna https://review.openstack.org/57667 | 14:18 |
| mordred | I have restarted the slave on 6-4 - I checked the workspace and it was there | 14:19 |
| portante | mordred: would this account for "SEVERE ERROR occurs: java.io.IOException: Remote call on centos6-4 failed" | 14:20 |
| ttx | mordred: when you're sure it's fixed you should issue another "#status notice" | 14:20 |
| *** salv-orlando has joined #openstack-infra | 14:21 | |
| portante | we are seeing a few jobs fail like that, nothing actually ran, just that message and few other lines in the console | 14:21 |
| portante | we being swift | 14:21 |
| *** bingbu has joined #openstack-infra | 14:21 | |
| ttx | portante: should be ok once the slave is restarted | 14:22 |
| portante | ttx, k thanks | 14:22 |
| mordred | the slave is running a trove job successfully now | 14:22 |
| portante | is there a bug id we should recheck under? | 14:23 |
| mordred | ttx: dammit. what't the "staus now ok" syntax? | 14:23 |
| mordred | portante: 1255041 | 14:23 |
| portante | thx | 14:23 |
| ttx | mordred: there is no "ko" or "ok" syntax, just info/notice/alert | 14:24 |
| ttx | so #status notice Gate should be unwedged now, thanks for your patience" should work | 14:24 |
| mordred | ttx: hah | 14:24 |
| mordred | #status notice Gate should be unwedged now, thanks for your patience" should work | 14:24 |
| mordred | it hates me | 14:25 |
| ttx | hmm | 14:25 |
| ttx | #status notice Gate should be unwedged now, thanks for your patience | 14:25 |
| openstackstatus | NOTICE: Gate should be unwedged now, thanks for your patience | 14:25 |
| mordred | the bot usually hates me | 14:25 |
| mordred | thank you ttx | 14:25 |
| *** osanchez has joined #openstack-infra | 14:25 | |
| *** Loquacity has quit IRC | 14:26 | |
| *** dkranz has quit IRC | 14:26 | |
| *** Loquacity has joined #openstack-infra | 14:27 | |
| *** bingbu has quit IRC | 14:28 | |
| *** sarob_ has joined #openstack-infra | 14:29 | |
| *** sarob__ has joined #openstack-infra | 14:30 | |
| fungi | we have a py26 slave problem? what's the situation? | 14:31 |
| *** oubiwann has joined #openstack-infra | 14:31 | |
| sdague | fungi: centos 6-4 has gone stupid | 14:31 |
| fungi | oh, a slave was rapid-fire failing every job assigned to it again? | 14:32 |
| sdague | fungi: yep | 14:32 |
| sdague | mordred seems to have fixed it | 14:32 |
| ttx | now the question is, I guess.. how could we avoid blocking people for 9 hours | 14:32 |
| sdague | unrelated, I really wish I had the power to yellow card people that recheck no bug on a patch of theirs which has a syntax error | 14:33 |
| sdague | and not let them submit code for 48hrs | 14:33 |
| ttx | maybe some detection for 100% quickfails | 14:33 |
| *** sarob_ has quit IRC | 14:33 | |
| sdague | like SergeyLukjanov and this patch - https://review.openstack.org/#/c/57325/ :{ | 14:33 |
| *** oubiwann_ has joined #openstack-infra | 14:33 | |
| ttx | sdague: "The QA hall of shame" could work | 14:34 |
| SergeyLukjanov | sdague, this code works ok before it and so I've post 'recheck no bug' | 14:34 |
| sdague | SergeyLukjanov: you didn't even look at the console log | 14:34 |
| sdague | there is 0 excuse for not looking at the console log on a fail | 14:34 |
| sdague | 2013-11-26 10:55:55.256 | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 208: syntax error near unexpected token `}' | 14:35 |
| *** wenlock has quit IRC | 14:36 | |
| fungi | perhaps there's a confusion on what "no bug" is referring to. it doesn't mean "there's no bug in my code" it means "the reason testing failed was due to something other than a bug" (or "i want to rerun tests even there was no failure") | 14:36 |
| *** wenlock has joined #openstack-infra | 14:36 | |
| sdague | except, there is a bug in that code | 14:36 |
| *** oubiwann has quit IRC | 14:36 | |
| sdague | it's not even valid code | 14:36 |
| fungi | that does make it a bit more egregious | 14:37 |
| sdague | if your code has a syntax error, and you run recheck... that demonstrates a pretty fundamental lack of understanding | 14:37 |
| ttx | fungi: could use a second infra +2 on https://review.openstack.org/#/c/57896/ | 14:39 |
| fungi | ttx: mordred: so as far as detecting slaves immediately failing any test assigned to them, my understanding is that we hadn't done that because jenkins lacks the ability to tell the difference, so we'd presumably need to do some console log analysis to determine that programmatically | 14:39 |
| SergeyLukjanov | sdague, no excuses, that's my fault for rechecking code w/o looking into the logs | 14:39 |
| sdague | fungi: yeh, I was thinking there should be some common fingerprints for what these nodes going into fail everything mode are. And could trigger something that shoots them in the head. | 14:40 |
| ttx | fungi: yeah, it's a bit hard... the alternative would be to build a red button that a few people in EU TZ could have the authorization to hit in such cases... if we think they might happen again | 14:41 |
| sdague | SergeyLukjanov: thanks for owning up on it | 14:41 |
| dstanek | is one of the Jenkins instances having trouble? | 14:41 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Add Savanna testing support https://review.openstack.org/57325 | 14:41 |
| dstanek | i just got this error: http://logs.openstack.org/42/48342/11/check/gate-keystone-python26/bc2d860/ | 14:41 |
| ttx | dstanek: one of the Jenkins instances used to be in trouble yes | 14:41 |
| fungi | dstanek: apparently one of the centos6 slaves went haywire | 14:41 |
| dstanek | fungi: well, that's not good | 14:42 |
| ttx | dstanek: you didn't "just got" it. was one hour ago, before it got fixed. | 14:42 |
| sdague | fungi: yeh, I think we need to either figure out a way to auto detect it, or to get people in more TZs able to reset the nodes | 14:42 |
| ttx | dstanek: should be good now | 14:42 |
| dstanek | ttx: thanks, i'll give it another shot | 14:43 |
| fungi | sounds like it started acting up just after i passed out last night | 14:43 |
| sdague | especially once mordred is back on the west coast, it means you're going to be first light on most of these issues that block .eu overnight | 14:43 |
| ttx | fungi: damn jenkins is pretty vicious | 14:43 |
| *** xeyed4good has left #openstack-infra | 14:44 | |
| fungi | mordred: you're on the west coast for a while, or just another quick trip? | 14:44 |
| ttx | fungi: saw my review link above ? | 14:44 |
| fungi | ttx: yes, jeblair was going through and -2'ing new project additions while mordred works through refactoring the race conditions out of the manage-projects script today. looks like he hadn't gotten to that one yet | 14:45 |
| fungi | i have a backlog of 29 changes i'm trying to review which already have at least one infra +2, and that one is far from the oldest | 14:46 |
| jpich | Is openstack-infra in charge of the Foundation's members profile directory or should bugs be reported elsewhere? | 14:46 |
| fungi | jpich: that's handled by toddmorey (which isn't often in irc) | 14:47 |
| *** mfer has joined #openstack-infra | 14:47 | |
| jpich | fungi: Thanks. Would there be an email I can send some info to in the meantime? | 14:49 |
| *** thedodd has joined #openstack-infra | 14:50 | |
| fungi | jpich: i just wind up sending it to his personal e-mail address because we don't seem to have any published role contact address for that site. let me see if his address is already public knowledge before i go giving it out | 14:50 |
| fungi | jpich: ahh, yes, it's on the wiki... https://wiki.openstack.org/wiki/Website todd@openstack.org | 14:51 |
| jpich | fungi: Great, thanks a lot! | 14:52 |
| fungi | feel free to ask him where the code is. maybe if enough people offer to help fix the bugs in it... | 14:52 |
| *** sarob__ is now known as sarob_ | 14:53 | |
| *** jroovers has quit IRC | 14:56 | |
| *** ryanpetrello has joined #openstack-infra | 14:56 | |
| *** rcleere has quit IRC | 14:57 | |
| *** michchap has joined #openstack-infra | 14:59 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Clone repo if it is not on disk https://review.openstack.org/52689 | 14:59 |
| *** avishayb has joined #openstack-infra | 15:00 | |
| *** dolphm has joined #openstack-infra | 15:01 | |
| avishayb | Hi | 15:02 |
| avishayb | My name is Avishay Balderman and I work in Radware. | 15:02 |
| avishayb | I am an active developer in Neutron focused on LBaaS. | 15:02 |
| avishayb | I have developed the Radware LBaaS driver for Havana. | 15:02 |
| avishayb | In Ice House the plugin / drivers of Neutron should support “Third Party Testing”. | 15:02 |
| avishayb | See http://ci.openstack.org/third_party.html | 15:02 |
| *** openstackstatus has quit IRC | 15:02 | |
| avishayb | So I have some code that “listen” to Gerrit stream, analyze the events and for some events I want to “look deeper” into the patchset and get the list of the involved files. | 15:02 |
| avishayb | The file list is not part of the JSON event. Is there any API / command that I can use to ask Gerrit “ Give me the file list of change X patchset Y” ? | 15:02 |
| avishayb | Thanks | 15:02 |
| fungi | avishayb: you're trying to find out the list of files modified by a particular patchset in gerrit? | 15:03 |
| *** michchap has quit IRC | 15:03 | |
| fungi | avishayb: read https://review.openstack.org/Documentation/cmd-query.html | 15:04 |
| fungi | avishayb: that should be the api command you need for querying gerrit to find out which files are touched by a patchset | 15:05 |
| *** dcramer_ has joined #openstack-infra | 15:07 | |
| *** che-arne has quit IRC | 15:08 | |
| avishayb | OK - let me have a look.. | 15:09 |
| *** colinmcnamara has quit IRC | 15:12 | |
| *** dkranz has joined #openstack-infra | 15:15 | |
| *** ryanpetrello has quit IRC | 15:15 | |
| avishayb | Lets take https://review.openstack.org/#/c/56821/ as a an example. Patch Set #4 contains 2 files: tests.py,workflows.py. | 15:16 |
| *** herndon has joined #openstack-infra | 15:18 | |
| portante | has anybody see this error installing warlock before? http://logs.openstack.org/97/58397/2/check/check-grenade-devstack-vm/280c76f/console.html | 15:18 |
| avishayb | The command: ssh -p 29418 review.openstack.org gerrit query <args here> | 15:19 |
| *** sarob_ has quit IRC | 15:19 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: GC git.Repo objects in merger.Repo operations. https://review.openstack.org/55943 | 15:19 |
| portante | and prior to that jsonpatch timed out downloading from pypi.openstack.org | 15:19 |
| *** sarob has joined #openstack-infra | 15:20 | |
| avishayb | Cant figure out how specify the chabge id (56821) and the patchset number (4) | 15:20 |
| avishayb | *change id | 15:21 |
| avishayb | OK - it works (ssh -p 29418 review.openstack.org gerrit query --patch-sets 56821 --files) - thanks | 15:24 |
| *** sarob has quit IRC | 15:24 | |
| fungi | avishayb: yes, or ssh -p 29418 review.openstack.org gerrit query --current-patch-set --files -- 56821 | 15:26 |
| fungi | which according to the api docs should give you files for the current patch set on change 56821 | 15:27 |
| avishayb | and can I specify the patchset number? | 15:27 |
| fungi | avishayb: it looks like the only two options are --current-patch-set (which reports the most recent one) or --patch-sets (which gives you all of them and then you'd have to parse it out) | 15:28 |
| avishayb | or i will get the last one ("current") with your command? | 15:28 |
| fungi | if you add --format JSON then it shouldn't be too hard to pick out the patchset you want | 15:28 |
| *** adarshdeep has quit IRC | 15:28 | |
| avishayb | Yep. thanks. | 15:28 |
| *** datsun180b has joined #openstack-infra | 15:29 | |
| fungi | portante: the occurrences of download timeouts from pypi.o.o which i've dug into so far have looked (from apache logs on the pypi end) like the request never got through. clouds, internet, et cetera | 15:30 |
| *** pblaho has joined #openstack-infra | 15:31 | |
| portante | k | 15:32 |
| portante | thanks fungi, I filed a bug just the same | 15:32 |
| *** DennyZhang has joined #openstack-infra | 15:32 | |
| portante | and rechecked against the bug I filed | 15:32 |
| fungi | this might be something which we could improve by putting a mirror in each az or something (for most devstack jobs the communication between the slave and pypi.o.o is going between hpcloud and rackspace) | 15:32 |
| portante | oh | 15:33 |
| portante | yes, that would help | 15:33 |
| *** DennyZhang has quit IRC | 15:34 | |
| fungi | it was previously cropping up infrequently, but if incidents of that are on teh rise, it may be something we should start actively trending to see whether it's worth the effort | 15:34 |
| portante | agreed | 15:34 |
| portante | so where are we at with reverify no bug | 15:34 |
| portante | did that get turned off? | 15:34 |
| portante | or will it be turned off? | 15:34 |
| *** DennyZhang has joined #openstack-infra | 15:36 | |
| *** alexpilotti has joined #openstack-infra | 15:38 | |
| *** eharney has joined #openstack-infra | 15:39 | |
| fungi | i think it's in the process of being socialized with the -dev ml for the moment | 15:40 |
| mordred | jpich: we'd like to be in charge of that though | 15:40 |
| mordred | ttx: I'm working on getting someone hired in EU timezone | 15:40 |
| mordred | ttx: it's too bad that I'm still the only person trying to hire people for such things, but such is life | 15:41 |
| *** dhellmann is now known as dhellmann-afk | 15:41 | |
| fungi | mordred: that log which portante linked above might be of interest to you (looks like maybe a dependency issue with warlock?) | 15:42 |
| jpich | mordred: That'd be cool, and more consistent | 15:42 |
| fungi | distutils looks like it's barfing about not understanding install_requires | 15:42 |
| mordred | it doesn't | 15:43 |
| mordred | distutils doesn't do install_requires | 15:43 |
| fungi | not sure if that's benign though, or related to the traceback which follows it | 15:43 |
| mordred | that's not the real error | 15:43 |
| fungi | "TypeError: dist must be a Distribution instance" | 15:43 |
| mordred | that's what the errors look like when a) setuptools has been borked or upgraded wrong or b) when easy_install has gotten involved when it shouldn't | 15:44 |
| fungi | oic, further up there's "URLError: <urlopen error timed out> | 15:44 |
| mordred | awesome | 15:44 |
| fungi | so it is just another pypi.o.o connection timeout | 15:44 |
| * mordred has to run for a sec - back in a bit and I'll look at it | 15:44 | |
| mordred | how are we getting those? seems crazy | 15:44 |
| fungi | mordred: no worries, this is a known pattern after all | 15:45 |
| mordred | also, perhaps we should run a pypi.o.o in hp cloud... | 15:45 |
| *** rnirmal has joined #openstack-infra | 15:45 | |
| fungi | mordred: internets and clouds i think | 15:45 |
| * mordred afks | 15:45 | |
| fungi | mordred: and yes, that has been suggested. seems maybe wise | 15:45 |
| portante | fungi: I filed bugs on both, just in case | 15:45 |
| fungi | portante: appreciated! | 15:45 |
| ttx | mordred: we have an extra infra engineer position opened at the Foundation. But it's a rather specific profile (primary job as security coordinator) so we'll have to not be too picky about location | 15:46 |
| *** herndon has quit IRC | 15:46 | |
| ttx | mordred: but that person should be able to pick up most of the security tooling infra and more | 15:47 |
| * fungi is extremely excited about that | 15:48 | |
| * ttx should look into the job description etherpad to see if it's in postable form | 15:48 | |
| *** briancline has joined #openstack-infra | 15:50 | |
| *** MarkAtwood has joined #openstack-infra | 15:51 | |
| sdague | ttx: coolness | 15:55 |
| *** jgrimm has joined #openstack-infra | 15:56 | |
| *** dhellmann-afk is now known as dhellmann | 15:58 | |
| *** reed has joined #openstack-infra | 15:59 | |
| *** michchap has joined #openstack-infra | 15:59 | |
| *** mgagne has joined #openstack-infra | 16:01 | |
| *** afazekas has quit IRC | 16:02 | |
| *** jgrimm has quit IRC | 16:02 | |
| *** UtahDave has joined #openstack-infra | 16:02 | |
| *** michchap has quit IRC | 16:04 | |
| *** pblaho has quit IRC | 16:05 | |
| *** DinaBelova has quit IRC | 16:06 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: create a jobwrapper span to address long job names spillover https://review.openstack.org/58360 | 16:08 |
| *** SergeyLukjanov has quit IRC | 16:08 | |
| sdague | fungi: any chance you could take a look at that today, it's a UX work around for the long devstack job names | 16:09 |
| openstackgerrit | Swann Croiset proposed a change to openstack-infra/config: Add Climate-nova sub-project https://review.openstack.org/58560 | 16:09 |
| *** yamahata_ has quit IRC | 16:10 | |
| fungi | sdague: signs point to no, but i'll see how far i catch up on reviews. i'm trying to tackle oldest outstanding reviews first, particularly those which already have at least one +2, and work my way forward | 16:10 |
| sdague | fungi: ok | 16:10 |
| sdague | well, it would be nice to know which jobs succeeded already :) | 16:10 |
| avishayb | fungi: if I want to do ssh inside python - which lib I should use? | 16:11 |
| *** mrodden has quit IRC | 16:11 | |
| *** markwash has quit IRC | 16:11 | |
| *** ^d has joined #openstack-infra | 16:11 | |
| *** ^d has joined #openstack-infra | 16:11 | |
| fungi | avishayb: i think we tend to use paramiko mostly for that | 16:11 |
| sdague | you know, because this is fun to decypher - http://ubuntuone.com/5rgUnES1lnCoHZAna4JNx6 | 16:11 |
| sdague | avishayb: yeh, paramiko is what's used broadly for that | 16:12 |
| *** jgrimm has joined #openstack-infra | 16:13 | |
| avishayb | ok guys. thanks | 16:13 |
| *** yaguang has quit IRC | 16:16 | |
| *** dhellmann is now known as dhellmann-afk | 16:17 | |
| *** dcramer_ has quit IRC | 16:18 | |
| *** dcramer_ has joined #openstack-infra | 16:20 | |
| *** mrodden has joined #openstack-infra | 16:21 | |
| *** mihgen has quit IRC | 16:22 | |
| *** branen has joined #openstack-infra | 16:24 | |
| jeblair | ttx, fungi: there are two ways we have discussed dealing with jenkins not knowing that it has a bad slave: (a) use nodepool for all slaves, (b) stop using jenkins | 16:24 |
| *** rcleere has joined #openstack-infra | 16:24 | |
| jeblair | ttx, fungi: if someone wants to spend dev effort on this, i'd rather it go toword those efforts | 16:25 |
| *** sarob has joined #openstack-infra | 16:25 | |
| ttx | jeblair: yeah | 16:26 |
| jeblair | (a) is short-term, (b) is longer | 16:27 |
| sdague | so... did the recheck bot die? | 16:32 |
| sdague | http://status.openstack.org/rechecks/ looks extra stale | 16:32 |
| jeblair | sdague: process is running... | 16:34 |
| jeblair | -rw-rw-r-- 1 recheckwatch recheckwatch 8932 Nov 26 16:32 scoreboard.pickle | 16:34 |
| jeblair | -rw-rw-r-- 1 recheckwatch recheckwatch 15602 Nov 26 16:35 /var/www/recheckwatch/rechecks.html | 16:35 |
| sdague | so something is wrong, because the last seen of everything on that page is Oct 20th | 16:35 |
| jeblair | sdague: i think we're out of debugging options for this one. i'll restart it. | 16:36 |
| fungi | sdague: jog0 was asking about it last night. i looked into it, was last started on october 23 and had not touched its html or pickle file since then. it doesn't do any logging, so i restarted it | 16:36 |
| *** avishayb has quit IRC | 16:36 | |
| fungi | i ran an strace on it for a while first. looked like it thought it had an open socket to gerrit but didn't seem to be seeing any events on the stream | 16:37 |
| portante | fungi: where do I beg for reviews on devstack changes? | 16:37 |
| jeblair | restarted. | 16:37 |
| portante | here, or in -qa? | 16:37 |
| fungi | portante: mostly #openstack-qa | 16:37 |
| portante | k thanks | 16:37 |
| sdague | maybe we should change recheck page to be cron based instead of going after the gerrit stream raw | 16:41 |
| *** ywu has joined #openstack-infra | 16:41 | |
| *** markwash has joined #openstack-infra | 16:42 | |
| sdague | fungi: are we ok with service accounts on gerrit to do cron things like that? | 16:42 |
| fungi | sdague: mine the comments via a query or something you mean> | 16:42 |
| sdague | fungi: yep | 16:42 |
| jeblair | sdague: er, aren't we getting rid of the recheck page? :) | 16:43 |
| sdague | yeh, well :) | 16:43 |
| sdague | yes... though we still need access to what people recheck on as well, to find bugs we aren't er-ing | 16:43 |
| *** MarkAtwood has quit IRC | 16:44 | |
| *** beagles has quit IRC | 16:45 | |
| jeblair | sdague: then it probably makes sense to incorporate that into the e-r page, which is run in cron; and that moves us forward in the plan | 16:45 |
| sdague | yep | 16:46 |
| *** beagles has joined #openstack-infra | 16:47 | |
| *** SergeyLukjanov has joined #openstack-infra | 16:47 | |
| jeblair | ttx, mordred: http://ci.openstack.org/irc.html#statusbot | 16:47 |
| *** hashar has quit IRC | 16:48 | |
| *** pcrews has joined #openstack-infra | 16:49 | |
| *** DinaBelova has joined #openstack-infra | 16:50 | |
| *** Adri2000 has quit IRC | 16:54 | |
| *** mrodden has quit IRC | 16:56 | |
| *** boris-42 has joined #openstack-infra | 16:58 | |
| ttx | jeblair: ack, didn't need #status ok since I didn't set topic with #status alert. | 16:58 |
| ttx | only used #status notice. | 16:59 |
| *** flaper87 is now known as flaper87|afk | 16:59 | |
| fungi | jog0: i'm trying to test your https://review.openstack.org/58019 change, but the two gate lines for yesterday look like they exceeded 100% somehow... or we have gaps in the data? ideas there? | 16:59 |
| *** michchap has joined #openstack-infra | 17:00 | |
| *** Adri2000 has joined #openstack-infra | 17:00 | |
| *** pete5 has joined #openstack-infra | 17:04 | |
| *** michchap has quit IRC | 17:05 | |
| *** mrodden has joined #openstack-infra | 17:07 | |
| *** jgrimm has quit IRC | 17:08 | |
| *** markmcclain has joined #openstack-infra | 17:09 | |
| openstackgerrit | Abel Lopez proposed a change to openstack-infra/config: Added openstack-chef-repo to list https://review.openstack.org/58585 | 17:12 |
| *** markmc has quit IRC | 17:12 | |
| jeblair | woo! jog0's patch is down to #24 on my review list! | 17:14 |
| fungi | heh | 17:15 |
| *** dizquierdo has quit IRC | 17:15 | |
| marun | has anyone else noticed that unstacking is no longer killing nova processes? screen gets killed, but nova-* keeps running, indicating daemonization | 17:17 |
| *** gyee has joined #openstack-infra | 17:18 | |
| *** sarob has quit IRC | 17:21 | |
| *** sarob has joined #openstack-infra | 17:21 | |
| *** david-lyle is now known as david-lyle_afk | 17:22 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/zuul: Add the possibility to pass the zuul url https://review.openstack.org/54196 | 17:22 |
| *** SergeyLukjanov has quit IRC | 17:23 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:23 | |
| *** sarob_ has joined #openstack-infra | 17:23 | |
| zaro | any infra-core want to do a thanksgiving release of jjb? | 17:24 |
| jeblair | zaro: sure; i'm not completely caught up on reviews, but i'll take your word that it's a good time to do a release. :) | 17:25 |
| *** sarob has quit IRC | 17:26 | |
| jeblair | zaro: would you like HEAD tagged with 0.6.0 now? | 17:26 |
| *** jcoufal has quit IRC | 17:27 | |
| *** reaper has joined #openstack-infra | 17:27 | |
| *** sarob_ has quit IRC | 17:28 | |
| zaro | jeblair: sure. that sounds good. | 17:28 |
| *** reed has quit IRC | 17:28 | |
| fungi | jgriffith: are you cool with Alex_Gaynor's https://review.openstack.org/57953 to start gating cinderclient on pypy? (he doesn't say specifically, but i assume it's confirmed to work at this point) | 17:30 |
| Alex_Gaynor | fungi: yes, they pass :-) | 17:30 |
| * Alex_Gaynor isn't nearly that mean | 17:30 | |
| fungi | Alex_Gaynor: right, i'm pretty certain you test these things | 17:30 |
| UtahDave | fungi: I'm building out a reactor example right now on a new etherpad. | 17:31 |
| jeblair | ArxCruz: sorry, i had a change of heart last night on one of my comments on your zuul change in https://review.openstack.org/#/c/54196/ | 17:31 |
| fungi | UtahDave: very cool, and much appreciated | 17:31 |
| jeblair | ArxCruz: (it's a small change though) | 17:31 |
| *** MarkAtwood has joined #openstack-infra | 17:32 | |
| fungi | UtahDave: i've confirmed that all our minion interactions seem stable at this point. 63 various jenkins slaves enrolled with the salt master, and the master is also enrolled with itself as a minion, and communication seems solid for a while now | 17:32 |
| UtahDave | great! | 17:32 |
| jeblair | zaro: 0.6.0 pushed | 17:32 |
| ArxCruz | jeblair: np, I will fix :) | 17:32 |
| zaro | jeblair: thx. | 17:33 |
| *** markmcclain has quit IRC | 17:39 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/zuul: Add the possibility to pass the zuul url https://review.openstack.org/54196 | 17:40 |
| *** FallenPegasus has joined #openstack-infra | 17:41 | |
| *** dolphm has quit IRC | 17:43 | |
| *** dolphm has joined #openstack-infra | 17:43 | |
| *** rongze has quit IRC | 17:44 | |
| zaro | jeblair: question about https://review.openstack.org/#/c/57333 | 17:44 |
| jeblair | zaro: go for it | 17:45 |
| zaro | jeblair: i assume things that should be different are jenkins url and max values? what else? | 17:45 |
| jeblair | ttx: https://review.openstack.org/#/c/57667/3 | 17:45 |
| jeblair | zaro: i think the config file should be much smaller -- we don't really need very many servers spun up, so we don't want to waste our quota on dev servers. | 17:46 |
| jeblair | zaro: you're wanting to test the new jenkins, right? so you probably only just need a couple of servers from one AZ in order to just see that everything is working. | 17:47 |
| *** mrodden has quit IRC | 17:47 | |
| zaro | jeblair: that's what i mean with the max value. how much is reasonable? | 17:47 |
| openstackgerrit | A change was merged to openstack-infra/config: Drop the saz-gearman module (we don't use it) https://review.openstack.org/57527 | 17:47 |
| jeblair | zaro: 2? | 17:47 |
| zaro | jeblair: from each AZ, right? | 17:48 |
| SergeyLukjanov | jeblair, I have a question about your comment for "Setup devstack-gate tests for Savanna" (https://review.openstack.org/#/c/57317/) | 17:48 |
| jeblair | zaro: nah, just one. | 17:48 |
| jeblair | SergeyLukjanov: sure | 17:48 |
| clarkb | morning | 17:48 |
| SergeyLukjanov | jeblair, I'm looking at the layout.yaml and there are no project-specific jobs there | 17:48 |
| SergeyLukjanov | clarkb, hi | 17:48 |
| SergeyLukjanov | so, the question is, will be savanna included to the common d-g jobs? | 17:49 |
| zaro | jeblair: ok. so i'll change it to point to jenkins-dev and set max to 2. which AZ? | 17:49 |
| SergeyLukjanov | eventually | 17:49 |
| SergeyLukjanov | jeblair, I've moved jobs to the separated file thinking about them as temp jobs | 17:50 |
| *** Ryan_Lane has joined #openstack-infra | 17:50 | |
| *** sarob has joined #openstack-infra | 17:52 | |
| jeblair | SergeyLukjanov: I understand what you're saying. | 17:52 |
| jeblair | zaro: az2 | 17:53 |
| ekarlso | what's a good way of integrating a stackforge project with devstack ? | 17:53 |
| jeblair | SergeyLukjanov: clarkb's work is trying to make it so that the bitrot jobs for the stable branches are in-sync with the gate jobs | 17:53 |
| *** derekh has quit IRC | 17:54 | |
| jeblair | SergeyLukjanov: so i think the question is whether these jobs might end up as bitrot jobs someday. | 17:54 |
| *** vipul is now known as vipul-away | 17:54 | |
| jeblair | SergeyLukjanov: and they might -- if these jobs are still the jobs we're running when we release icehouse | 17:55 |
| *** nicedice has joined #openstack-infra | 17:56 | |
| *** FallenPegasus has quit IRC | 17:56 | |
| jeblair | clarkb: do you have thoughts on https://review.openstack.org/#/c/57317 ? | 17:56 |
| clarkb | right, my interest is in making the post release transition easy for infra. It was a bit painful to untangle the jobs we added during havana in order to have them run periodically or against devstack-gate's check pipeline | 17:56 |
| clarkb | jeblair: looking | 17:56 |
| SergeyLukjanov | jeblair, ok, got it | 17:57 |
| jeblair | clarkb: my thought is that the savanna jobs could benefit from your work; SergeyLukjanov was making the point that these may be temporary, but i think there's a good chance that they'll still be around in some form... | 17:58 |
| *** nati_ueno has joined #openstack-infra | 17:58 | |
| jeblair | clarkb: so my inclination is to say they should wait or build on your change. but i think your input should weigh heavily. :) | 17:58 |
| openstackgerrit | Mauro S. M. Rodrigues proposed a change to openstack-infra/zuul: Add the possibility to pass the zuul url https://review.openstack.org/54196 | 17:59 |
| clarkb | jeblair: I agree with what you have said. Is the counter to that that the jobs will be temporary so shouldn't need to follow the new templating? | 17:59 |
| *** nati_ueno has quit IRC | 17:59 | |
| clarkb | I really don't want to need to do this again in a couple months because we have bit rot | 18:00 |
| *** nati_ueno has joined #openstack-infra | 18:00 | |
| *** sarob has quit IRC | 18:00 | |
| jeblair | clarkb: yes, that is the counter, but i think it will be better to get savanna using the new templates sooner. | 18:00 |
| SergeyLukjanov | clarkb, jeblair, btw I'm not proposing to do it in the incorrect way, just trying to understand the correct one :) | 18:00 |
| *** michchap has joined #openstack-infra | 18:01 | |
| clarkb | SergeyLukjanov: for the new proposed correct way, we basically need two job templates for each job. One template that runs against proposed changes in the check or gate pipelines and another template that run against the current state of the branch it is testing in the periodic queue | 18:02 |
| fungi | ekarlso: you use the hooks built into devstack-gate which allow you to specify alternative functions for setup and run, changes to the list of projects to include and start, et cetera. see the requirements integration job and swift functional testing job for examples | 18:02 |
| clarkb | SergeyLukjanov: there are enough subtle differences in the way that zuul deals with those pipelines that we can't use a single template for everything | 18:02 |
| clarkb | SergeyLukjanov: the end result is that when icehouse happens we already have all of the job templating we need to test icehouse post release | 18:02 |
| SergeyLukjanov | clarkb, yep, I understand it | 18:03 |
| jeblair | fungi: i was proposing that we not add new _non-openstack_ projects until 1242569 is resolved | 18:03 |
| SergeyLukjanov | clarkb, my main concern is about how savanna jobs should be added | 18:03 |
| *** reed has joined #openstack-infra | 18:03 | |
| fungi | jeblair: oh! reversing my -2 in that case and approving | 18:03 |
| fungi | jeblair: thanks, and sorry for the confusion | 18:03 |
| SergeyLukjanov | clarkb, looks like I need to add templates for savanna jobs to both pipelines | 18:04 |
| SergeyLukjanov | clarkb, eventually we'll change them to common jobs if savanna we'll be enabled to them | 18:04 |
| SergeyLukjanov | or keep separated templates | 18:04 |
| clarkb | SergeyLukjanov: let me put a paste together that shows what we are doing for the other jobs | 18:04 |
| fungi | jeblair: once it merges and updates, i'll hammer it through manage-projects manually and confirm zuul and git.o.o repos are sane | 18:05 |
| jeblair | fungi: i'll stop puppet on zuul, so we can avoid the zuul part of that race. | 18:05 |
| fungi | okay, great | 18:05 |
| *** michchap has quit IRC | 18:05 | |
| jeblair | jgriffith: when you have a sec, can you weigh in on https://review.openstack.org/#/c/57953/ | 18:06 |
| clarkb | SergeyLukjanov: for each logical job (eg run devstack with mysql then run tempest against that to test) we have a pair of templates http://paste.openstack.org/show/54015/ | 18:06 |
| clarkb | SergeyLukjanov: savanna should do the same for each logical test | 18:06 |
| SergeyLukjanov | clarkb, yep, that's exactly what I mean | 18:07 |
| openstackgerrit | A change was merged to openstack-infra/config: Add openstack/oslo.rootwrap project https://review.openstack.org/57896 | 18:09 |
| *** senk has joined #openstack-infra | 18:10 | |
| openstackgerrit | A change was merged to openstack-infra/config: Alphabetize projects.yaml again https://review.openstack.org/57959 | 18:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix SmokeStack link to be the ssl version. https://review.openstack.org/57964 | 18:11 |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/zuul: Add the possibility to pass the zuul url https://review.openstack.org/54196 | 18:12 |
| jeblair | fungi: may want to read my comment on https://review.openstack.org/#/c/57987/ | 18:13 |
| fungi | thx | 18:13 |
| jeblair | fungi: also, i'm apparently now one hour behind you on reviews! | 18:13 |
| * clarkb is so much further behind | 18:14 | |
| fungi | jeblair: thanks for the note on 57987 | 18:14 |
| jog0 | fungi: yeah I noticed that too, things seem to be back to normal now | 18:15 |
| fungi | jeblair: also, you may be ahead of me. i'm following a slightly different pattern, trying to first focus on oldest changes which already have at least one +2 so we can see faster gains on reducing the overall outstanding change count | 18:15 |
| jeblair | i'm on jog0's change! | 18:15 |
| jog0 | jeblair: cool beans | 18:16 |
| fungi | jog0: any ideas on what was happening for that? i didn't check the data dump for comparison, but wonder whether there are no values reported for those times or whether they're off-scale somehow | 18:16 |
| *** atiwari has joined #openstack-infra | 18:16 | |
| jog0 | fungi: the data is transformed into a percent | 18:17 |
| *** DennyZhang has quit IRC | 18:17 | |
| jog0 | so I am thinking something was funny with the data, not sure | 18:17 |
| jog0 | it looks like all failure percents are below 10% which sounds about right | 18:18 |
| fungi | clarkb: curious whether you're okay on https://review.openstack.org/50400 (didn't want to merge that until you had a chance to weigh back in on your earlier comment) | 18:19 |
| clarkb | looking | 18:19 |
| SergeyLukjanov | clarkb, what's the correct name pattern for savanna d-g jobs? smth like {pipeline}-tempest-devstack-vm-savanna-neutron-{branch-designator}? | 18:19 |
| clarkb | SergeyLukjanov: that will work | 18:19 |
| jeblair | jog0: i think the weird spot is due to the moving average | 18:21 |
| jeblair | jog0: http://graphite.openstack.org/render/?width=586&height=308&_salt=1385490092.033&target=stats.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.SUCCESS&target=stats.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.FAILURE | 18:21 |
| jeblair | jog0: combined with a slow period | 18:22 |
| jog0 | ohhh a slow period makes sense | 18:22 |
| jeblair | jog0: so only one test ran during a window, and it failed | 18:22 |
| lifeless | jeblair: / clarkb: / mordred: did you lot get a chance to peek at our gear confusion ? | 18:22 |
| jog0 | ahh that makes a lot more sense | 18:22 |
| jeblair | lifeless: i think i left a comment? | 18:22 |
| lifeless | jeblair: cool, thanks. | 18:23 |
| mordred | jeblair: status from me: I'm in the keystone meeting right now, then I have 3 powerpoint slides to make for $dayjob, then I'm on fixing manage-projects | 18:23 |
| lifeless | jeblair: (I was lazy asking here, sorry) | 18:23 |
| fungi | jog0: and that brings up a good point... what's the anticipated usefulness of the graph when a given gate job runs once or less an hour? | 18:23 |
| *** boris-42 has quit IRC | 18:23 | |
| mordred | jeblair: it's one of those days where I'm having to keep strict scheduling | 18:23 |
| jeblair | lifeless: i have not swung back around to see if followup is required; ping me if needed | 18:23 |
| fungi | jog0: agreed that the numbers are helpful during higher volume of course | 18:23 |
| SergeyLukjanov | clarkb, so, just to clarify - I'll add -savanna-, -savanna-pg- and -savanna-network- templates to periodic and 'common' jobs | 18:24 |
| jeblair | mordred: need anyone else in keystone mtg? | 18:24 |
| jog0 | fungi: make up more work? but agreed | 18:24 |
| mordred | jeblair: nah. I got asked to lurk on a couple of topics, and I'm also making suggestions on code org for kds | 18:24 |
| fungi | jog0: oh, true, this plays into the idle pool idea well | 18:24 |
| jog0 | not sure how we can convey load information in the same graph | 18:24 |
| fungi | code org for kids! | 18:24 |
| *** herndon has joined #openstack-infra | 18:24 | |
| lifeless | jeblair: so if we invert things - and the test is the worker, the environment the client, it seems simpler ? | 18:25 |
| clarkb | SergeyLukjanov: yup | 18:25 |
| jog0 | fungi: yeah we have all this cloud capacity lets use it | 18:25 |
| fungi | jog0: okay, so i think i'm okay with the data as presented, noting that we'll hopefully (soon) be making sure that at least a few of tehse are run every hour even during "slow" periods | 18:25 |
| clarkb | fungi: I am mulling over that git review thing. I think I am fine with mordred's response. let me finish properly rereviewing the code and will leave a vote | 18:26 |
| jog0 | fungi: sounds good to me, thanks | 18:26 |
| clarkb | oh I should ask for someone to make devstack do keystone apache if that is their suggestion /me lurks too | 18:26 |
| mordred | clarkb: which respose? | 18:26 |
| fungi | clarkb: okay, cool. it was an older one and just trying to make sure it gets some love (and maybe cleared out of the mass of open changes) | 18:26 |
| clarkb | mordred: git rebase -p -i | 18:26 |
| mordred | ah. that one | 18:26 |
| SergeyLukjanov | clarkb, ok, thx, going to rebase to your change | 18:27 |
| SergeyLukjanov | clarkb, hope that it'll be landed soon :) | 18:27 |
| clarkb | SergeyLukjanov: me too, I expect it to get some eyes on it pretty soon now as jeblair and fungi are getting through review backlog | 18:27 |
| jeblair | lifeless: inasmuch as that's the direction where a failure can be reported by gearman, but i don't know if that solves your other requirements. for zuul workers i expect worker to be the environment and test runner. | 18:27 |
| mordred | if we don't like that, then I think we should remove the autorebase - because autrebase will totally do the wrong thing for people who have merge commits | 18:28 |
| fungi | review.o.o has decided to run manage-projects now for the oslo.rootwrap addition. should know shortly if it manages to succeed or needs a shove | 18:28 |
| morganfainberg | clarkb, if you need any help with httpd+keystone let us know. we're happy to help ya out :) | 18:28 |
| clarkb | fungi: should I go ahead and approve that git-review change? | 18:28 |
| clarkb | morganfainberg: well, I want someone else to do it :) | 18:28 |
| fungi | clarkb: feel free | 18:29 |
| clarkb | fungi: done | 18:29 |
| *** vipul-away is now known as vipul | 18:29 | |
| SergeyLukjanov | clarkb, I'm looking on it now too, but don't think that my review will be very useful due to the lack of knowledge in this area :( | 18:29 |
| morganfainberg | clarkb, i mean from the stand point of helping with the reviews as well even if one of us isn't the one to implement | 18:29 |
| fungi | clarkb: is this the "apache should take care of fronting keystone's api endpoint so we don't get occasional ephemeral port collisions" thing? | 18:31 |
| clarkb | fungi: yes | 18:31 |
| clarkb | fungi: I think it will take some major forklifting in devstack | 18:32 |
| clarkb | as simply changing the port from 35357 to 32357 makes everything break today | 18:32 |
| *** ruhe has quit IRC | 18:32 | |
| lifeless | jeblair: I think I interpreted your point correctly; if you could fact check my comment that would be awesome | 18:34 |
| lifeless | jeblair: https://review.openstack.org/#/c/58305/ | 18:34 |
| fungi | manage-projects seems to have dtrt to gerrit for a change. maybe github is in an unusually good mood. checking the git farm next | 18:35 |
| *** ruhe has joined #openstack-infra | 18:35 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add merge-release-tags job to Savanna https://review.openstack.org/57667 | 18:35 |
| jeblair | lifeless: task enqueued :) | 18:36 |
| *** DinaBelova has quit IRC | 18:36 | |
| *** SergeyLukjanov_ has joined #openstack-infra | 18:37 | |
| *** SergeyLukjanov has quit IRC | 18:37 | |
| lifeless | jeblair: ack() :) | 18:37 |
| *** jerryz has joined #openstack-infra | 18:39 | |
| *** SergeyLukjanov has joined #openstack-infra | 18:41 | |
| *** david-lyle_afk is now known as david-lyle | 18:42 | |
| *** fbo is now known as fbo_away | 18:42 | |
| fungi | jeblair: clones of oslo.rootwrap on gerrit, all of our git farm and github check out. are you okay with me starting puppet back up on zuul now? | 18:42 |
| jeblair | fungi: ++ | 18:42 |
| fungi | done | 18:42 |
| *** SergeyLukjanov_ has quit IRC | 18:42 | |
| SergeyLukjanov | clarkb, should be savanna jobs added to the job-group "devstack-jobs"? | 18:43 |
| *** mrmartin has joined #openstack-infra | 18:43 | |
| annegentle | hiya reed | 18:43 |
| reed | hi annegentle | 18:44 |
| reed | hello mrmartin | 18:44 |
| clarkb | SergeyLukjanov: good question, I think it is fine to have them there, particularly since yo uare in the file and incubated | 18:44 |
| clarkb | SergeyLukjanov: and being in that job group means you don't need to update projects.yaml | 18:44 |
| *** rongze has joined #openstack-infra | 18:44 | |
| SergeyLukjanov | clarkb, ok, thx | 18:45 |
| *** ryanpetrello has joined #openstack-infra | 18:45 | |
| fungi | reed: after you mentioned your key signature imports from my message not working in enigmail the other day, i stumbled onto http://sourceforge.net/p/enigmail/bugs/87/ (no idea if you're on 1.4.6 or later, but thought it might be your issue) | 18:45 |
| jog0 | zuul is about to merge 10 patches | 18:47 |
| reed | fungi, I'm on enigmail 1.6 | 18:47 |
| jog0 | gate == not wedged | 18:47 |
| *** ryanpetrello has quit IRC | 18:47 | |
| reed | fungi, I solved the problem yesterday by switching to another view | 18:47 |
| fungi | reed: oh, weird | 18:47 |
| reed | fungi, in the other view I saw the attachments | 18:47 |
| reed | still needed to save them and import them in gpg | 18:48 |
| fungi | reed: that's where i rather line mutt. i can just select the attachment in the attachments list, hit | and then enter gpg2 --import | 18:48 |
| fungi | er. s/line/like/ | 18:48 |
| fungi | being able to arbitrarily cram e-mail attachments into other programs' stdin as a pipe is great | 18:49 |
| reed | fungi, indeed ... i wish I had time, energy, will power to setup and learn mutt :) | 18:49 |
| fungi | jog0: what just happened there? was that a gate wedgie? | 18:50 |
| reed | I'm betting on Geary Mail but I know it's a long shot | 18:50 |
| jog0 | fungi: things just worked | 18:50 |
| *** jcoufal has joined #openstack-infra | 18:51 | |
| reed | jcoufal, what do you think of the request to post new topics from your askbot to the dev mailing list? | 18:52 |
| fungi | jog0: did all 10 get through? it looked like the new head was suddenly restarted for all tests, which i wouldn't have expected (it should already have had some progress, right?) | 18:52 |
| *** rongze has quit IRC | 18:53 | |
| jog0 | fungi: hmm I blinked | 18:53 |
| jog0 | I think they did all go through though | 18:53 |
| jcoufal | reed: hi! yeah we discussed that and it is great idea. I'll try to do regular weekly summary, I was just buried in other stuff last week, so I''l get to that this week for the first time | 18:54 |
| fungi | jog0: yeah, at the moment they should have merged, there was a gate reset on everything all the way to the head of the queue, but i didn't think to note what the previous changes near the head had been | 18:54 |
| *** sarob has joined #openstack-infra | 18:55 | |
| jeblair | jcoufal, reed: did we get the new askbot running in infra while i was gone? | 18:55 |
| jcoufal | jeblair: nope, it's still on openshift temporary server | 18:55 |
| jog0 | fungi: checking openstack/openstack.git to see | 18:55 |
| jcoufal | jeblair: but we discussed with Monty, since storyboard is coming soon, that we might be early adpoters ant testers | 18:55 |
| *** wenlock has quit IRC | 18:55 | |
| jcoufal | and it might substitute askbot in the end | 18:56 |
| jcoufal | so we are waiting on storyboarding at the moment | 18:56 |
| fungi | i also wonder whether the zuul status page might benefit from a timestamp in the header of each queue in a dependent pipeline indicating when the oldest change in that queue was approved | 18:56 |
| jeblair | jcoufal: okay, that sounds like a good plan! | 18:56 |
| reed | jeblair, I wish it did :( | 18:56 |
| jog0 | fungi: hmm *shrug* the submodules openstack repo hasn't updated yet | 18:56 |
| jeblair | reed: well, for jcoufal at least. still a little sad for ask.o.o. | 18:57 |
| jog0 | fungi: anyway gate is holding up well today | 18:57 |
| *** chuck__ has joined #openstack-infra | 18:57 | |
| *** chuck__ has quit IRC | 18:57 | |
| jcoufal | jeblair: they are still weak in mysql support | 18:57 |
| * fungi nods. the change at the front of the main integrated gate queue has only been in there at little over 2 hours, so probably roughly 2.5 hours from approval to merge if it succeeds | 18:58 | |
| fungi | better than this time yesterday | 18:58 |
| jog0 | yeah | 18:59 |
| jog0 | fungi: and graphite reflects that | 18:59 |
| *** UtahDave has quit IRC | 18:59 | |
| fungi | meeting time | 19:01 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add graphite graph of gate failure rates to e-r page https://review.openstack.org/58019 | 19:02 |
| *** michchap has joined #openstack-infra | 19:02 | |
| *** sarob has quit IRC | 19:02 | |
| *** sarob has joined #openstack-infra | 19:03 | |
| *** osanchez has quit IRC | 19:06 | |
| *** michchap has quit IRC | 19:06 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove queries for dead bugs https://review.openstack.org/57367 | 19:07 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add doc on queries.yaml https://review.openstack.org/57368 | 19:07 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add launchpad support to check_success https://review.openstack.org/57374 | 19:07 |
| dkranz | So stable/grizzly is broken, I think due to novaclient updating iso8601>=0.1.4 to iso8601>=0.1.8 | 19:09 |
| dkranz | How is this supposed to work? | 19:09 |
| dkranz | If I change the minimum version in a client then it can't be installed with previous versions. | 19:09 |
| dkranz | of the servers. | 19:09 |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Unpin keyring, just skip keyring 2.0 https://review.openstack.org/58362 | 19:12 |
| dkranz | jeblair: ^^^ | 19:13 |
| *** wenlock has joined #openstack-infra | 19:20 | |
| fungi | dkranz: yes, i've previously theorized that we might consider having separate virtualenvs for "fooclient as a library dependency used by barserver" and "fooclient used as a tool in the test itself" | 19:21 |
| dkranz | fungi: I see you guys are meeting now | 19:22 |
| fungi | yup | 19:22 |
| dkranz | fungi: We can discuss when you are done | 19:22 |
| dkranz | fungi: I think at the moment we need some action that can get things working again but not sure what that is. | 19:22 |
| fungi | dkranz: however it's also been pointed out that expecting our operators/deployers to stay on an old fooclient when running stable barserver may mean that we're shorting them on security fixes, which is bad (like the cursed frogurt) | 19:23 |
| clarkb | fwiw havana is probably affected too maybe | 19:23 |
| fungi | agreed. need to think on it. i have some security backports which need to pass tests asap before announcing, so it's cranking up my annoyance meter as well | 19:24 |
| dkranz | fungi: In real life the new client would probably not be installed on the same machines as the servers. | 19:24 |
| dkranz | fungi: It's devstack that does that. | 19:24 |
| *** melwitt has joined #openstack-infra | 19:24 | |
| dkranz | fungi: Tricky problem. Seems like out client/server story is still not fully-baked. | 19:25 |
| *** nati_uen_ has joined #openstack-infra | 19:26 | |
| *** wusui has joined #openstack-infra | 19:26 | |
| fungi | dkranz: right, which was teh reason for my separate virtualenv suggestion, but still, security backports for clients used as libraries by server projects... in a cd situation, how to they get security support if we pin the clients there? | 19:26 |
| fungi | though i guess you wouldn't cd a stable branch, so might not be an issue after all | 19:27 |
| fungi | rely on teh distros to backport master fixes to the client releases they're carrying and hope for the best? | 19:27 |
| *** nati_ueno has quit IRC | 19:28 | |
| *** WarrenUsui has quit IRC | 19:28 | |
| jog0 | mordred: https://review.openstack.org/#/c/53372/ | 19:28 |
| dkranz | fungi: I'll think about that. BTW, I am not convinced most operators will be doing cd on any branches | 19:28 |
| fungi | yeah, same here | 19:28 |
| dkranz | fungi: Some will of course. | 19:28 |
| *** nati_uen_ has quit IRC | 19:30 | |
| *** nati_ueno has joined #openstack-infra | 19:30 | |
| mordred | what's that? | 19:32 |
| mordred | I believe CD from stale branches is and most certainly will be happening | 19:33 |
| mordred | remember, there are more operators than just people running public clouds :) | 19:33 |
| *** fbo_away is now known as fbo | 19:34 | |
| fungi | so anyway, dkranz, short term we need to make sure grizzly and savana work with iso8601==0.1.8 and that should get things moving? | 19:34 |
| mordred | dkranz: in real life I expect HP to ship product that includes latest client and stable server and for them to track stable releases | 19:34 |
| mordred | dkranz: because that's what our support policy is as an upstream | 19:34 |
| *** dkliban has quit IRC | 19:35 | |
| fungi | may just be a matter of updating requirements files on stable branches of various projects relying on iso8601 | 19:35 |
| SergeyLukjanov | fungi, we'reinly using timeutils.py from oslo | 19:36 |
| SergeyLukjanov | we're only* | 19:36 |
| dkranz | fungi: That would be the right answer for getting things working on devstack. | 19:36 |
| dkranz | mordred: This becomes hard I think only if you install client/server of different versions on the same machine. Is there a reason to do that outside of devstack? | 19:40 |
| dkranz | mordred: Forcing a stable branch to take a library upgrade just because a client changed doesn't seem right to me. | 19:40 |
| *** mrodden has joined #openstack-infra | 19:45 | |
| *** sarob has quit IRC | 19:46 | |
| *** sarob has joined #openstack-infra | 19:47 | |
| *** smarcet has quit IRC | 19:47 | |
| *** smarcet has joined #openstack-infra | 19:48 | |
| *** rongze has joined #openstack-infra | 19:49 | |
| *** dkliban has joined #openstack-infra | 19:50 | |
| *** sarob_ has joined #openstack-infra | 19:51 | |
| *** sarob has quit IRC | 19:51 | |
| *** pblaho has joined #openstack-infra | 19:52 | |
| *** pblaho has quit IRC | 19:53 | |
| mriedem | sorry if this has been asked, but it sounds like there were some downed jenkins slaves, but it's fixed now - should we recheck/reverify against a specific bug? | 19:53 |
| *** rongze has quit IRC | 19:54 | |
| clarkb | mriedem: yes there is a bug in scroll back (I don't have it handy following the meeting atm) | 19:54 |
| mriedem | clarkb: thanks, looking | 19:54 |
| *** sarob_ has quit IRC | 19:55 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Up the default concurrency on tempest runs https://review.openstack.org/58605 | 19:56 |
| *** SanD_ has joined #openstack-infra | 19:56 | |
| *** sparkycollier has joined #openstack-infra | 19:57 | |
| *** markmc has joined #openstack-infra | 19:58 | |
| mriedem | clarkb: found it, fyi in case anyone asks again :) 1255041 | 19:59 |
| Hunner | mordred: It sounds like the big missing bits in mysql are arrays and hierarchical queries, as well as fast joins, GIN, or partial indexes | 20:02 |
| mordred | well... | 20:02 |
| mordred | fast joins lost me | 20:02 |
| *** michchap has joined #openstack-infra | 20:03 | |
| mordred | since mysql is very fast | 20:03 |
| mordred | also partial index is not a non-feature of postgres | 20:03 |
| Hunner | Perhaps that doesn't mean speed... I'm not DBA enough to know | 20:03 |
| Hunner | Also, "recursive queries" was mentioned, but that may be the same as hierarchical queries | 20:03 |
| *** mrmartin has quit IRC | 20:03 | |
| mordred | sounds like someone who is a postgres fan is saying "it's not postgres" to me - but I've got another meeting I need to be in -let me troll you better in an bit | 20:04 |
| Hunner | mordred: Okay. I'll poke them back to see if they can defend better than fanboy | 20:04 |
| mordred | :) | 20:05 |
| *** derekh has joined #openstack-infra | 20:05 | |
| mordred | Hunner: warn them that I used to be a consultant for MySQL, Inc. specializing in scale out architectures and performance tuning ... so I might be a tough nut to bully :) | 20:05 |
| *** mrmartin has joined #openstack-infra | 20:06 | |
| fungi | Hunner: of particular interest would be finding out what postgres-only features are relied on due to assumptions puppetdb's underlying design | 20:06 |
| *** lcestari has quit IRC | 20:07 | |
| clarkb | I am going to AFK for a bit, need to find lunch. Will dive into codereview afterwards hopefully | 20:07 |
| *** michchap has quit IRC | 20:07 | |
| *** rfolco has quit IRC | 20:07 | |
| *** Ajaeger1 has joined #openstack-infra | 20:11 | |
| Ajaeger1 | Hi infra-team, the openstack-doc team likes to have a new git repository docutils for our tools. | 20:12 |
| Ajaeger1 | Could somebody point me in the right direction on how to request this properly, please? | 20:12 |
| *** datsun180b is now known as datsun_F40PH | 20:12 | |
| *** CaptTofu has quit IRC | 20:13 | |
| *** vipul is now known as vipul-away | 20:14 | |
| *** vipul-away is now known as vipul | 20:14 | |
| *** CaptTofu has joined #openstack-infra | 20:14 | |
| *** hashar has joined #openstack-infra | 20:14 | |
| *** dcramer_ has quit IRC | 20:14 | |
| *** Nayana_ has joined #openstack-infra | 20:16 | |
| nikhil__ | are the gates still kind of slow? | 20:16 |
| *** julim has quit IRC | 20:17 | |
| nikhil__ | there is one job which I've been waiting for last 2 hours, is that something expected atm? | 20:17 |
| Nayana_ | Testing channel conn | 20:17 |
| Hunner | fungi: I think #puppet-dev is where it will be talked about. | 20:17 |
| *** ryanpetrello has joined #openstack-infra | 20:17 | |
| *** sarob has joined #openstack-infra | 20:18 | |
| *** nati_uen_ has joined #openstack-infra | 20:18 | |
| fungi | Ajaeger1: i'm unable to parse your statement. what is "a new git repository docutils"? are you saying your tools rely on features in later versions of docutils than we're currently using on our infrastructure? | 20:21 |
| *** nati_ueno has quit IRC | 20:21 | |
| Ajaeger1 | fungi: We want to create a new git repository called "docutils" and put our scripts in there so that we can share them accross repositories | 20:23 |
| Ajaeger1 | "docutils" is just a name, "doctools" works as well ;) | 20:23 |
| *** ^d is now known as ^demon|away | 20:23 | |
| fungi | Ajaeger1: oh, i see | 20:23 |
| openstackgerrit | Solly Ross proposed a change to openstack-infra/os-loganalyze: Use JS to Up-Filter https://review.openstack.org/57786 | 20:24 |
| openstackgerrit | Solly Ross proposed a change to openstack-infra/os-loganalyze: Introduce Generic Parsing/Filtering Framework https://review.openstack.org/57784 | 20:24 |
| openstackgerrit | Solly Ross proposed a change to openstack-infra/os-loganalyze: Introduce Console Version, Move Common Code https://review.openstack.org/57785 | 20:24 |
| fungi | Ajaeger1: right, confused initially because docutils is already a (rather popular) free software project on its own | 20:24 |
| Ajaeger1 | fungi: in that case, let's use a different name to not confuse further people ;) | 20:24 |
| fungi | Ajaeger1: a great recent example would be the openstack/oslo.rootwrap project addition which just merged earlier today | 20:25 |
| *** hub_cap is now known as pub_tap | 20:25 | |
| * fungi gets you a link to the change | 20:25 | |
| Ajaeger1 | fungi: https://review.openstack.org/#/c/57896/ ? | 20:25 |
| fungi | Ajaeger1: https://review.openstack.org/#/c/57896/ | 20:25 |
| fungi | ha, you're faster than i am ;) | 20:25 |
| Ajaeger1 | fungi: so, this would create the git repository as well as setup acls and gates? | 20:26 |
| fungi | Ajaeger1: yes | 20:27 |
| fungi | Ajaeger1: we also have some documentation written up for adding stackforge projects, which is basically the same thing but the target audience is slightly different of course: http://ci.openstack.org/stackforge.html#add-a-project-to-stackforge | 20:27 |
| fungi | but that adds some great explanations of what those various files are | 20:27 |
| fungi | so it might answer some of your questions as well | 20:28 |
| Ajaeger1 | fungi: Do I need an initial upstream project or is that optional? | 20:28 |
| Ajaeger1 | We want to start from scratch but I can create also something to import... | 20:28 |
| mgagne | Ajaeger1: it is optional | 20:29 |
| Ajaeger1 | mgagne: thanks. | 20:29 |
| fungi | Ajaeger1: it's optional. in the case of oslo.rootwrap it was created using git filter-branch because it was being split out of another existing project and we wanted to preserve the commit history for it | 20:29 |
| *** Nayana_ has quit IRC | 20:29 | |
| *** SanD__ has joined #openstack-infra | 20:29 | |
| fungi | Ajaeger1: if you don't specify an "upstream" parameter in there it will just give you an empty repository with one initial commit adding a .gitreview file | 20:30 |
| Ajaeger1 | Ah, good idea - we could do that as well since we move a file over. | 20:30 |
| *** Nayana has joined #openstack-infra | 20:31 | |
| *** Nayana has quit IRC | 20:32 | |
| fungi | Ajaeger1: when you have the change mostly in the state you want it, i'm happy to help set up a temporary filter-branch'd version with just the file(s) you want to keep for the new project | 20:32 |
| *** SanD__ has quit IRC | 20:32 | |
| Ajaeger1 | fungi: thanks, will ponder now how to do it best ;) | 20:33 |
| fungi | i would offer to write up teh change, but i'm kinda backlogged so trying not to take on more until i get things under control | 20:33 |
| *** nono has joined #openstack-infra | 20:34 | |
| *** denis_makogon_ has joined #openstack-infra | 20:34 | |
| *** nono has quit IRC | 20:35 | |
| Ajaeger1 | fungi: I like a challenge, let me give it a try ;) Thanks! | 20:36 |
| fungi | it's appreciated! | 20:38 |
| *** SanD_ has quit IRC | 20:39 | |
| *** pblaho has joined #openstack-infra | 20:40 | |
| *** yolanda has quit IRC | 20:41 | |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: add an export option https://review.openstack.org/57345 | 20:41 |
| *** mrmartin has quit IRC | 20:41 | |
| *** denis_makogon_ is now known as denis_makogon | 20:41 | |
| *** vipul is now known as vipul-away | 20:41 | |
| *** pblaho has quit IRC | 20:43 | |
| *** dprince has quit IRC | 20:43 | |
| *** dhellmann-afk is now known as dhellmann | 20:44 | |
| *** DennyZhang has joined #openstack-infra | 20:45 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: add nodepool to jenkins-dev server https://review.openstack.org/57333 | 20:45 |
| *** hartsocks has joined #openstack-infra | 20:49 | |
| *** senk has quit IRC | 20:50 | |
| *** rongze has joined #openstack-infra | 20:50 | |
| *** sarob has quit IRC | 20:50 | |
| *** senk has joined #openstack-infra | 20:51 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Setup devstack-gate tests for Savanna https://review.openstack.org/57317 | 20:54 |
| *** rongze has quit IRC | 20:56 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: add nodepool to jenkins-dev server https://review.openstack.org/57333 | 20:56 |
| *** jcoufal has quit IRC | 20:57 | |
| mordred | fungi: don't forget about cookiecutter | 20:57 |
| fungi | mordred: i merged a cookiecutter change earlier today (i think) | 20:58 |
| mordred | fungi: woot | 20:58 |
| mordred | (just saying related ot the Ajaeger1 convo above) | 20:58 |
| fungi | mordred: my problem is, i think, when i manually test these changes i keep finding other broken things which need fixing | 20:58 |
| fungi | mordred: for example, review-dev's puppetry is now broken since https://review.openstack.org/56306 merged yesterday | 20:59 |
| mordred | fungi: really? rats | 20:59 |
| fungi | seems a file got overlooked. change incoming | 20:59 |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/zuul: Add the possibility to pass the zuul url https://review.openstack.org/54196 | 20:59 |
| ArxCruz | jeblair: after 17 iterations I deserve a merge huh? :) | 21:00 |
| fungi | ArxCruz: that's far from a high score ;) | 21:00 |
| fungi | i think i've seen upwards of 50 patchsets on a (not machine generated) change | 21:01 |
| zaro | mgagne: a little late with your approvals, new jjb 0.6 was just released. | 21:01 |
| ArxCruz | fungi: but the person in question was as cool as me? :) | 21:01 |
| mgagne | zaro: it's ok, I read about it in infra meeting notes | 21:01 |
| fungi | ArxCruz: oh, true. i don't think anyone can probably compete on that | 21:02 |
| *** sandywalsh has quit IRC | 21:02 | |
| ArxCruz | :D and I have beard and a big hat :) | 21:02 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add missed milestone-proposed rule https://review.openstack.org/58616 | 21:02 |
| zaro | hashar: did patch to gearman plugin work for you? | 21:03 |
| *** boris-42 has joined #openstack-infra | 21:03 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Added support for Jenkins plugin Blame upstream committers https://review.openstack.org/58341 | 21:03 |
| *** michchap has joined #openstack-infra | 21:03 | |
| hashar | zaro: sorry haven't had time yet to put it on my testing instance. But at least I managed to compile it (for the context I am java/maven illiterate) | 21:03 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: use jjb tests as the examples https://review.openstack.org/57068 | 21:04 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Cleaned up fixtures. Removing vim editor settings https://review.openstack.org/58342 | 21:04 |
| hashar | zaro: definitely want to have your changed tested out in my use case by the end of the week, most probably on friday if not tomororw. | 21:04 |
| hashar | zaro: as soon as I validate it is working for me, I will promptly reply back on the laucnhpad bug and or the gerrit change. | 21:04 |
| SergeyLukjanov | should I file an issue in launchpad to create savanna-milestone group in gerrit or that's enough to create a review? (missed milestone-proposed rule was added to the savanna acl - Add missed https://review.openstack.org/58616) | 21:04 |
| *** sarob has joined #openstack-infra | 21:05 | |
| *** vipul-away is now known as vipul | 21:05 | |
| zaro | hashar: i guess you are not in US. i will probably not be in that day. | 21:05 |
| *** senk has quit IRC | 21:05 | |
| hashar | zaro: that is fine, it can wait a bit :] Thank you for your prompt patch, much appreciated. That granted OpenStack more karma among Wikimedia folks! | 21:06 |
| zaro | mgagne: your on a tear with those approvals. thanks. | 21:06 |
| mgagne | zaro: trying to clear my review queue | 21:07 |
| *** senk has joined #openstack-infra | 21:07 | |
| mgagne | zaro: while I fight/struggle to get my windows images built =) | 21:07 |
| *** michchap has quit IRC | 21:08 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Pass $mysql_password through, part two https://review.openstack.org/58618 | 21:09 |
| fungi | Hunner: ^ | 21:10 |
| fungi | applied manually in a development puppet environemnt and seemed to fix it | 21:11 |
| *** paul-- has joined #openstack-infra | 21:11 | |
| *** smarcet has left #openstack-infra | 21:14 | |
| *** yassine has quit IRC | 21:20 | |
| *** sandywalsh has joined #openstack-infra | 21:20 | |
| *** sarob has quit IRC | 21:22 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Setup devstack-gate tests for Savanna https://review.openstack.org/57317 | 21:25 |
| *** sarob has joined #openstack-infra | 21:26 | |
| *** julim has joined #openstack-infra | 21:26 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Add Savanna testing support https://review.openstack.org/57325 | 21:26 |
| *** jhesketh has joined #openstack-infra | 21:29 | |
| *** rnirmal has quit IRC | 21:31 | |
| *** ArxCruz has quit IRC | 21:32 | |
| *** yamahata_ has joined #openstack-infra | 21:36 | |
| *** ArxCruz has joined #openstack-infra | 21:36 | |
| *** CaptTofu has quit IRC | 21:36 | |
| *** ArxCruz has quit IRC | 21:36 | |
| *** CaptTofu has joined #openstack-infra | 21:36 | |
| *** atiwari has quit IRC | 21:39 | |
| *** dims has quit IRC | 21:40 | |
| *** paul-- has quit IRC | 21:41 | |
| *** Ajaeger1 has quit IRC | 21:44 | |
| *** ArxCruz has joined #openstack-infra | 21:45 | |
| *** dcramer_ has joined #openstack-infra | 21:48 | |
| *** paul-- has joined #openstack-infra | 21:49 | |
| *** dizquierdo has joined #openstack-infra | 21:49 | |
| *** alexpilotti has quit IRC | 21:52 | |
| *** rongze has joined #openstack-infra | 21:52 | |
| *** dims has joined #openstack-infra | 21:54 | |
| *** dcramer_ has quit IRC | 21:58 | |
| *** rongze has quit IRC | 21:59 | |
| *** paul-- has quit IRC | 21:59 | |
| *** DennyZhang has quit IRC | 22:00 | |
| *** yamahata_ has quit IRC | 22:00 | |
| *** pub_tap is now known as hub_cap | 22:01 | |
| *** sparkycollier has quit IRC | 22:01 | |
| *** paul-- has joined #openstack-infra | 22:02 | |
| *** markmc has quit IRC | 22:02 | |
| *** paul-- has quit IRC | 22:03 | |
| *** michchap has joined #openstack-infra | 22:04 | |
| *** derekh has quit IRC | 22:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: Create jenkins03 and jenkins04 https://review.openstack.org/57510 | 22:05 |
| fungi | jeblair: i generated new keys/self-signed certs for those and added them to hiera (even set them to expire the same day as the existing ones for 01 and 02) | 22:05 |
| dolphm | jeblair: i'd love to have one for keystone | 22:05 |
| jeblair | fungi: oh neat! | 22:05 |
| jeblair | dolphm: done! ttx, sdague, jog0: keystone-bugs is open | 22:05 |
| sdague | yay! | 22:06 |
| ttx | yay | 22:06 |
| clarkb | mordred: I am reviewing wheels stuff again | 22:06 |
| ttx | sdague: see in https://lists.launchpad.net/openstack/msg11678.html my proposal to have a single team | 22:06 |
| jog0 | dolphm: thanks | 22:06 |
| dolphm | jeblair: thanks! | 22:06 |
| jog0 | although keystone hasn't had any gate blocking bugs yet :) | 22:06 |
| ttx | sdague: maybe that was too radical though | 22:06 |
| jog0 | dolphm: so keep that up ^ | 22:06 |
| dolphm | jog0: blocking, no... but we've had a couple transients pre-rechecks | 22:07 |
| sdague | yeh, open groups is good enough | 22:07 |
| mordred | clarkb: awesome. I'm afk for hte next couple of hours - my turn driving | 22:07 |
| dolphm | jog0: i think both were only due to flaky tests, though | 22:07 |
| fungi | dolphm: those are sdague's favorites ;) | 22:07 |
| openstackgerrit | Khai Do proposed a change to openstack-dev/pbr: show transitive dependencies https://review.openstack.org/54639 | 22:08 |
| *** ruhe has quit IRC | 22:08 | |
| *** michchap has quit IRC | 22:09 | |
| fungi | i need to do food things between now and my next meeting in ~hour, so am semi-around for the next couple hours | 22:09 |
| *** ArxCruz has quit IRC | 22:10 | |
| *** ArxCruz has joined #openstack-infra | 22:10 | |
| clarkb | fungi: have fun | 22:11 |
| openstackgerrit | Michael Still proposed a change to openstack-infra/config: Sort subscriber map alphabetically https://review.openstack.org/58633 | 22:11 |
| *** michchap has joined #openstack-infra | 22:12 | |
| dolphm | dstanek: do you have a way to join this team? https://launchpad.net/~keystone-bugs | 22:12 |
| dolphm | whoops :) | 22:12 |
| dolphm | dstanek: ooh, you are in this channel! | 22:12 |
| jeblair | mikal: what is that file? | 22:13 |
| mikal | jeblair: the file in the review I just sent? | 22:13 |
| jeblair | mikal: yes | 22:13 |
| jeblair | mikal: i found https://review.openstack.org/#/c/57332/1 | 22:14 |
| *** yamahata_ has joined #openstack-infra | 22:14 | |
| notmyname | I've got a question about git review | 22:14 |
| *** paul-- has joined #openstack-infra | 22:14 | |
| jeblair | mikal: have you written the docs you mentioned in that review? maybe i can just read them? | 22:14 |
| clarkb | notmyname: go for it | 22:14 |
| jeblair | mikal: or otherwise, do you have a big-picture summary? | 22:14 |
| notmyname | clarkb: I'm trying to push a change to milestone-proposed, and I get an error. not sure what I'm doing wrong | 22:14 |
| notmyname | clarkb: " ! [remote rejected] HEAD -> refs/publish/milestone-proposed/bp (branch milestone-proposed/bp not found)" | 22:14 |
| *** paul-- has quit IRC | 22:14 | |
| clarkb | jeblair: https://review.openstack.org/#/c/56816/ and https://review.openstack.org/#/c/56817/ will get pbr to start consuming wheels in its testing (which appears to speed those tests up quite a bit) if you are still reviewing stuff getting eyes on those would be cool | 22:15 |
| notmyname | clarkb: my local branch is "bp" | 22:15 |
| mikal | jeblair: ahhh, that only landed the other day and I am sort of on vacation, so no docs yet | 22:15 |
| mikal | jeblair: perhaps I should add a block comment to the top of the config file explaining its purpose? | 22:15 |
| mikal | jeblair: or would you prefer a summary here? | 22:15 |
| clarkb | notmyname: you are running `git review milestone-proposed` ? | 22:15 |
| notmyname | clarkb: yup | 22:15 |
| notmyname | clarkb: https://gist.github.com/notmyname/1043ada6bbddd0a649e4 | 22:15 |
| jeblair | mikal: it can wait for your return from vacation. :) | 22:15 |
| notmyname | clarkb: full output | 22:15 |
| mikal | jeblair: heh, ok. The short answer is its used by the jeepyb notify_impact trigger to subscribe people to docimpact bugs as they are created. | 22:16 |
| mikal | jeblair: with the intent of encouraging more dev teams to embed a tech writer. | 22:16 |
| jeblair | mikal: got it, thx. | 22:16 |
| mikal | jeblair: nP | 22:16 |
| clarkb | notmyname: I wonder if that is a bug in either git-review or gerrit. It seems to be trying to set bp as the topic since that is the branch you are on, but then gerrit is reading that as part of the branch name instead of being part of the topic | 22:17 |
| *** mrodden has quit IRC | 22:17 | |
| jeblair | clarkb, notmyname: 'git review -v' output? | 22:17 |
| notmyname | clarkb: ah. the branch name "bp" is a weird edge case | 22:17 |
| jeblair | clarkb, notmyname: 'git review -v milestone-proposed' actually | 22:17 |
| notmyname | jeblair: https://gist.github.com/notmyname/40cd59eb2083c4713cef | 22:18 |
| notmyname | clarkb: nope. changing the branch name to "backport" still errors | 22:19 |
| clarkb | notmyname: what if you checkout milestone-proposed locally? git-review will not try setting a topic if the local branch and dest branch have the same name | 22:19 |
| fungi | jeblair: clarkb: swift doesn't have a milestone-proposed branch on gerrit yet | 22:20 |
| jeblair | notmyname: also 'git-review --version' ? | 22:20 |
| clarkb | fungi: ooooohhhhhhhh | 22:20 |
| *** wenlock has quit IRC | 22:20 | |
| notmyname | fungi: ah | 22:20 |
| jeblair | fungi: i made a bad assumption. | 22:20 |
| clarkb | me too | 22:20 |
| notmyname | fungi: so I want to do a backport. is that not the place to do it? | 22:20 |
| fungi | notmyname: a stable backport would get reviewed on the stable/grizzly or stable/havana branch | 22:20 |
| notmyname | fungi: and so I `git review stable/havana`? | 22:21 |
| fungi | notmyname: correct | 22:21 |
| notmyname | fungi: thanks | 22:21 |
| *** vipul is now known as vipul-away | 22:22 | |
| *** vipul-away is now known as vipul | 22:22 | |
| fungi | notmyname: any time! ;) | 22:23 |
| * fungi goes back to pseudolurk mode | 22:24 | |
| *** SergeyLukjanov has quit IRC | 22:25 | |
| *** UtahDave has joined #openstack-infra | 22:26 | |
| notmyname | jeblair: critical issue here: the "gate-tempest-devstack-vm-neutron-large-ops" job name is so long that it's causing the bar to bump down to the next line http://d.not.mn/zuul-wrap-lines.png | 22:26 |
| jeblair | notmyname: there's a patch for that in review | 22:27 |
| notmyname | jeblair: :-) | 22:27 |
| portante | jeblair, notmyname: font size choice in browser affects that, too | 22:28 |
| portante | I shrank my font to make it fit. :) | 22:28 |
| notmyname | portante: critical, critical issue. major blocker. | 22:29 |
| portante | ;) | 22:29 |
| jeblair | notmyname gets my vote for bug triage rights in all projects! | 22:29 |
| portante | use your fonts, luke | 22:29 |
| *** dizquierdo has quit IRC | 22:30 | |
| *** thedodd has quit IRC | 22:33 | |
| *** mriedem has quit IRC | 22:35 | |
| openstackgerrit | Michael Still proposed a change to openstack-infra/config: Add some documentation to the start of subscriber_map https://review.openstack.org/58637 | 22:36 |
| jeblair | mikal: you suck at taking vacation | 22:37 |
| mikal | jeblair: that is true | 22:37 |
| clarkb | mordred: jeblair: I think https://review.openstack.org/#/c/56777/2 can be merged out of the order suggested by mordred | 22:37 |
| mikal | jeblair: I consider this "pottering". I am currently waiting for my wife to be ready to leave the house. | 22:37 |
| clarkb | should I go ahead and approve that change? | 22:37 |
| jeblair | mikal: but i like that your idea of taking vacation is working on infra. you should take more vacation. :) | 22:37 |
| mikal | jeblair: heh | 22:37 |
| mikal | jeblair: well, we had a new guy start this week and his background is in java dev | 22:38 |
| *** yamahata_ has quit IRC | 22:38 | |
| mikal | jeblair: I am trying to trick him into hacking on gerrit... | 22:38 |
| fungi | mikal: it seems the greatest trick in hacking on gerrit it not so much knowing java as convincing google gerrit devs upstream to take your features | 22:39 |
| jeblair | mikal: if he's interested, there may be some concrete work we can point him at. be warned, hacking on gerrit is _hard_. | 22:39 |
| jeblair | mikal: because of what fungi said | 22:39 |
| fungi | by comparison, our reviewers are all unicorns and rainbows | 22:40 |
| clarkb | I think some of that is we aren't well enough integrated into their process to know how they want to do it (they have ~30! contributors in the 2.8 release). But is hard to get to that point because all attemtpts leave us in fail | 22:40 |
| jeblair | mikal: and we're adopting an upstream-first policy for gerrit, because we've been burned too many times. | 22:40 |
| mikal | jeblair: I think that's a fine policy | 22:40 |
| mikal | jeblair: I will get him to come here and have a chat | 22:41 |
| jeblair | mikal: cool | 22:41 |
| sdague | mikal: so you don't like this new guy very much... eh? | 22:41 |
| mikal | sdague: LOL | 22:41 |
| jeblair | clarkb: i don't understand what you mean about 56777 ? | 22:41 |
| fungi | mikal: also, google contributor agreement, so i guess make sure your mgmt are apprised of whatever they have to clear for it | 22:41 |
| mikal | And I quote his welcome email: "He also has a strong background in software development, in both | 22:42 |
| mikal | Java and Python, and has worked on Open Source projects before. Because Matthew can program in | 22:42 |
| mikal | a language none of the rest of us understand, we're hoping to trick him into doing all that icky Java | 22:42 |
| mikal | stuff so none of the rest of us get tainted." | 22:42 |
| *** mattoliverau has joined #openstack-infra | 22:42 | |
| *** hashar has quit IRC | 22:42 | |
| jeblair | mikal: it only took me a few months to get the google cla signed by rax! | 22:42 |
| sdague | unpure! unpure! | 22:42 |
| mikal | jeblair: I've been told that Van Lindburg can make this happen for me | 22:42 |
| *** mrda has joined #openstack-infra | 22:42 | |
| mikal | So, I have to run | 22:42 |
| mikal | But, world meet mattoliverau. | 22:42 |
| mikal | mattoliverau: meet world | 22:43 |
| fungi | mikal: go take a vacation. sheesh | 22:43 |
| mattoliverau | Hi everyone | 22:43 |
| mikal | mattoliverau is the dude with the Java background who we asked to take a look at gerrit. Please scare him off. | 22:43 |
| mikal | :P | 22:43 |
| MarkAtwood | hi mattoliverau | 22:43 |
| mikal | Laters | 22:43 |
| jeblair | mattoliverau: hello! mikal promised us you're interested in working on gerrit | 22:43 |
| mattoliverau | see ya mikal | 22:43 |
| fungi | welcome, mattoliverau | 22:43 |
| clarkb | ahoy | 22:43 |
| jeblair | zaro: ping | 22:43 |
| jeblair | mattoliverau: if that's true, here are some basics about our instance: | 22:44 |
| mattoliverau | jeblair: yeah he has a way of promising :P but yeah, I'm happy to do what I can to make your lives easier :) | 22:44 |
| jeblair | mattoliverau: we're running a local fork of gerrit 2.4.4, maintained in the openstack-infra/gerrit git repo | 22:44 |
| jeblair | mattoliverau: we're running a fork because we needed to add a few things upstream. we got some, but not all of that work upstreamed. | 22:45 |
| sdague | jeblair: where do you have the e-r cron job in config? | 22:45 |
| jeblair | mattoliverau: the last important piece that we did not get upstreamed is our "work in progress" feature | 22:45 |
| sdague | I'd like to carve /elastic-recheck/ out of the config repo, into another repo, so we can CD it in and not make it part of config review process | 22:45 |
| sdague | I'm just trying to make sure I have all the right pieces to start a new repo | 22:46 |
| jeblair | mattoliverau: because of our experience in failing to get that upstream, we have adopted an upstream-first policy | 22:46 |
| jeblair | mattoliverau: so we're not going to upgrade until we have all the features we need upstream (and we also aren't going to develop new features on our local fork) | 22:46 |
| clarkb | mordred: the first change series has been reviewed, a couple of them have comments from either me or jeblair | 22:46 |
| jeblair | mattoliverau: zaro and _david_ (who isn't here right now) are working on getting WIP implemented upstream | 22:46 |
| clarkb | sdague: in modules/openstack_project/manifests/static.pp iirc | 22:47 |
| mattoliverau | jeblair: cool, makes sense, I like the upstream-first policy. | 22:47 |
| jeblair | sdague: hold please | 22:47 |
| sdague | jeblair: ok | 22:48 |
| clarkb | sdague: but should tat be removed from config? it is a config thing to setup cronjobs | 22:48 |
| *** dkranz has quit IRC | 22:48 | |
| jeblair | mattoliverau: you may want to talk to zaro to see if there's something you can help with there... | 22:48 |
| jeblair | mattoliverau: if not, and instead you want to try to hack on something else | 22:48 |
| zaro | yo! | 22:48 |
| mattoliverau | zaro: greetings | 22:49 |
| jeblair | mattoliverau: maybe send something to the openstack-infra@lists.openstack.org mailing list to make sure we're on the same page, and then work on getting that implemented upstream | 22:49 |
| jeblair | mattoliverau: there are some bugs in https://bugs.launchpad.net/openstack-ci/ tagged with 'gerrit' | 22:49 |
| jeblair | mattoliverau: and finally, see http://ci.openstack.org/ for lots of documentation about how the project infrastructure is run (the team, servers, code reviews, etc.) | 22:50 |
| jeblair | mattoliverau: and we're here for any questions. whew. | 22:50 |
| jeblair | sdague: why? | 22:50 |
| zaro | many of those bugs are basically fixed with latest gerrit. | 22:50 |
| jeblair | zaro: good point; maybe we should mark them fix-committed | 22:50 |
| mattoliverau | cool, I'll take a look... and guess see if there are still bugs ;) | 22:51 |
| sdague | jeblair: why what? | 22:51 |
| jeblair | sdague: "I'd like to carve /elastic-recheck/ out of the config repo, into another repo, so we can CD it in and not make it part of config review process"; why? | 22:51 |
| zaro | jeblair: that wouldn't be true right? only fixed upstream. | 22:51 |
| clarkb | zaro: right that is the distinctions between fix committed and fix released | 22:51 |
| sdague | jeblair: so that we can make changes more rapidly than can make it through config review | 22:51 |
| jeblair | zaro: as opposed to fix-released. i'm open to other suggestions though. | 22:51 |
| jeblair | sdague: is it a problem? | 22:52 |
| mattoliverau | Well I'm happy to be of service, I'll take a look at the forked code, bugs, and come back to ask questions as I get up to speed with gerrit. | 22:52 |
| sdague | https://review.openstack.org/#/c/58360/ hasn't seen an infra core look at it yet for 24hrs | 22:52 |
| *** hartsocks1 has joined #openstack-infra | 22:52 | |
| sdague | and like e-r, I expect this is going to evolve more rapidly at the beginning | 22:52 |
| jeblair | sdague: i have a backlog but am getting caught up. i can't speak for other infra-core people. but who would you have review that? | 22:53 |
| sdague | the same group as are on e-r | 22:53 |
| sdague | it's basically e-r dashboard | 22:53 |
| zaro | mattoliverau: you can review my patch. https://gerrit-review.googlesource.com/#/c/48254 | 22:53 |
| zaro | mattoliverau: long history there, maybe another + vote will help spearce to remove his -2 | 22:54 |
| jeblair | sdague: i'm not sure review speed is the right reason to split something like that out. | 22:54 |
| clarkb | sdague: jeblair: I am slightly confused is all of the dashboardy stuff in e-r today and we just have a cron on static? or is there actual code in config? | 22:54 |
| sdague | clarkb: the cron generates a .json | 22:54 |
| *** medberry is now known as med_ | 22:55 | |
| sdague | then it's all d3 for graphs | 22:55 |
| *** rongze has joined #openstack-infra | 22:55 | |
| reed | sdake: yt? | 22:55 |
| reed | sdake_: yt? | 22:55 |
| jeblair | the code is in e-r; the cron and web page are in config. | 22:55 |
| *** hartsocks has quit IRC | 22:55 | |
| *** thomasem has quit IRC | 22:55 | |
| clarkb | gotcha | 22:56 |
| *** denis_makogon has quit IRC | 22:56 | |
| sdake | reed whats up | 22:56 |
| sdake | is yt short for yeti? | 22:56 |
| *** yamahata_ has joined #openstack-infra | 22:57 | |
| jeblair | sdague: do you want to move the web page itself into e-r? | 22:57 |
| reed | sdake: i thought it meant 'you there' | 22:57 |
| reed | :) | 22:57 |
| *** nono1 has joined #openstack-infra | 22:57 | |
| sdague | so I really would like to be able to +2 things into this dashboard that we need to be able express the state of what we find in the gate. So, like with e-r, and os-loganalyze, I think being outside of config makes sense | 22:57 |
| sdake | ;) just kidding around | 22:57 |
| sdague | jeblair: we could go that route | 22:57 |
| sdague | it seemed a little different to me, but I'm ok going that route if you think it best | 22:57 |
| *** jcooley_ has joined #openstack-infra | 22:58 | |
| *** paul-- has joined #openstack-infra | 22:58 | |
| *** nono1 has quit IRC | 22:59 | |
| jeblair | sdague: it probably does make sense; i'm not necessarily arguing that; i think your initial statement raised a lot of red flags, likely due to insufficient development | 22:59 |
| sdague | yep, no worries | 22:59 |
| *** rongze has quit IRC | 22:59 | |
| *** dolphm is now known as dolphm_afk | 23:00 | |
| sdague | so I've got basically one more half day before holidays, was just trying to get a repo set up in a way that it would support fast iteration once I get back. | 23:00 |
| sdague | as I figured that would be concrete :) | 23:00 |
| sdague | and mean no blockers on monday | 23:00 |
| jeblair | sdague: so how about putting it in e-r? | 23:00 |
| sdague | sure | 23:00 |
| clarkb | ++ to putting it in e-r | 23:01 |
| jeblair | sdague: it at least seems like a good first step, and we can split out from there if we're wrong. | 23:01 |
| *** bpokorny has quit IRC | 23:03 | |
| *** che-arne has joined #openstack-infra | 23:03 | |
| *** hartsocks1 has quit IRC | 23:03 | |
| *** jpich has quit IRC | 23:04 | |
| sdague | jog0: you got any issue with that before I start the transplant? | 23:07 |
| *** che-arne has quit IRC | 23:07 | |
| zaro | clarkb: i think you should release this https://github.com/cboylan/jenkins-log-console-log | 23:08 |
| clarkb | zaro: its upstreamed | 23:08 |
| *** nono1 has joined #openstack-infra | 23:08 | |
| clarkb | well it got sucked into the scp plugin | 23:08 |
| *** hartsocks has joined #openstack-infra | 23:08 | |
| zaro | clarkb: scp plugin has ability to save console logs? | 23:09 |
| *** amotoki has quit IRC | 23:09 | |
| mgagne | zaro: yes but it won't copy console timestamps with latest version of console timestamp plugin. | 23:10 |
| clarkb | zaro: it has the ability to copy the console logs to an arbitrary location | 23:10 |
| *** pcm_ has quit IRC | 23:10 | |
| *** che-arne has joined #openstack-infra | 23:10 | |
| *** oubiwann_ has quit IRC | 23:10 | |
| zaro | huh? so you use the scp plugin to copy build console log to the workspace folder? | 23:11 |
| clarkb | zaro: no we use the scp plugin to copy the console logs to the log server | 23:11 |
| zaro | what if you just want it in your workspace so you can include it as a build artifact? | 23:12 |
| clarkb | zaro: that repo is a dead end, you don't actually want to write the log to the workspace then copy it because then you miss out on the data written to the log after you copy | 23:12 |
| clarkb | zaro: you would need to tackle that differently for the reason mentioned above | 23:13 |
| jeblair | the feature is in the scp plugin git repo | 23:13 |
| clarkb | you would probably need to directly artifact the log rather than writing it to the workspace | 23:13 |
| *** datsun_F40PH has quit IRC | 23:15 | |
| sdague | question, is there an easy way to make python setup.py install also install the web files into the right place? or we going to live with a puppet rule to copy them over? | 23:15 |
| *** julim has quit IRC | 23:16 | |
| clarkb | sdague: data_files can be used for that iirc | 23:16 |
| * zaro gets it now. | 23:16 | |
| clarkb | by the way, sitting next to a wood stove with a fire going is really nice | 23:16 |
| sdague | clarkb: you got a good example? | 23:16 |
| clarkb | sdague: let me see if I can find one | 23:17 |
| sdague | clarkb: yeh, I have the wood stove running, though it's on the other side of my house right now :) | 23:17 |
| jog0 | sdague: with regard to putting your dashboard stuff in e-r, sounds like a logical first place to put it | 23:18 |
| *** eharney has quit IRC | 23:18 | |
| sdague | yeh, I'm going to start with just moving over the existing stuff, and get the deploy flow working from there | 23:18 |
| *** senk has quit IRC | 23:19 | |
| jog0 | sdague: awesome, I was just about to suggest that | 23:19 |
| jog0 | make sure my pending patch gets moved as wel | 23:19 |
| jog0 | l | 23:19 |
| clarkb | sdague: in pbr's doc/source/index.rst there is an example of data_files with pbr | 23:20 |
| *** yamahata_ has quit IRC | 23:20 | |
| clarkb | sdague: I am not sure if e-r is pbr'd. if not then in setup.py setup() have a data_files = [(dest_path, source_path), (...), (...)] | 23:21 |
| *** ryanpetrello has quit IRC | 23:22 | |
| *** nono1_ has joined #openstack-infra | 23:24 | |
| *** reaper has quit IRC | 23:26 | |
| *** ryanpetrello has joined #openstack-infra | 23:28 | |
| *** boris-42 has quit IRC | 23:32 | |
| *** mriedem has joined #openstack-infra | 23:32 | |
| sdague | yeh, it seems to be | 23:33 |
| sdague | so those are only system level paths? | 23:33 |
| clarkb | I think so | 23:35 |
| *** paul-- has quit IRC | 23:37 | |
| jog0 | anyone see this before http://logs.openstack.org/62/58362/2/check/check-tempest-devstack-vm-postgres-full/95b9806/console.html | 23:41 |
| jog0 | looks like a real infra bug | 23:41 |
| jog0 | clarkb: ^ | 23:43 |
| *** vipul has quit IRC | 23:43 | |
| clarkb | jog0: review.o.o failed to serve those files for whatever reason, we should look in the apache logs there and see why | 23:43 |
| *** vipul has joined #openstack-infra | 23:44 | |
| *** mfer has quit IRC | 23:44 | |
| jog0 | the funny thing is there are logs for that job | 23:44 |
| jog0 | on logs.o.o | 23:44 |
| jog0 | clarkb: in spirit of not running 'no bug' I wanted to find a bug for this | 23:45 |
| *** vkmc has quit IRC | 23:45 | |
| clarkb | jog0: I don't think there is an existing bug, go ahead and submit one against openstack-ci and copy a link to the build logs there so we know what timeframe to look in logs | 23:45 |
| jog0 | will do thanks | 23:45 |
| jeblair | huh, i think the jjb defs for devstack-gate need to be updated. | 23:46 |
| jeblair | to use git.o.o instead of review. | 23:46 |
| clarkb | oh, that could be, we might have overwhelmed that apache server | 23:46 |
| *** ryanpetrello has quit IRC | 23:47 | |
| jog0 | that makes sense, I'm filing a bug anyway and you can fill in the details as relevant | 23:47 |
| clarkb | ++ | 23:47 |
| jeblair | the graphs look sensible. | 23:47 |
| jog0 | bug 1255353 | 23:50 |
| uvirtbot | Launchpad bug 1255353 in openstack-ci "during gate job 'error: Unable to get pack index https://review.openstack.org/p/openstack-infra/devstack-gate/objects/pack/pack-*.idx'" [Undecided,New] https://launchpad.net/bugs/1255353 | 23:50 |
| jeblair | clarkb: git only tries to fetch pack files like that if it falls back to the dumb client, right? | 23:51 |
| *** weshay has quit IRC | 23:51 | |
| jeblair | clarkb: so presumably gerrit was momentarily slow, and it did that. | 23:51 |
| clarkb | jeblair: I believe that is correct | 23:51 |
| jeblair | clarkb: so changing to git.o.o is still the best fix. | 23:52 |
| clarkb | smart http will stream the bits you need rather than all the things | 23:52 |
| jeblair | triaged and tagged low-hanging-fruit | 23:54 |
| *** MarkAtwood has quit IRC | 23:54 | |
| *** rongze has joined #openstack-infra | 23:56 | |
| *** jcooley_ has quit IRC | 23:56 | |
| *** oubiwann has joined #openstack-infra | 23:56 | |
| jog0 | thanks that was quick | 23:56 |
| *** rcleere has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
