| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: zuul status: Collapse changes by default https://review.openstack.org/63019 | 00:00 |
|---|---|---|
| marun | sdague: the tests themselves will need to manage service instantiation through fixtures | 00:00 |
| russellb | fungi: jeblair we talked about that earlier this morning ^^^ | 00:00 |
| russellb | oh, and sdague ... | 00:00 |
| russellb | but i don't actually know javascript, so *shrug* :-) | 00:00 |
| russellb | works on my box, heh | 00:00 |
| clarkb | I first step of your test can be an unstack | 00:00 |
| sdague | russellb: heh | 00:00 |
| marun | clarkb: gotcha | 00:01 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 00:02 |
| *** senk1 has quit IRC | 00:02 | |
| *** HenryG has quit IRC | 00:05 | |
| jeblair | clarkb, fungi: so we know that with 8 logstash workers running without crm114 we can keep up, so i just changed 09-16 to use crm114. that gives us 16 workers without crm114 and 16 with; it should start to drive the load up a bit but we're still at >2x what we need to keep up. | 00:08 |
| clarkb | jeblair: ok | 00:08 |
| clarkb | jeblair: the cacti graphs for 09-16 seem incomplete | 00:09 |
| *** mriedem has joined #openstack-infra | 00:09 | |
| jeblair | clarkb: they were probably created before the hosts existed so they have bad query data; i'll re-run the queries | 00:09 |
| *** senk1 has joined #openstack-infra | 00:11 | |
| clarkb | jeblair: for error_pr can you give a tl;dr on what the values should be and what they mean? | 00:12 |
| clarkb | looks like they are not in the range 0.0 to 1.0 | 00:12 |
| *** dizquierdo has quit IRC | 00:12 | |
| jeblair | clarkb: actually they should be -320 320, but i don't expect we're there yet | 00:13 |
| clarkb | I see. once training has been done +320 indicates badness -320 goodness? | 00:13 |
| clarkb | I plan on reading lots of docs this weekend to catch up | 00:13 |
| jeblair | clarkb: i'm looking for linkable docs | 00:14 |
| *** vipul is now known as vipul-away | 00:14 | |
| clarkb | btw doing a lucene search for a range looks like error_pr:[0.0 TO 200.0] | 00:14 |
| jeblair | clarkb: cool | 00:15 |
| jeblair | clarkb: quick explanation: http://osdir.com/ml/mail.spam.crm114/2004-09/msg00164.html | 00:15 |
| jeblair | clarkb: there's a longer one in the crm114 book | 00:15 |
| openstackgerrit | A change was merged to openstack-infra/config: Replace DEVSTACK_GATE_ZEROMQ for DEVSTACK_GATE_MQ_DRIVER in jjb https://review.openstack.org/59995 | 00:16 |
| openstackgerrit | A change was merged to openstack-infra/config: Add git2lp mapping for stevedore https://review.openstack.org/61696 | 00:16 |
| jeblair | clarkb: but what i saw in my test was that -1..1 was basically as good as unsure; 1.0<abs(x)<10.0 started to have some correlation; 10.0<abs(x) really meant something | 00:17 |
| jeblair | clarkb: it got up to 37 with my 20 test corpus. | 00:17 |
| *** senk1 has quit IRC | 00:18 | |
| jeblair | clarkb: so i suspect our queries will eventually be looking error_pr:[-1000.0 TO -10.0]; and if we're lucky error_pr:[-1000.0 TO -100.0] | 00:18 |
| clarkb | yup. for those following along pR = log10(PSuccess/PFail) so large negative means really high correlation to fail | 00:19 |
| jeblair | clarkb: sort of -- remember their success/fail are referring to bayesian matches; so it's really the log10 probability that it matches the winning class; that value always start positive (because it's the winning class) but i negate it if the winning class is a jenkins failure as opposed to a jenkins success. | 00:21 |
| clarkb | gotcha | 00:22 |
| jeblair | clarkb: so pretty much, but mostly by coincidence. :) | 00:22 |
| *** dstanek has joined #openstack-infra | 00:22 | |
| *** senk1 has joined #openstack-infra | 00:24 | |
| *** senk1 has quit IRC | 00:24 | |
| *** prad has joined #openstack-infra | 00:27 | |
| *** dstanek has quit IRC | 00:27 | |
| jeblair | oh yeah, these nodes don't have swap. we'll want to do something about that before we start using them for more servers | 00:28 |
| clarkb | jeblair: ya, I think the logstash workers should be fine (they don't use a ton of memory) | 00:29 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: cacti: re-run data queries each time we try to add graphs https://review.openstack.org/63029 | 00:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Add git2lp mapping for all Murano projects https://review.openstack.org/61716 | 00:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Run rtfd hook for stevedore on each merge https://review.openstack.org/62040 | 00:31 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't run requirements integration for Grizzly https://review.openstack.org/62055 | 00:32 |
| *** oubiwann has quit IRC | 00:33 | |
| clarkb | jeblair: https://review.openstack.org/#/c/61850/ would be a good one to get eyes on | 00:33 |
| clarkb | to help with the debugging of failures | 00:33 |
| *** eharney has quit IRC | 00:35 | |
| *** boris-42 has quit IRC | 00:35 | |
| *** dims_ is now known as dims | 00:37 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/elastic-recheck: Add query for bug 1261433 https://review.openstack.org/63030 | 00:38 |
| uvirtbot | Launchpad bug 1261433 in heat "heat-tempest-xx already has an action (DELETE) in progress" [High,Triaged] https://launchpad.net/bugs/1261433 | 00:38 |
| *** senk has joined #openstack-infra | 00:38 | |
| *** AaronGr is now known as AaronGr_afk | 00:40 | |
| *** dstanek has joined #openstack-infra | 00:41 | |
| jog0 | russellb: can you add a filename to that query | 00:41 |
| *** fifieldt has joined #openstack-infra | 00:41 | |
| russellb | jog0: sure, i'm a query noob | 00:42 |
| *** senk has quit IRC | 00:43 | |
| *** senk has joined #openstack-infra | 00:43 | |
| jog0 | russellb: np, i'll help you with this one | 00:44 |
| russellb | let me take another stab | 00:44 |
| russellb | revised logstash query ... http://goo.gl/SiZhwp | 00:45 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/elastic-recheck: Add query for bug 1261433 https://review.openstack.org/63030 | 00:46 |
| uvirtbot | Launchpad bug 1261433 in heat "heat-tempest-xx already has an action (DELETE) in progress" [High,Triaged] https://launchpad.net/bugs/1261433 | 00:46 |
| fungi | mmm, missing swap... should we just mkswap some sparse files on / and point fstab at them? | 00:46 |
| fungi | or do we want to partition the ephemeral disk? | 00:47 |
| fungi | (or use lvm, which is also a perfectly fine way to add a swap device) | 00:47 |
| jog0 | russellb: so the other trick is to find a query that has no false positives | 00:47 |
| jog0 | clicking build status on logstash.o.o shows you how good your query is | 00:48 |
| russellb | ok | 00:48 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Use Fedora 20 for DEVSTACK_GATE_TEMPEST_HEAT_SLOW https://review.openstack.org/61576 | 00:48 |
| russellb | jog0: ah cool | 00:48 |
| russellb | needs more work then :) | 00:48 |
| *** praneshp has quit IRC | 00:48 | |
| jog0 | russellb: yup | 00:49 |
| *** esker has joined #openstack-infra | 00:49 | |
| russellb | jog0: so, how long ago did the switch get flipped on checking for new errors in logs | 00:50 |
| russellb | because that's what is failing jobs here | 00:50 |
| jog0 | russellb: ahhh that makes sense | 00:50 |
| jog0 | a few weeks ago | 00:50 |
| jog0 | so then false positives are OK | 00:50 |
| russellb | well my query was just the last few days and i saw false positives | 00:50 |
| russellb | will play some more | 00:51 |
| jog0 | dkranz sent an email out about when that started | 00:51 |
| russellb | yeah | 00:51 |
| fungi | um, afaik it was late last week that we actually started failing tests on presence of error messages in the output | 00:51 |
| jog0 | russellb: hmm maybe this one is white listed | 00:51 |
| fungi | (back to my "week feels like a year" comment earlier) | 00:51 |
| russellb | jog0: or maybe not enforced on all jobs? | 00:52 |
| fungi | i could be totally wrong on that. i probably just have a very lousy sense of time due to the whole never sleeping thing | 00:52 |
| jog0 | sdague would know better then me | 00:52 |
| russellb | the two SUCCESS cases i've found are both check, neutronclient, check-tempest-dsvm-neutron-pg-isolated | 00:52 |
| russellb | or the same job but neutron | 00:53 |
| *** senk1 has joined #openstack-infra | 00:53 | |
| jog0 | ohh neutron | 00:53 |
| lifeless | I will /lastlog lifeless | 00:53 |
| lifeless | bah | 00:53 |
| russellb | using http://goo.gl/09jMjL | 00:53 |
| lifeless | jog0: btw - 20:14 < lifeless> btw https://bugs.launchpad.net/nova/+bug/1262075 might be the slowdown culprit? 196 | 00:53 |
| uvirtbot | Launchpad bug 1262075 in nova "tempest.api.compute.servers.test_server_rescue.ServerRescueTestXML.test_rescue_unrescue_instance[gate,smoke] failed on postgresql only" [Undecided,New] | 00:53 |
| lifeless | seconds to rescue a server.. | 00:53 |
| lifeless | from last night | 00:53 |
| fungi | dkranz said neutron wasn't being enforced for now because it was virtually impossible to keep up with, right? | 00:54 |
| jog0 | fungi: I thought that was only for neutron stacktraces not all | 00:54 |
| russellb | sdague: heh, you gave that query a -1 without a comment | 00:54 |
| sdague | I did? | 00:55 |
| fungi | ahh, maybe. so much firehose | 00:55 |
| russellb | yar | 00:55 |
| russellb | sdague: may be left as a draft comment on rev1? | 00:55 |
| fungi | russellb: that's so he can game your review stats | 00:55 |
| jog0 | http://logs.openstack.org/43/59743/8/check/check-tempest-dsvm-neutron-pg-isolated/03a1f05/console.html#_2013-12-18_12_22_52_210 | 00:55 |
| sdague | oh, it was draft on v1 | 00:55 |
| jog0 | russellb: that query looks good to me | 00:55 |
| russellb | fungi: damn those stats | 00:55 |
| sdague | it was the and thing | 00:55 |
| jeblair | fungi: i think we should do something with the ephemeral disk; not sure what yet (because we might want to mount part of it somewhere for some servers); needs thinking | 00:55 |
| russellb | sdague: cool :) | 00:55 |
| sdague | I just saved it through | 00:56 |
| jog0 | russellb: although I would prefer to use h-api instead of console.html | 00:56 |
| russellb | thanks | 00:56 |
| jog0 | lifeless: thanks looking | 00:56 |
| russellb | jog0: it's h-api in the commit | 00:56 |
| *** senk has quit IRC | 00:56 | |
| *** CaptTofu has quit IRC | 00:56 | |
| *** CaptTofu has joined #openstack-infra | 00:56 | |
| * jog0 double checks latest patch | 00:56 | |
| sdague | jog0: yeh, it's right in the last patch | 00:57 |
| russellb | woot | 00:57 |
| jog0 | russellb: +A | 00:57 |
| clarkb | fungi: correct, neutron is too noisy | 00:57 |
| russellb | yay | 00:58 |
| sdague | russellb: so probably should get someone on heat to bump this to critical | 00:58 |
| jog0 | SpamapS: ^ | 00:58 |
| sdague | if it's causing that many resets | 00:58 |
| russellb | ack, on it | 00:58 |
| fungi | also, i'm switching to low-bandwidth early tonight on orders from my housemate, so not a lot of reviewing for the remainder of the evening | 00:58 |
| sdague | thanks... and ... really leaving my computer now for the night | 00:58 |
| clarkb | fungi: gnite | 00:59 |
| jog0 | https://launchpad.net/~heat-bugs isn't a open group | 00:59 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1261433 https://review.openstack.org/63030 | 00:59 |
| uvirtbot | Launchpad bug 1261433 in heat "heat-tempest-xx already has an action (DELETE) in progress" [High,Triaged] https://launchpad.net/bugs/1261433 | 00:59 |
| jog0 | lifeless: back to your bug | 00:59 |
| *** hogepodge has quit IRC | 01:02 | |
| *** zhiyan has joined #openstack-infra | 01:04 | |
| clarkb | jeblair: http://logs.openstack.org/47/60347/9/check/check-tempest-dsvm-postgres-full/076523b/logs/devstack-gate-setup-workspace-new.txt.gz swap appears to be working. do we care about the alignment warning | 01:05 |
| *** zhiyan has left #openstack-infra | 01:05 | |
| jeblair | clarkb: not really | 01:05 |
| clarkb | jeblair: I am approving then | 01:06 |
| clarkb | oh wait I suppose I should check hpcloud nodes still work too | 01:06 |
| clarkb | yup http://logs.openstack.org/47/60347/9/check/check-tempest-dsvm-full/4ef9a79/logs/devstack-gate-setup-workspace-new.txt.gz approving | 01:06 |
| *** masayukig has joined #openstack-infra | 01:08 | |
| clarkb | error_pr:[-1000.0 TO -10.0] has results over the last hour now. some of them are lolzy others maybe not so much | 01:09 |
| clarkb | but we probably need a lot more time training | 01:09 |
| *** krotscheck has joined #openstack-infra | 01:10 | |
| jeblair | yeah, let's not evaluate it too early. :) | 01:13 |
| jeblair | also, having a lot of workers means we have a lot of separate datasets to train, so even once all the nodes are running it, depending on how many we keep, it may take a while. | 01:14 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix the yaml-alphabetized job https://review.openstack.org/62951 | 01:14 |
| clarkb | oh right, they have independent css files | 01:14 |
| *** MarkAtwood has quit IRC | 01:15 | |
| *** rwsu has quit IRC | 01:18 | |
| *** rwsu has joined #openstack-infra | 01:19 | |
| clarkb | jeblair: https://review.openstack.org/#/c/48371/1 is WIP because we didn't need it at the time but may be an alternative if we need higher throughput for e-r | 01:19 |
| clarkb | jeblair: once we have swap for d-g nodes on rax performance we should update nodepool to use all performance nodes and allow those nodes to run gate tests too | 01:22 |
| clarkb | jeblair: I am very excited about doing that :) | 01:22 |
| jeblair | clarkb: i think nodepool is doing all performance; at least for devstack nodes; maybe not bare nodes | 01:23 |
| clarkb | jeblair: I think some of the iad nodes are not performance (I saw a job timeout there earlier today) | 01:23 |
| clarkb | hmm nevermind it is | 01:24 |
| *** reed has quit IRC | 01:24 | |
| clarkb | could've been the test at fault too | 01:24 |
| *** senk1 has quit IRC | 01:26 | |
| *** zhiyan has joined #openstack-infra | 01:29 | |
| *** zhiyan has joined #openstack-infra | 01:29 | |
| clarkb | jeblair: https://review.openstack.org/#/c/59993/ is ready now | 01:30 |
| *** jhesketh_ has joined #openstack-infra | 01:31 | |
| *** jhesketh___ has joined #openstack-infra | 01:33 | |
| *** jhesketh___ is now known as jhesketh | 01:33 | |
| *** zhiyan has quit IRC | 01:34 | |
| lifeless | clarkb: hey | 01:34 |
| lifeless | clarkb: remember our back of envelope figures in https://etherpad.openstack.org/p/tripleo-test-cluster | 01:34 |
| lifeless | clarkb: we didn't have numbers for checks. | 01:34 |
| lifeless | clarkb: I've got vendors now sizing racks for us... would be the AWESOME if we could put some numbers on that | 01:35 |
| lifeless | 'We don't have good figures on what checks require - how many concurrent tempest runs are happening at the moment?We don't have good figures on what checks require - how many concurrent tempest runs are happening at the moment?' | 01:35 |
| clarkb | lifeless: graphite is still probably the best place for that info. the graphite link in the etherpad would need updating to use the newer job name | 01:36 |
| *** ryanpetrello has quit IRC | 01:36 | |
| lifeless | clarkb: any chance I can look pitiful and needy and have it magically update through the wonders of parallel editing? | 01:36 |
| clarkb | lifeless: possibly :) | 01:36 |
| * lifeless looks pitiful and needy | 01:37 | |
| *** nosnos has joined #openstack-infra | 01:38 | |
| *** yamahata has joined #openstack-infra | 01:39 | |
| *** ryanpetrello has joined #openstack-infra | 01:40 | |
| *** ryanpetrello has quit IRC | 01:40 | |
| clarkb | lifeless: there you go | 01:40 |
| *** ryanpetrello has joined #openstack-infra | 01:41 | |
| clarkb | lifeless: check appears to peak at ~30 tests per hour or ~100 tests per hour of the big 3 | 01:41 |
| lifeless | clarkb: awesome, thank you! | 01:41 |
| lifeless | they are 30m runs right? | 01:41 |
| clarkb | lifeless: closer to 40something now | 01:41 |
| *** senk1 has joined #openstack-infra | 01:41 | |
| lifeless | so (40m + baremetal overheads) * 100 compute minutes/hour | 01:42 |
| clarkb | we have that data hidden in graphite somewhere too, but I am less knowledgeable on where to find it | 01:42 |
| *** senk1 has quit IRC | 01:43 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Use the ephemeral disk on rackspace performance nodes https://review.openstack.org/60347 | 01:47 |
| *** julim has quit IRC | 01:49 | |
| *** harlowja has quit IRC | 01:51 | |
| *** senk1 has joined #openstack-infra | 01:52 | |
| *** jcooley_ has quit IRC | 01:55 | |
| *** yaguang has joined #openstack-infra | 02:03 | |
| *** harlowja has joined #openstack-infra | 02:03 | |
| lifeless | clarkb: by big three you mean nova/neutron/somethingelse? | 02:04 |
| clarkb | lifeless: nova-network + mysql, nova-network + postgres, neutron + mysql | 02:05 |
| lifeless | oh right | 02:05 |
| lifeless | so the 30 is the incoming patch rate across the integrated gate? | 02:05 |
| clarkb | yup | 02:05 |
| *** slong has quit IRC | 02:07 | |
| *** ryanpetrello has quit IRC | 02:10 | |
| *** slong has joined #openstack-infra | 02:10 | |
| *** rongze has joined #openstack-infra | 02:12 | |
| *** ryanpetrello has joined #openstack-infra | 02:14 | |
| *** senk1 has quit IRC | 02:16 | |
| *** reed has joined #openstack-infra | 02:17 | |
| *** jcooley_ has joined #openstack-infra | 02:27 | |
| *** nosnos_ has joined #openstack-infra | 02:30 | |
| *** nosnos has quit IRC | 02:33 | |
| *** wenlock has joined #openstack-infra | 02:34 | |
| *** jcooley_ has quit IRC | 02:37 | |
| *** wenlock has quit IRC | 02:38 | |
| *** reed has quit IRC | 02:47 | |
| *** nati_ueno has quit IRC | 02:48 | |
| *** fifieldt has quit IRC | 02:51 | |
| *** krotscheck has quit IRC | 02:59 | |
| *** fifieldt has joined #openstack-infra | 03:02 | |
| *** wenlock has joined #openstack-infra | 03:02 | |
| *** vkozhukalov has joined #openstack-infra | 03:06 | |
| *** mriedem has quit IRC | 03:13 | |
| *** ArxCruz has quit IRC | 03:18 | |
| *** dstanek has quit IRC | 03:19 | |
| *** zhiyan has joined #openstack-infra | 03:23 | |
| openstackgerrit | Bill Maxwell proposed a change to openstack-infra/jenkins-job-builder: Implements: Archive publisher allow-empty setting. https://review.openstack.org/62806 | 03:25 |
| *** talluri has joined #openstack-infra | 03:26 | |
| *** jergerber has joined #openstack-infra | 03:27 | |
| *** pcrews has quit IRC | 03:29 | |
| openstackgerrit | Bill Maxwell proposed a change to openstack-infra/jenkins-job-builder: Implements: Archive publisher allow-empty setting. https://review.openstack.org/62806 | 03:30 |
| *** jergerber has quit IRC | 03:30 | |
| *** nicedice has quit IRC | 03:31 | |
| *** nicedice has joined #openstack-infra | 03:32 | |
| *** nati_ueno has joined #openstack-infra | 03:35 | |
| *** jergerber has joined #openstack-infra | 03:37 | |
| *** CaptTofu has quit IRC | 03:40 | |
| *** ryanpetrello has quit IRC | 03:44 | |
| *** CaptTofu has joined #openstack-infra | 03:45 | |
| *** sandywalsh_ has quit IRC | 03:50 | |
| *** julim has joined #openstack-infra | 03:54 | |
| *** krotscheck has joined #openstack-infra | 03:56 | |
| *** julim has quit IRC | 03:57 | |
| *** vipul-away is now known as vipul | 04:01 | |
| *** nati_ueno has quit IRC | 04:01 | |
| *** vipul has joined #openstack-infra | 04:02 | |
| *** krotscheck has quit IRC | 04:04 | |
| *** wenlock has quit IRC | 04:09 | |
| *** sandywalsh has joined #openstack-infra | 04:09 | |
| *** guohliu has joined #openstack-infra | 04:11 | |
| *** yamahata has quit IRC | 04:11 | |
| *** yamahata has joined #openstack-infra | 04:11 | |
| *** yamahata__ has quit IRC | 04:12 | |
| *** yamahata__ has joined #openstack-infra | 04:13 | |
| *** yamahata__ has quit IRC | 04:13 | |
| *** yamahata_ has joined #openstack-infra | 04:13 | |
| *** AaronGr_afk is now known as AaronGr_Zzz | 04:14 | |
| *** dstanek has joined #openstack-infra | 04:15 | |
| *** ryanpetrello has joined #openstack-infra | 04:19 | |
| *** olaph has quit IRC | 04:24 | |
| *** talluri has quit IRC | 04:34 | |
| *** jcooley_ has joined #openstack-infra | 04:39 | |
| *** rcleere has joined #openstack-infra | 04:41 | |
| *** boris-42 has joined #openstack-infra | 04:43 | |
| *** rongze has quit IRC | 04:44 | |
| *** nati_ueno has joined #openstack-infra | 04:46 | |
| *** lcheng has joined #openstack-infra | 04:49 | |
| *** hcc has joined #openstack-infra | 04:55 | |
| *** hcc is now known as hdd_ | 04:55 | |
| *** CaptTofu has quit IRC | 04:56 | |
| *** dstanek has quit IRC | 04:59 | |
| *** chandankumar has joined #openstack-infra | 05:02 | |
| *** rcleere has quit IRC | 05:06 | |
| *** nosnos_ has quit IRC | 05:06 | |
| *** nosnos has joined #openstack-infra | 05:07 | |
| *** talluri has joined #openstack-infra | 05:10 | |
| *** prad has quit IRC | 05:14 | |
| *** CaptTofu has joined #openstack-infra | 05:16 | |
| *** hdd_ has quit IRC | 05:16 | |
| *** sarob has joined #openstack-infra | 05:17 | |
| *** rongze has joined #openstack-infra | 05:18 | |
| *** nicedice has quit IRC | 05:18 | |
| *** basha has joined #openstack-infra | 05:22 | |
| *** rongze has quit IRC | 05:23 | |
| *** sdake2 has quit IRC | 05:28 | |
| *** olaph has joined #openstack-infra | 05:33 | |
| *** jhesketh has quit IRC | 05:33 | |
| *** jhesketh has joined #openstack-infra | 05:35 | |
| *** harlowja is now known as harlowja_away | 05:40 | |
| *** basha has quit IRC | 05:48 | |
| *** basha has joined #openstack-infra | 05:56 | |
| *** harlowja_away is now known as harlowja | 05:59 | |
| *** jergerber has quit IRC | 06:02 | |
| *** sarob has quit IRC | 06:02 | |
| *** sarob has joined #openstack-infra | 06:02 | |
| *** nati_ueno has quit IRC | 06:05 | |
| *** sarob has quit IRC | 06:07 | |
| *** rongze has joined #openstack-infra | 06:20 | |
| *** rongze has quit IRC | 06:25 | |
| *** rongze has joined #openstack-infra | 06:27 | |
| *** rongze has quit IRC | 06:28 | |
| *** senk has joined #openstack-infra | 06:28 | |
| *** dcramer_ has quit IRC | 06:29 | |
| *** dcramer_ has joined #openstack-infra | 06:29 | |
| *** CaptTofu has quit IRC | 06:36 | |
| *** nati_ueno has joined #openstack-infra | 06:37 | |
| *** rongze has joined #openstack-infra | 06:37 | |
| *** CaptTofu has joined #openstack-infra | 06:37 | |
| *** SushilKM has joined #openstack-infra | 06:47 | |
| *** Abhishek_ has joined #openstack-infra | 06:48 | |
| *** jhesketh_ has quit IRC | 06:48 | |
| *** jhesketh has quit IRC | 06:49 | |
| *** vkozhukalov has quit IRC | 06:52 | |
| *** dstanek has joined #openstack-infra | 06:57 | |
| *** praneshp has joined #openstack-infra | 07:00 | |
| *** jhesketh_ has joined #openstack-infra | 07:01 | |
| *** dstanek has quit IRC | 07:01 | |
| *** SushilKM has quit IRC | 07:02 | |
| *** jhesketh has joined #openstack-infra | 07:02 | |
| *** sarob has joined #openstack-infra | 07:03 | |
| *** sarob has quit IRC | 07:08 | |
| *** basha has quit IRC | 07:13 | |
| *** Abhishek_ has quit IRC | 07:13 | |
| *** elasticio has quit IRC | 07:14 | |
| *** zhiwei has joined #openstack-infra | 07:19 | |
| *** yolanda has joined #openstack-infra | 07:19 | |
| zhiwei | hi, we have a task to rename a stackforge project, anyone can tell me how to? | 07:19 |
| zhiwei | there is not desc about renaming projects: http://ci.openstack.org/stackforge.html | 07:20 |
| *** zhiyan has quit IRC | 07:21 | |
| *** zhiyan has joined #openstack-infra | 07:23 | |
| clarkb | zhiwei it requires a gerrit downtime | 07:23 |
| zhiwei | is there any document about this? | 07:23 |
| clarkb | zhiwei: in the past we have asked folks put it on the infra meeting agenda and we go over it there | 07:23 |
| zhiwei | ok, got it. | 07:25 |
| clarkb | I don't think we have a specific doc since it is so involved. it isn't something you can just do | 07:25 |
| *** talluri_ has joined #openstack-infra | 07:26 | |
| clarkb | zhiwei: but it doesnt look like we will have that meeting the next 2 weeks due to holidays. can you send mail to the infra list about it? | 07:28 |
| zhiwei | clarkb: ok, thanks. | 07:28 |
| zhiwei | the project is a Chef cookbook, we need to change cookbook-opentack-metering to cookbook-openstack-telemetry. | 07:29 |
| *** talluri has quit IRC | 07:29 | |
| *** vipul is now known as vipul-away | 07:34 | |
| *** MarkAtwood has joined #openstack-infra | 07:38 | |
| *** talluri has joined #openstack-infra | 07:38 | |
| *** SushilKM has joined #openstack-infra | 07:39 | |
| *** talluri_ has quit IRC | 07:41 | |
| *** Loquacities has quit IRC | 07:44 | |
| *** boris-42 has quit IRC | 07:48 | |
| *** rongze has quit IRC | 07:49 | |
| *** Loquacities has joined #openstack-infra | 07:49 | |
| *** rongze has joined #openstack-infra | 07:49 | |
| *** Loquacities is now known as Loquacity | 07:49 | |
| *** esker has quit IRC | 07:49 | |
| *** nati_ueno has quit IRC | 07:54 | |
| *** esker has joined #openstack-infra | 07:54 | |
| *** senk has quit IRC | 07:54 | |
| *** praneshp_ has joined #openstack-infra | 07:56 | |
| *** che-arne has quit IRC | 07:56 | |
| *** praneshp has quit IRC | 07:58 | |
| *** praneshp_ is now known as praneshp | 07:58 | |
| *** SushilKM__ has joined #openstack-infra | 08:03 | |
| *** SushilKM has quit IRC | 08:04 | |
| *** MarkAtwood has quit IRC | 08:04 | |
| *** sarob has joined #openstack-infra | 08:04 | |
| *** harlowja is now known as harlowja_away | 08:04 | |
| *** sarob has quit IRC | 08:09 | |
| *** basha has joined #openstack-infra | 08:09 | |
| *** jcoufal has joined #openstack-infra | 08:19 | |
| *** sarob has joined #openstack-infra | 08:19 | |
| *** jhesketh has quit IRC | 08:20 | |
| *** jhesketh_ has quit IRC | 08:21 | |
| marun | i think 3rd party testing is great, but the curent gerrit integration solution (spew into the comments) just doesn't scale. | 08:23 |
| *** sarob has quit IRC | 08:23 | |
| *** basha has quit IRC | 08:28 | |
| *** vkozhukalov has joined #openstack-infra | 08:29 | |
| *** dizquierdo has joined #openstack-infra | 08:29 | |
| *** jhesketh has joined #openstack-infra | 08:33 | |
| *** basha has joined #openstack-infra | 08:33 | |
| *** jhesketh_ has joined #openstack-infra | 08:33 | |
| *** flaper87|afk is now known as flaper87 | 08:36 | |
| *** romcheg has joined #openstack-infra | 08:38 | |
| *** pblaho has joined #openstack-infra | 08:45 | |
| *** dstanek has joined #openstack-infra | 08:52 | |
| *** enikanorov has quit IRC | 08:53 | |
| *** marun has quit IRC | 08:53 | |
| *** jcooley_ has quit IRC | 08:54 | |
| *** dstanek has quit IRC | 08:56 | |
| *** vogxn has joined #openstack-infra | 09:01 | |
| *** sarob has joined #openstack-infra | 09:03 | |
| *** yongli is now known as yongli_away | 09:08 | |
| *** sarob has quit IRC | 09:09 | |
| *** zhiyan has quit IRC | 09:11 | |
| *** vogxn has quit IRC | 09:12 | |
| *** zhiyan has joined #openstack-infra | 09:12 | |
| *** zhiyan has quit IRC | 09:13 | |
| *** zhiwei has quit IRC | 09:14 | |
| *** fbo_away is now known as fbo | 09:16 | |
| *** katyafervent has quit IRC | 09:18 | |
| *** yassine has joined #openstack-infra | 09:19 | |
| *** masayukig has quit IRC | 09:23 | |
| *** basha has quit IRC | 09:24 | |
| *** basha has joined #openstack-infra | 09:26 | |
| *** jpich has joined #openstack-infra | 09:27 | |
| *** agordeev has joined #openstack-infra | 09:27 | |
| *** jcooley_ has joined #openstack-infra | 09:38 | |
| *** johnthetubaguy has joined #openstack-infra | 09:47 | |
| *** ruhe has joined #openstack-infra | 09:47 | |
| *** hashar has joined #openstack-infra | 09:48 | |
| *** boris-42 has joined #openstack-infra | 09:52 | |
| *** ruhe has quit IRC | 09:58 | |
| *** ruhe has joined #openstack-infra | 09:58 | |
| *** NikitaKonovalov has joined #openstack-infra | 09:59 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:03 | |
| *** talluri has quit IRC | 10:04 | |
| *** talluri_ has joined #openstack-infra | 10:04 | |
| *** DinaBelova has joined #openstack-infra | 10:04 | |
| *** sarob has joined #openstack-infra | 10:04 | |
| *** thomasbiege1 has quit IRC | 10:07 | |
| *** enikanorov has joined #openstack-infra | 10:08 | |
| *** jcooley_ has quit IRC | 10:09 | |
| *** rongze has quit IRC | 10:11 | |
| *** derekh has joined #openstack-infra | 10:15 | |
| *** thomasbiege has joined #openstack-infra | 10:16 | |
| *** jcooley_ has joined #openstack-infra | 10:19 | |
| *** talluri_ has quit IRC | 10:22 | |
| *** rongze has joined #openstack-infra | 10:23 | |
| *** talluri has joined #openstack-infra | 10:23 | |
| *** sarob has quit IRC | 10:26 | |
| *** NikitaKonovalov has quit IRC | 10:28 | |
| *** NikitaKonovalov has joined #openstack-infra | 10:29 | |
| *** DinaBelova has quit IRC | 10:30 | |
| *** DinaBelova has joined #openstack-infra | 10:30 | |
| *** katyafervent has joined #openstack-infra | 10:31 | |
| *** praneshp has quit IRC | 10:32 | |
| *** fbo is now known as fbo_away | 10:32 | |
| *** dizquierdo has quit IRC | 10:37 | |
| *** lcheng_ has joined #openstack-infra | 10:38 | |
| *** rossella_s has joined #openstack-infra | 10:38 | |
| *** greghaynes has quit IRC | 10:39 | |
| *** fbo_away has quit IRC | 10:39 | |
| *** lcheng has quit IRC | 10:41 | |
| *** greghaynes has joined #openstack-infra | 10:41 | |
| *** fbo has joined #openstack-infra | 10:43 | |
| *** jcooley_ has quit IRC | 10:49 | |
| *** dstanek has joined #openstack-infra | 10:53 | |
| *** ArxCruz has joined #openstack-infra | 10:54 | |
| *** Abhishek_ has joined #openstack-infra | 10:57 | |
| *** dstanek has quit IRC | 10:57 | |
| *** ruhe has quit IRC | 10:59 | |
| *** SushilKM__ has quit IRC | 10:59 | |
| *** nosnos has quit IRC | 11:00 | |
| *** yaguang has quit IRC | 11:00 | |
| *** sarob has joined #openstack-infra | 11:03 | |
| *** ruhe has joined #openstack-infra | 11:04 | |
| *** dizquierdo has joined #openstack-infra | 11:05 | |
| *** rpodolyaka has quit IRC | 11:06 | |
| *** rongze has quit IRC | 11:13 | |
| *** guohliu has quit IRC | 11:18 | |
| *** pblaho has quit IRC | 11:18 | |
| *** mancdaz_away is now known as mancdaz | 11:24 | |
| *** talluri has quit IRC | 11:25 | |
| *** talluri has joined #openstack-infra | 11:26 | |
| *** basha has quit IRC | 11:27 | |
| *** yamahata has quit IRC | 11:28 | |
| *** thomasem has joined #openstack-infra | 11:28 | |
| *** ruhe has quit IRC | 11:32 | |
| *** jcooley_ has joined #openstack-infra | 11:32 | |
| *** ruhe has joined #openstack-infra | 11:32 | |
| *** SushilKM__ has joined #openstack-infra | 11:34 | |
| *** sarob has quit IRC | 11:36 | |
| *** romcheg1 has joined #openstack-infra | 11:47 | |
| *** romcheg has quit IRC | 11:47 | |
| *** weshay has joined #openstack-infra | 11:53 | |
| *** CaptTofu has quit IRC | 12:01 | |
| *** CaptTofu has joined #openstack-infra | 12:01 | |
| *** sarob has joined #openstack-infra | 12:03 | |
| *** jcooley_ has quit IRC | 12:04 | |
| *** senk has joined #openstack-infra | 12:04 | |
| *** rongze has joined #openstack-infra | 12:04 | |
| *** ruhe is now known as ruhe_ | 12:05 | |
| *** ruhe_ has quit IRC | 12:06 | |
| *** smarcet has joined #openstack-infra | 12:08 | |
| *** senk has quit IRC | 12:09 | |
| *** yamahata has joined #openstack-infra | 12:14 | |
| *** dkranz has joined #openstack-infra | 12:16 | |
| *** ruhe has joined #openstack-infra | 12:20 | |
| *** romcheg1 has quit IRC | 12:24 | |
| *** rongze_ has joined #openstack-infra | 12:24 | |
| *** NikitaKonovalov has quit IRC | 12:24 | |
| *** rongze has quit IRC | 12:28 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add support for classification rate https://review.openstack.org/62694 | 12:35 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add per job percentages for responsibility of fails https://review.openstack.org/62738 | 12:35 |
| *** sarob has quit IRC | 12:35 | |
| *** jhesketh_ has quit IRC | 12:37 | |
| *** CaptTofu has quit IRC | 12:37 | |
| *** CaptTofu has joined #openstack-infra | 12:37 | |
| *** xchu has joined #openstack-infra | 12:37 | |
| *** jhesketh has quit IRC | 12:38 | |
| *** CaptTofu has quit IRC | 12:41 | |
| *** dolphm_ has joined #openstack-infra | 12:47 | |
| *** fifieldt has quit IRC | 12:50 | |
| *** weshay has quit IRC | 12:51 | |
| *** SushilKM__ has quit IRC | 12:51 | |
| *** prad has joined #openstack-infra | 12:54 | |
| *** prad has quit IRC | 12:54 | |
| *** dstanek has joined #openstack-infra | 12:55 | |
| *** pblaho has joined #openstack-infra | 12:55 | |
| *** jcooley_ has joined #openstack-infra | 12:57 | |
| *** hashar has quit IRC | 12:58 | |
| *** HenryG has joined #openstack-infra | 13:00 | |
| *** dstanek has quit IRC | 13:00 | |
| *** sarob has joined #openstack-infra | 13:03 | |
| *** hashar has joined #openstack-infra | 13:05 | |
| *** weshay has joined #openstack-infra | 13:06 | |
| *** NikitaKonovalov has joined #openstack-infra | 13:07 | |
| *** hashar_ has joined #openstack-infra | 13:10 | |
| *** hashar has quit IRC | 13:12 | |
| *** hashar_ is now known as hashar | 13:12 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Change mysql-devel to community-mysql-devel in Fedora https://review.openstack.org/62739 | 13:13 |
| *** esker has quit IRC | 13:13 | |
| *** lcheng_ has quit IRC | 13:13 | |
| *** rossella_s has quit IRC | 13:14 | |
| *** dstanek has joined #openstack-infra | 13:18 | |
| *** jcooley_ has quit IRC | 13:27 | |
| *** boris-42 has quit IRC | 13:28 | |
| *** dkranz has quit IRC | 13:29 | |
| *** Abhishek_ has quit IRC | 13:29 | |
| *** heyongli has joined #openstack-infra | 13:31 | |
| *** romcheg has joined #openstack-infra | 13:35 | |
| *** SushilKM__ has joined #openstack-infra | 13:36 | |
| *** sarob has quit IRC | 13:36 | |
| *** rossella_s has joined #openstack-infra | 13:38 | |
| *** zhiyan has joined #openstack-infra | 13:40 | |
| *** miqui has joined #openstack-infra | 13:41 | |
| *** chandankumar_ has joined #openstack-infra | 13:44 | |
| *** chandankumar_ has quit IRC | 13:44 | |
| *** dolphm_ has quit IRC | 13:47 | |
| *** xchu has quit IRC | 13:48 | |
| *** lcheng has joined #openstack-infra | 13:50 | |
| *** dprince has joined #openstack-infra | 13:50 | |
| *** jooools has joined #openstack-infra | 13:51 | |
| *** gothicmindfood has joined #openstack-infra | 13:53 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 13:53 |
| *** sergmelikyan has joined #openstack-infra | 13:53 | |
| sergmelikyan | Can someone take a look at https://review.openstack.org/61703 ? :) | 13:53 |
| *** heyongli has quit IRC | 13:54 | |
| *** yolanda has quit IRC | 13:55 | |
| *** yamahata has quit IRC | 13:58 | |
| *** yamahata has joined #openstack-infra | 13:59 | |
| *** derekh has quit IRC | 14:00 | |
| *** jpich has quit IRC | 14:02 | |
| *** weshay has quit IRC | 14:03 | |
| *** tma996 has joined #openstack-infra | 14:03 | |
| *** sarob has joined #openstack-infra | 14:03 | |
| *** krotscheck has joined #openstack-infra | 14:04 | |
| *** chandankumar has quit IRC | 14:06 | |
| *** sarob has quit IRC | 14:08 | |
| openstackgerrit | Akihiro Motoki proposed a change to openstack-infra/elastic-recheck: Add query for bug 1262613 https://review.openstack.org/63126 | 14:14 |
| uvirtbot | Launchpad bug 1262613 in tempest "tempest.scenario.test_cross_tenant_connectivity.TestNetworkCrossTenant.test_cross_tenant_traffic[compute,gate,network,smoke] failed" [Undecided,New] https://launchpad.net/bugs/1262613 | 14:14 |
| *** dizquierdo has quit IRC | 14:17 | |
| *** jcooley_ has joined #openstack-infra | 14:18 | |
| *** sergmelikyan has quit IRC | 14:27 | |
| *** esker has joined #openstack-infra | 14:27 | |
| *** julim has joined #openstack-infra | 14:30 | |
| *** markmcclain has quit IRC | 14:30 | |
| *** BobBallAway is now known as BobBall | 14:31 | |
| *** mattymo has quit IRC | 14:32 | |
| *** mattymo has joined #openstack-infra | 14:32 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add eavesdrop to monitor Murano channel https://review.openstack.org/61703 | 14:33 |
| *** yaguang has joined #openstack-infra | 14:37 | |
| *** mfer has joined #openstack-infra | 14:42 | |
| *** boris-42 has joined #openstack-infra | 14:45 | |
| hughsaunders | hey infra, I pushed a new patchset to an existing review by mistake. I want to push the previous patch set back, but I get a git error "no changes made" how can I avoid this and revert to the previous patchset? | 14:45 |
| fungi | hughsaunders: you can use 'git review -d changenum,patchsetnum' to download a specific patchset, and then 'git review' to propose it back to that change | 14:47 |
| fungi | so something like 'git review -d 12345,6 && git review' should do it | 14:47 |
| fungi | assuming you want to switch back to patchset #6 of change 12345 | 14:48 |
| hughsaunders | thanks fungi, will try that | 14:48 |
| *** openstack has joined #openstack-infra | 14:49 | |
| fungi | flaper87: did you finish investigating your unit test issue on that temporary vm? if so i'll go ahead and discard it | 14:50 |
| flaper87 | fungi: yup, I pingged you yda, I guess you missed that message | 14:50 |
| flaper87 | fungi: thank you very much! | 14:50 |
| fungi | flaper87: oops! i probably did. was busy yesterday--you're welcome of course! | 14:51 |
| *** jcooley_ has quit IRC | 14:52 | |
| *** mriedem has joined #openstack-infra | 14:53 | |
| hughsaunders | fungi: I can't review the previous patchset directly after fetching it as there have been commits since, so I have to rebase first. After rebasing, I tried a review and get the same errors "remote rejected" "no changes made". | 14:54 |
| openstackgerrit | Sahid Orentino Ferdjaoui proposed a change to openstack/requirements: Tox fails to build environment because of MySQL-Python https://review.openstack.org/62028 | 14:55 |
| fungi | hughsaunders: what's the change and patchset? i'll try and see if i can tell what's going on | 14:55 |
| *** dcramer_ has quit IRC | 14:55 | |
| hughsaunders | fungi: https://review.openstack.org/#/c/62949/ | 14:55 |
| hughsaunders | trying to revert to patchset 5 | 14:55 |
| *** johnthetubaguy1 has joined #openstack-infra | 14:57 | |
| fungi | doesn't seem to have really needed a rebase on master--i didn't get any merge conflicts | 14:58 |
| *** kgriffs_afk is now known as kgriffs | 14:58 | |
| *** johnthetubaguy has quit IRC | 14:58 | |
| fungi | normally you shouldn't rebase changes in code review unless they're in conflict (or unless you want to stack them on top of another open review) since it creates unnecessary deltas for reviewers to have to wade through | 14:58 |
| *** markmcclain has joined #openstack-infra | 15:01 | |
| fungi | hughsaunders: though i just downloaded patchset 5, rebased it on master, then downloaded patchset 6 and diffed the two working branches... they're identical | 15:01 |
| fungi | so that's why gerrit's telling you there are no changes | 15:02 |
| hughsaunders | so why was patchset 6 accepted with no changes? | 15:02 |
| fungi | patchset 6 was patchset 5 rebased on master | 15:02 |
| *** prad has joined #openstack-infra | 15:03 | |
| fungi | master hasn't changed since then, so rebasing patchset 5 on master again will get you a change which is identical to patchset 6 | 15:03 |
| hughsaunders | ahh | 15:03 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Turn on devstack exercises for neutron (non-voting) https://review.openstack.org/62930 | 15:03 |
| *** julim has quit IRC | 15:03 | |
| *** sarob has joined #openstack-infra | 15:03 | |
| *** eharney has joined #openstack-infra | 15:04 | |
| hughsaunders | In that case, I have no need to revert | 15:04 |
| fungi | based on your comments in that review, i'm guessing maybe you had other alterations you meant to submit for review but never committed them? | 15:04 |
| *** kgriffs has left #openstack-infra | 15:04 | |
| hughsaunders | the original patch contained more changes, I have been splitting the patch set into two, I now need to submit the second half as a dependant review. | 15:05 |
| fungi | ahh, yeah looks like earlier patchsets were indeed larger | 15:06 |
| *** julim has joined #openstack-infra | 15:07 | |
| *** DinaBelova has quit IRC | 15:07 | |
| *** thomasbiege has left #openstack-infra | 15:07 | |
| fungi | so you do that by making your new changes on top of the current working branch, but then do a 'git commit -a' instead of 'git commit -a --amend' and you should end up with a new change dependent on the existing one that way | 15:07 |
| *** rnirmal has joined #openstack-infra | 15:08 | |
| fungi | when you 'git review' that, you'll see the dependency between those changes tracked in the "dependencies" list in the gerrit webui | 15:08 |
| *** blamar has joined #openstack-infra | 15:08 | |
| *** jergerber has joined #openstack-infra | 15:11 | |
| *** DinaBelova has joined #openstack-infra | 15:12 | |
| hughsaunders | success: https://review.openstack.org/#/c/63138/ thanks for your assistance fungi | 15:12 |
| fungi | hughsaunders: no problem. always happy to help! | 15:12 |
| *** mestery_ has joined #openstack-infra | 15:12 | |
| *** sarob has quit IRC | 15:12 | |
| *** mestery has quit IRC | 15:16 | |
| *** blamar has quit IRC | 15:16 | |
| *** krotscheck has quit IRC | 15:16 | |
| *** blamar has joined #openstack-infra | 15:17 | |
| *** DinaBelova has quit IRC | 15:22 | |
| *** mestery has joined #openstack-infra | 15:22 | |
| *** NikitaKonovalov has quit IRC | 15:22 | |
| *** mestery_ has quit IRC | 15:25 | |
| *** dizquierdo has joined #openstack-infra | 15:26 | |
| *** rcleere has joined #openstack-infra | 15:27 | |
| *** lcheng has quit IRC | 15:29 | |
| *** NikitaKonovalov has joined #openstack-infra | 15:30 | |
| *** lcheng has joined #openstack-infra | 15:32 | |
| *** senk has joined #openstack-infra | 15:35 | |
| *** lcheng has quit IRC | 15:37 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 15:38 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Add SQLalchemy database model https://review.openstack.org/62239 | 15:38 |
| *** rcleere has quit IRC | 15:39 | |
| *** lcheng has joined #openstack-infra | 15:39 | |
| *** vkozhukalov has quit IRC | 15:40 | |
| *** DinaBelova has joined #openstack-infra | 15:41 | |
| *** rongze has joined #openstack-infra | 15:42 | |
| *** dcramer_ has joined #openstack-infra | 15:43 | |
| *** jcooley_ has joined #openstack-infra | 15:44 | |
| *** rongze_ has quit IRC | 15:45 | |
| *** dripton_ is now known as dripton | 15:47 | |
| *** gothicmindfood has quit IRC | 15:48 | |
| jd__ | mordred: ping | 15:49 |
| *** CaptTofu has joined #openstack-infra | 15:50 | |
| *** CaptTofu has quit IRC | 15:52 | |
| *** CaptTofu has joined #openstack-infra | 15:53 | |
| *** rossella_s has quit IRC | 15:54 | |
| *** NikitaKonovalov has quit IRC | 15:55 | |
| *** SergeyLukjanov has quit IRC | 15:56 | |
| *** CaptTofu has quit IRC | 15:57 | |
| *** lcheng has quit IRC | 15:58 | |
| *** fbo has quit IRC | 15:59 | |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Rename the subscriber map to be a more generic config file. https://review.openstack.org/62073 | 16:00 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Turn on devstack exercises for neutron (non-voting) https://review.openstack.org/62930 | 16:00 |
| *** julim has quit IRC | 16:01 | |
| *** fbo has joined #openstack-infra | 16:01 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Rename launchpad parameter to group https://review.openstack.org/63157 | 16:02 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Remove the hardcoded project list https://review.openstack.org/63158 | 16:02 |
| mordred | jd__: sup? | 16:02 |
| *** julim has joined #openstack-infra | 16:02 | |
| jd__ | mordred: I wanna know what's your plan for Python 3 in requirements/pbr/whatever because my patch has been blocked for too long for that :) | 16:02 |
| mordred | jd__: ah! piddle. you are asking sensible questions | 16:02 |
| jd__ | that sounds like me | 16:03 |
| * jd__ ties mordred | 16:03 | |
| *** DinaBelova has quit IRC | 16:03 | |
| *** sarob has joined #openstack-infra | 16:03 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Turn on devstack exercises for neutron (non-voting) https://review.openstack.org/62930 | 16:04 |
| *** rcleere has joined #openstack-infra | 16:04 | |
| *** SushilKM__ has quit IRC | 16:05 | |
| mordred | jd__: looking for the link with the example ... | 16:05 |
| jd__ | :) | 16:06 |
| *** sarob has quit IRC | 16:07 | |
| *** mancdaz is now known as mancdaz_away | 16:08 | |
| mordred | jd__: ok - so the basis is this: http://www.python.org/dev/peps/pep-0426/#environment-markers | 16:11 |
| mordred | jd__: and what I want to do is add support into the requires-dist field of setup.cfg | 16:12 |
| jd__ | ah, PEP 426, I see where that is going :) | 16:12 |
| mordred | for doing something like "foo; python_version >= '3.3'" | 16:13 |
| mordred | so, for things that have python version things, we'd put them in setup.cfg instead - however, this is problematic for about 12 reasons, so I might be wrong about it | 16:14 |
| jd__ | yeah I'm trying to get my head around that | 16:15 |
| *** rongze has quit IRC | 16:15 | |
| *** jcooley_ has quit IRC | 16:16 | |
| *** rongze has joined #openstack-infra | 16:16 | |
| jd__ | I'm going to hit haypo with that | 16:17 |
| *** pcrews has joined #openstack-infra | 16:18 | |
| *** SushilKM__ has joined #openstack-infra | 16:18 | |
| *** pblaho has quit IRC | 16:18 | |
| mordred | jd__: it's also quite possibly just a TERRIBLE idea - but if we can get it right, it seems nice and flexible | 16:19 |
| jd__ | agreed | 16:20 |
| jd__ | especially if we envision a Windows port | 16:20 |
| jd__ | *cough* | 16:20 |
| *** sdake_ has joined #openstack-infra | 16:22 | |
| *** sdake_ has quit IRC | 16:22 | |
| *** sdake_ has joined #openstack-infra | 16:22 | |
| *** rongze has quit IRC | 16:22 | |
| *** rongze has joined #openstack-infra | 16:22 | |
| jd__ | mordred: seriously, you think you'll have time to work on that anytime soon or should I bring my workforce on that? | 16:23 |
| jd__ | by work on that, that could even be "look if it's a good idea" | 16:23 |
| mordred | hrm. if you have a workforce to whom that makes sense, that would be handy! | 16:23 |
| jd__ | I do have | 16:23 |
| * jd__ tries to invocate it | 16:24 | |
| *** thomasbiege1 has joined #openstack-infra | 16:24 | |
| *** haypo has joined #openstack-infra | 16:24 | |
| jd__ | <haypo> jd_: https://distlib.readthedocs.org/en/0.1.3/tutorial.html#using-markers | 16:24 |
| jd__ | that looks nice | 16:24 |
| jd__ | haypo, mordred, mordred, haypo | 16:24 |
| *** yaguang has quit IRC | 16:25 | |
| haypo | mordred: hi ;) | 16:25 |
| *** talluri has quit IRC | 16:25 | |
| jd__ | mordred: haypo's a Python code dev and he's helping with Python 3 port | 16:25 |
| haypo | mordred, jd__ : the purpose of distlib is to avoid duplicated and incompatible libraries for distribution, we should reuse it | 16:25 |
| jd__ | mordred: so that's a good workforce to use | 16:25 |
| *** talluri has joined #openstack-infra | 16:25 | |
| mordred | haypo: distlib is abandoned | 16:25 |
| mordred | no? | 16:25 |
| *** dims has quit IRC | 16:26 | |
| mordred | haypo: if it's not, then I agree | 16:26 |
| *** vipul-away is now known as vipul | 16:26 | |
| haypo | mordred: oh? distutils2/packaging was abandoned but the best code was copied to a new clean module, distlib | 16:26 |
| mordred | ah - neat | 16:26 |
| jd__ | I think it failed to be in Python 3.4 but not sure it's abandoned | 16:27 |
| jd__ | somehting like that | 16:27 |
| haypo | distlib is supposed to be the new base for pip and the "new distutils" (distutils3? i don't know) | 16:27 |
| * jd__ lols at distutils3 | 16:27 | |
| mordred | hrm. we'd have to extern it | 16:27 |
| mordred | because adding depends to pbr is a no-go | 16:27 |
| mordred | since that gets processed by easy_install which deos not work :) | 16:28 |
| haypo | mordred: i'm trying to stay avoid from packaging, so i may not have the latest news | 16:28 |
| mordred | haypo: tell me about it | 16:28 |
| *** ruhe has quit IRC | 16:28 | |
| *** talluri has quit IRC | 16:30 | |
| mordred | oh - this is on the other side - it's for interacting with distributions (I remember now more things about the split) | 16:30 |
| *** dims has joined #openstack-infra | 16:30 | |
| haypo | mordred: oh, pip already uses distlib, and it has a copy in pip._vendor.distlib, https://github.com/pypa/pip/tree/develop/pip/_vendor/distlib | 16:32 |
| mordred | ugh. everytime I dig more into the state of this I want to die | 16:32 |
| haypo | mordred: serisouly, it's moving forward, and fast. there is now an up to date guide: https://python-packaging-user-guide.readthedocs.org/en/latest/ | 16:32 |
| * jd__ holds a spoon out to mordred | 16:33 | |
| *** jcooley_ has joined #openstack-infra | 16:33 | |
| haypo | mordred: a module to install pip will be part of Python 3.4 (ensurepip), the new standard to bundle sources if the new "wheel" format (faster, and safer: don't execute code to install package), distlib is the new core library to build tools | 16:33 |
| mordred | yup | 16:34 |
| mordred | except | 16:34 |
| haypo | is* the new | 16:34 |
| mordred | there is a bootstrapping problem that none of that solves | 16:34 |
| haypo | mordred: which bootstrap issue? | 16:34 |
| mordred | haypo: people, like us, who won't be on 3.4 for a good 8 years | 16:34 |
| mordred | haypo: so the decision to not backport ensurepip to old pythons is a bit of a killer | 16:35 |
| haypo | mordred: the problem will be solved in 3.4. if you cannot upgrade, yes, it's tricky to install pip | 16:35 |
| mordred | yeah. we cannot upgrade for a LONG TIME | 16:35 |
| haypo | mordred: i just proposed to reuse distlib to not write a new implementation of markers, i don't want to change anything else :) | 16:36 |
| fungi | particularly problematic to not have it backported to 2.7, since "enterprise" linux distros like rhel will be a long time before they default to py3k | 16:36 |
| mordred | which is why I get frustrated at the attempts to solve this in the standard lib | 16:36 |
| mordred | haypo: :) | 16:36 |
| mordred | haypo: I certainly don't want to write a new implementation of markers :) | 16:36 |
| haypo | fungi: RHEL provides Python 3.3. did you read my article i published this morning? http://techs.enovance.com/6521/openstack_python3 | 16:36 |
| mordred | haypo: but I will need to write something customer probably to be able to pull them out of setup.cfg that we use and pass them to distlib | 16:36 |
| haypo | fungi: i'm porting OpenStack stuff to python 3 ;) | 16:37 |
| fungi | haypo: excellent--we have lots of people porting openstack to python 3, so i'm sure your efforts will be most welcome | 16:37 |
| mordred | I welcome these efforts | 16:37 |
| haypo | mordred: sorry, but i'm lost. i joined at the middle of the discussion. which program uses requirements.txt? pip? | 16:38 |
| *** HenryG_ has joined #openstack-infra | 16:38 | |
| haypo | mordred: pip does not already support markers by the way? | 16:38 |
| mordred | haypo: yes. which is why it's problematic | 16:38 |
| mordred | because they do not (and will not) support putting markers into requirements.txt | 16:38 |
| *** tma996 has quit IRC | 16:38 | |
| *** SergeyLukjanov has joined #openstack-infra | 16:38 | |
| mordred | so if we want to use them, we'll need to get clever | 16:38 |
| mordred | which makes me sad | 16:38 |
| haypo | mordred: oh? i'm surprised that pip doesn't support markers right now | 16:39 |
| mordred | me too! | 16:39 |
| mordred | if it did, that would make this entire conversatoin go away actually :) | 16:39 |
| openstackgerrit | A change was merged to openstack-infra/config: Make savanna-ml able to create branches https://review.openstack.org/61400 | 16:39 |
| fungi | haypo: as for rhel providing py3k (still not "defaulting to it" but at least it's a start), you should join the effort to convince them to do it in a way so that it's not pay-only. right now their py3k support efforts can not be imported to free distros like centos as a result | 16:40 |
| haypo | mordred: where do you read that pip doesn't want to implement markers? | 16:40 |
| haypo | mordred: the work (supporting markers) should be done upstream (pip), not in openstack | 16:40 |
| mordred | haypo: dstufft said so in here when I asked him if I could write a patch to add support | 16:40 |
| mordred | haypo: ++ | 16:40 |
| mordred | haypo: I only ever want to do things here when upstream tells me to go away | 16:41 |
| *** HenryG has quit IRC | 16:41 | |
| mordred | (not that dstufft ever tells me to go away, he's quite awesome) | 16:41 |
| *** reed has joined #openstack-infra | 16:42 | |
| haypo | fungi: i don't know the details for RedHat, but the idea is that most Linux distro are moving to Python 3 | 16:42 |
| haypo | fungi: so if you want to invest on the future, you must be prepared for Python 3 | 16:42 |
| *** jcoufal has quit IRC | 16:42 | |
| reed | fungi, who's in gerrit group openstackid-core? | 16:42 |
| mordred | haypo: absolutely | 16:42 |
| fungi | haypo: yes, this is not news to me (i'm a huge proponent of that effort) | 16:42 |
| haypo | fungi: but yeah, today, it's still a pain to use Python 3 on some Linux distro (especially RedHat) | 16:42 |
| mordred | haypo: we want to be prepared for it - but we also have a requirement to still support back to 2.6 (which is crazypants) | 16:43 |
| haypo | fungi: "RedHat Software Collections" is not free? | 16:43 |
| fungi | reed: smarcet is the only current direct member of that group, but i have the group set to include the infra core group too so that we can help out if need be | 16:43 |
| fungi | haypo: nope | 16:43 |
| haypo | fungi: i suppose that if you choose RedHat, you are ready to pay RedHat for python3 support | 16:43 |
| reed | fungi, thanks | 16:43 |
| *** danger_f` is now known as dangers | 16:44 | |
| jeblair | SergeyLukjanov, mordred: low priority; but if one of you have a moment to look at my question in https://review.openstack.org/#/c/62074/ I would appreciate it. | 16:45 |
| fungi | haypo: right, though it does leave the rhel clone distros like centos and scientific-linux (or whatever it's called) out in the cold as far as having to reimplement python 3 support independently of red hat's work | 16:45 |
| haypo | jd__: (i don't know if someone is working on a "distutils3", i'm probably wrong. "distutils3" is in fact pip) | 16:45 |
| mordred | jeblair: I will look now | 16:45 |
| mordred | jeblair: I agree with you | 16:46 |
| *** dims_ has joined #openstack-infra | 16:46 | |
| jd__ | haypo: yeah it just the name that would make me laugh | 16:46 |
| mordred | jeblair: in fact, I have responded now | 16:46 |
| jeblair | mordred: cool, thx | 16:47 |
| haypo | mordred, jd__ : oh, PEP 426 is still a draft :-/ | 16:47 |
| *** ruhe has joined #openstack-infra | 16:47 | |
| haypo | mordred: you don't like jd__ proposition to have a different requirements.txt file per Python version (2 and 3)? | 16:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Finish the projects.yaml.erb rename https://review.openstack.org/62188 | 16:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Unlaunchpadify projects.yaml https://review.openstack.org/62189 | 16:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Track direct-release projects in projects.yaml https://review.openstack.org/62190 | 16:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Split config from projects list https://review.openstack.org/62187 | 16:47 |
| mordred | haypo: well, I kinda don't because although it's easy enough to do, it doesn't really solve the problem generally - it just makes a 3.x split | 16:48 |
| mordred | we currently have a list of packages that we actually strip out of things for 2.6/2.7 issues | 16:48 |
| mordred | and I'd love to just be able to express the actual needs | 16:49 |
| mordred | BUT - I could also just suck it up and admit that perfect is sometimes the enemy of good | 16:49 |
| mgagne | what was the sphinx ext someone found to generate graphs? | 16:49 |
| *** dizquierdo has quit IRC | 16:50 | |
| *** eharney_ has joined #openstack-infra | 16:50 | |
| *** eharney has quit IRC | 16:51 | |
| jd__ | mordred: well I think that at least good does not prevent us to do more good/perfect later | 16:51 |
| *** eharney_ is now known as eharney | 16:51 | |
| *** fbo_ has joined #openstack-infra | 16:51 | |
| *** jcooley_ has quit IRC | 16:52 | |
| *** dims has quit IRC | 16:52 | |
| *** fbo has quit IRC | 16:52 | |
| *** dkranz has quit IRC | 16:52 | |
| *** mattymo has quit IRC | 16:52 | |
| *** amotoki has quit IRC | 16:52 | |
| mordred | jd__: yah. I think you're right. however, we should probably add support for consuming those files to pbr | 16:52 |
| *** fbo_ is now known as fbo | 16:52 | |
| jd__ | mordred: I can do that | 16:52 |
| mordred | perhaps have it look for requirements-py$majorversion.txt and process it too if it's there | 16:53 |
| *** dolphm_ has joined #openstack-infra | 16:53 | |
| *** AaronGr_Zzz is now known as AaronGr | 16:54 | |
| *** dkranz has joined #openstack-infra | 16:55 | |
| *** jcooley_ has joined #openstack-infra | 16:55 | |
| *** mattymo has joined #openstack-infra | 16:55 | |
| *** amotoki has joined #openstack-infra | 16:55 | |
| *** SushilKM__ has quit IRC | 16:56 | |
| *** dkranz has quit IRC | 16:58 | |
| *** jcooley_ has quit IRC | 16:58 | |
| *** mattymo has quit IRC | 16:58 | |
| *** amotoki has quit IRC | 16:58 | |
| *** talluri has joined #openstack-infra | 16:59 | |
| haypo | jd__: i sent an email to PEP 426 authors to get a status of markers and pip | 17:00 |
| jd__ | haypo: cool | 17:01 |
| jd__ | I'll take a look at patching pbr for having a good enough solution the time being | 17:01 |
| haypo | nick is reactive, he's the one who wrote and implemented ensurepip | 17:01 |
| haypo | thanks to ensurepip many bugs were fixed recently (validate correctly SSL certificates, support python with threading support disabled, etc.) | 17:02 |
| *** SushilKM__ has joined #openstack-infra | 17:02 | |
| mordred | yeah - nick is great - and ensurepip is a great idea | 17:02 |
| *** thedodd has joined #openstack-infra | 17:03 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 17:03 | |
| *** lcheng has joined #openstack-infra | 17:04 | |
| *** _SergeyLukjanov has quit IRC | 17:04 | |
| *** dkranz has joined #openstack-infra | 17:04 | |
| *** jcooley_ has joined #openstack-infra | 17:04 | |
| *** mattymo has joined #openstack-infra | 17:04 | |
| *** amotoki has joined #openstack-infra | 17:04 | |
| jeblair | sdague: aren't we trying to stop running devstack exercises? (because tempest should ideally cover everything?) | 17:05 |
| fungi | jeblair: sdague: what's the suggestion for shoring up grenade to no longer rely on exercises to vet the base (old) devstack install? if this is about the devstack-neutron job addition, it's a temporary thing to get neutron able to be grenaded again | 17:07 |
| *** UtahDave has joined #openstack-infra | 17:07 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:07 | |
| haypo | ok, i have to leave. i will come back if i get some news ;) even if we don't have a solution, the discussion moved forward ;) | 17:07 |
| * mordred supports grenading neutron | 17:07 | |
| haypo | bye | 17:07 |
| mordred | haypo: awesome. thanks! | 17:07 |
| *** haypo has left #openstack-infra | 17:08 | |
| jeblair | fungi: that is what prompted it; i'm talking about in the context of tempest tests, not grenade | 17:08 |
| sdague | jeblair: we still need small sanity checks in grenade | 17:08 |
| fungi | (right now neutron grenade is busted and non-voting in part because devstack exercises for neutron instabroke as soon as they were removed from the tempest jobs) | 17:08 |
| sdague | jeblair: right, but how do you get the exercises working? | 17:08 |
| jeblair | sdague: fill me in here -- i'm not trying to argue a point, i just want to know where we are... | 17:08 |
| *** dkranz has quit IRC | 17:09 | |
| *** jcooley_ has quit IRC | 17:09 | |
| *** mattymo has quit IRC | 17:09 | |
| *** amotoki has quit IRC | 17:09 | |
| * mordred too - I thought that exercises were on the list-to-be-deleted | 17:09 | |
| sdague | grenade needs working exercises to sanity check at phases | 17:09 |
| sdague | which are devstack/old and devstack/new exercises | 17:09 |
| mordred | sdague: and it can't use tempest with smoke flag? | 17:09 |
| sdague | mordred: that's not it's current design | 17:09 |
| mordred | ok | 17:10 |
| sdague | it runs tempest smoke at the end | 17:10 |
| sdague | but tempest smoke is 15 minutes of tests | 17:10 |
| *** UtahDave has quit IRC | 17:10 | |
| sdague | and we are talking about something much smaller | 17:10 |
| sdague | so for neutron to have a grenade job it has to pass exercises | 17:10 |
| jeblair | ok, but that shouldn't interfere with the goal of removing exercises from (non-grenade) tempest runs, right? | 17:10 |
| sdague | jeblair: they are removed | 17:11 |
| sdague | but | 17:11 |
| sdague | they don't work for neutron | 17:11 |
| sdague | so how do you fix them | 17:11 |
| jeblair | excellent; it's hard to tell sometimes. :) | 17:11 |
| sdague | because they maybe be neutron or devstack changes to fix them (which then grenade pulls in later) | 17:11 |
| jeblair | sdague: how about add a neutron grenade job, since that's the one we want to get working? | 17:11 |
| sdague | so we need non voting jobs so yuo can test those fixes | 17:11 |
| sdague | there already is a grenade job | 17:12 |
| sdague | but that's like saying yuo test oslo with nova jobs | 17:12 |
| sdague | because grenade sources devstack | 17:12 |
| jeblair | sdague: you can add the neutron grenade job to devstack as experimental or non-voting | 17:12 |
| *** senk has quit IRC | 17:12 | |
| dtroyer_zz | what needs to be fixed is the stand-alone stable/havana devstack neutron exercise... | 17:12 |
| *** senk has joined #openstack-infra | 17:12 | |
| anteaya | neutron grenade job on offer: https://review.openstack.org/#/c/58695/ | 17:12 |
| *** amotoki has joined #openstack-infra | 17:12 | |
| fungi | one wrinkle here is that grenade wants working devstack exercises on the *previous* release stable branch, so in neutron's case they need a way to test working backports to havana to get exercises back in shape | 17:13 |
| sdague | jeblair: you have to fix on both sides, and possibly also on neutron | 17:13 |
| jeblair | sdague: i will grant that without a working forward grenade test, fixing havana devstack for neutron would be hard. | 17:14 |
| jeblair | sdague: so is that the reason a single-branch devstack exercises run is needed ^? | 17:14 |
| sdague | jeblair: right now we need it on havana & master of neutron & devstack | 17:14 |
| sdague | so that fixes can be sorted | 17:14 |
| sdague | we can drop it once it's all working | 17:15 |
| *** jcooley_ has joined #openstack-infra | 17:15 | |
| *** mattymo has joined #openstack-infra | 17:15 | |
| sdague | but that's the fastest way for the neutron team to be able to know if they are fixing everything needed for the grenade job | 17:15 |
| jeblair | okay, thanks. | 17:15 |
| *** lcheng_ has joined #openstack-infra | 17:15 | |
| * mordred understands | 17:17 | |
| *** markmcclain has quit IRC | 17:17 | |
| anteaya | jeblair: thanks | 17:18 |
| *** ruhe has quit IRC | 17:18 | |
| *** lcheng has quit IRC | 17:18 | |
| *** jooools has quit IRC | 17:21 | |
| *** BobBall is now known as BobBallAway | 17:21 | |
| *** jcooley_ has quit IRC | 17:22 | |
| * fungi needs to leave for lunch, but will return forthwith | 17:24 | |
| jeblair | clarkb, fungi, mordred: I'm going to switch logstash workers 01-08 A and B to start using crm114 | 17:26 |
| mordred | jeblair: I affirm you | 17:26 |
| *** GheRivero has quit IRC | 17:28 | |
| *** praneshp has joined #openstack-infra | 17:30 | |
| *** marun has joined #openstack-infra | 17:30 | |
| openstackgerrit | A change was merged to openstack-infra/config: Make gate-solum-devstack-dsvm voting https://review.openstack.org/62964 | 17:32 |
| *** hashar has quit IRC | 17:37 | |
| *** sarob has joined #openstack-infra | 17:37 | |
| *** SushilKM__ has left #openstack-infra | 17:39 | |
| *** rongze has quit IRC | 17:39 | |
| *** rongze has joined #openstack-infra | 17:40 | |
| *** GheRivero has joined #openstack-infra | 17:41 | |
| *** rongze has quit IRC | 17:44 | |
| *** DinaBelova has joined #openstack-infra | 17:45 | |
| *** hogepodge has joined #openstack-infra | 17:46 | |
| clarkb | morning | 17:46 |
| clarkb | jeblair: ok | 17:46 |
| *** vkozhukalov has joined #openstack-infra | 17:46 | |
| *** dolphm_ has quit IRC | 17:47 | |
| *** moted has joined #openstack-infra | 17:48 | |
| *** dolphm_ has joined #openstack-infra | 17:49 | |
| *** dims_ is now known as dims | 17:50 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Add statsd support https://review.openstack.org/63183 | 17:51 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Add statsd support https://review.openstack.org/63183 | 17:52 |
| jeblair | clarkb: ^ there we go | 17:52 |
| *** smarcet has quit IRC | 17:54 | |
| clarkb | cool | 17:56 |
| *** zhiyan has left #openstack-infra | 17:57 | |
| *** che-arne has joined #openstack-infra | 17:57 | |
| *** zhiyan has joined #openstack-infra | 17:58 | |
| *** nicedice has joined #openstack-infra | 17:59 | |
| *** mriedem has quit IRC | 17:59 | |
| *** sarob has quit IRC | 17:59 | |
| *** dkranz has joined #openstack-infra | 18:00 | |
| *** sarob has joined #openstack-infra | 18:03 | |
| *** smarcet has joined #openstack-infra | 18:04 | |
| *** senk has quit IRC | 18:04 | |
| *** harlowja_away is now known as harlowja | 18:04 | |
| *** sarob has quit IRC | 18:10 | |
| *** sarob has joined #openstack-infra | 18:11 | |
| openstackgerrit | sahid proposed a change to openstack/requirements: Tox fails to build environment because of MySQL-Python https://review.openstack.org/62028 | 18:15 |
| *** sarob has quit IRC | 18:15 | |
| *** sarob has joined #openstack-infra | 18:16 | |
| *** senk1 has joined #openstack-infra | 18:16 | |
| *** DinaBelova has quit IRC | 18:16 | |
| *** praneshp has quit IRC | 18:18 | |
| *** zhiyan has left #openstack-infra | 18:19 | |
| *** ruhe has joined #openstack-infra | 18:19 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Use \b to isolate Gerrit changeid comment links. https://review.openstack.org/63187 | 18:21 |
| *** sarob has quit IRC | 18:21 | |
| *** markmcclain has joined #openstack-infra | 18:22 | |
| *** sarob has joined #openstack-infra | 18:22 | |
| *** ladquin has joined #openstack-infra | 18:22 | |
| *** senk1 has quit IRC | 18:25 | |
| *** senk has joined #openstack-infra | 18:25 | |
| *** sarob has quit IRC | 18:26 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Create a swap partition if needed for new servers https://review.openstack.org/63190 | 18:27 |
| *** CaptTofu has joined #openstack-infra | 18:33 | |
| *** sarob has joined #openstack-infra | 18:34 | |
| *** senk1 has joined #openstack-infra | 18:37 | |
| *** rcleere has quit IRC | 18:37 | |
| *** ryanpetrello has quit IRC | 18:38 | |
| *** senk has quit IRC | 18:39 | |
| *** rongze has joined #openstack-infra | 18:40 | |
| *** senk1 has quit IRC | 18:41 | |
| *** smarcet has quit IRC | 18:42 | |
| *** rongze has quit IRC | 18:45 | |
| *** sarob has quit IRC | 18:46 | |
| *** ryanpetrello has joined #openstack-infra | 18:46 | |
| *** sarob has joined #openstack-infra | 18:46 | |
| fungi | awesome | 18:49 |
| fungi | clarkb: is https://review.openstack.org/63187 distinct from https://review.openstack.org/62523 | 18:50 |
| *** sarob has quit IRC | 18:51 | |
| *** praneshp has joined #openstack-infra | 18:51 | |
| *** lcheng__ has joined #openstack-infra | 18:53 | |
| *** sarob has joined #openstack-infra | 18:53 | |
| *** talluri has quit IRC | 18:56 | |
| *** lcheng_ has quit IRC | 18:56 | |
| *** sarob has quit IRC | 19:01 | |
| *** sarob has joined #openstack-infra | 19:01 | |
| *** AaronGr is now known as AaronGR_afk | 19:01 | |
| clarkb | fungi: no, I just didn't know the other existed | 19:02 |
| *** rnirmal has quit IRC | 19:02 | |
| clarkb | fungi: what does the anchor accomplish? is that necessary too? | 19:02 |
| *** senk has joined #openstack-infra | 19:03 | |
| fungi | clarkb: i think it's html vs link type replacement | 19:03 |
| fungi | jhesketh's should probably use link like yours did | 19:03 |
| fungi | unless there's some reason to perform additional replacement outside the existing text or outside the bounds of the hyperlink itself (or to alter more obscure link options) | 19:05 |
| *** reed has quit IRC | 19:05 | |
| *** sarob_ has joined #openstack-infra | 19:05 | |
| *** sarob has quit IRC | 19:06 | |
| *** sarob_ has quit IRC | 19:09 | |
| *** sarob has joined #openstack-infra | 19:09 | |
| *** che-arne has quit IRC | 19:10 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Add statsd support https://review.openstack.org/63183 | 19:11 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Add statsd support https://review.openstack.org/63183 | 19:12 |
| *** ruhe has quit IRC | 19:14 | |
| *** smarcet has joined #openstack-infra | 19:15 | |
| *** miqui has quit IRC | 19:18 | |
| clarkb | fungi: I don't think so, I can update jhesketh's change | 19:22 |
| fungi | clarkb: it can also currently be tested on review-dev, since i was able to successfully roll back the upgrade so it's on 2.4.4 again for the moment | 19:24 |
| clarkb | oh cool | 19:24 |
| fungi | but if we restart puppet on it, it will roll forward and break again | 19:24 |
| fungi | unless we want to land a revert of teh upgrade change (it's just a one-liner after all) | 19:24 |
| *** yolanda has joined #openstack-infra | 19:26 | |
| jeblair | since we're keeping up and not fully loaded yet, i'm going to shutdown logstash workers on 01-04 to compress load onto the remainder | 19:27 |
| clarkb | jeblair: is that a gear thing? it prefers the other nodes? | 19:27 |
| fungi | i think manually scaling/bursting those is a good way to speed up classification | 19:27 |
| fungi | any ideas whether it's possible to cross-pollinate the bayesian classification data between the workers periodically? | 19:28 |
| jeblair | i want to try to get the workers fairly close to continually busy, then evaluate their cpu usage (currently looks like < 25% utilization), then increase the number of gearman workers on each node until the node is close to fully cpu-utilized. | 19:28 |
| *** hashar has joined #openstack-infra | 19:28 | |
| fungi | or do we just want to work on the assumption that they'll all eventually approach consistency over sufficiently large bodies of data? | 19:29 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Check for word start/end around changeid searches https://review.openstack.org/62523 | 19:29 |
| *** CaptTofu has quit IRC | 19:29 | |
| clarkb | there we go | 19:29 |
| clarkb | normally I wouldn't push this through but it is annoying folks and the quiet time around holidays is perfect for gerrit restarts | 19:29 |
| jeblair | fungi: i don't know; i'm working on the second assumption for now, but would like to have a global set if possible. | 19:29 |
| jeblair | clarkb: ok, though gerrit restarts are fast and i think we can do them anytime. | 19:30 |
| * fungi keeps it in the back of his mind for now | 19:30 | |
| clarkb | jeblair: k | 19:30 |
| clarkb | going to apply that on review-dev in just a minute | 19:30 |
| jeblair | fungi: but for now, having fewer servers with more workers will reduce the number of data sets (not to mention, just be more efficient) | 19:31 |
| jeblair | fungi: so thus the somewhat manual tuning | 19:31 |
| fungi | clarkb: awesome. i tried jhesketh's previous patchset on review-dev, thus the -1 and suggestions i gave it | 19:31 |
| fungi | jeblair: agreed, completely | 19:31 |
| fungi | clarkb: there are some comments i added to the "Thingy" review on review-dev already which you can use to easily vet the adjustment against the original behavior | 19:32 |
| *** mrmartin has joined #openstack-infra | 19:33 | |
| clarkb | fungi: I think I parse that statement :) | 19:35 |
| fungi | i try to make sure all my statements do double duty as tests of your parser | 19:35 |
| *** reed has joined #openstack-infra | 19:36 | |
| *** senk has quit IRC | 19:37 | |
| clarkb | greghaynes: you can talk here :P | 19:38 |
| greghaynes | not just lurk? | 19:38 |
| greghaynes | fungi: looks like theres cssmerge | 19:38 |
| greghaynes | which might be useful for periodic data sharing | 19:39 |
| clarkb | fungi: looks like puppet still uses the new 2.8 war so I have to apply changes by hand? | 19:40 |
| * clarkb does that | 19:40 | |
| fungi | greghaynes: neat! | 19:40 |
| fungi | sounds like a possible solution | 19:41 |
| *** senk1 has joined #openstack-infra | 19:41 | |
| jeblair | or we can use afs | 19:41 |
| clarkb | restarting review-dev's gerrit now | 19:41 |
| *** gyee has joined #openstack-infra | 19:41 | |
| greghaynes | Yep, although there would probably have to be a bit if node remove/insert dance to swap which training dataset is used | 19:41 |
| fungi | jeblair: it's funny how afs solves every problem ever | 19:42 |
| jeblair | fungi: right? :) | 19:42 |
| openstackgerrit | Steve Baker proposed a change to openstack-infra/devstack-gate: Fix broken HEAT_FETCHED_TEST_IMAGE https://review.openstack.org/63202 | 19:43 |
| fungi | clarkb: what i ended up doing, for cleanliness, was to revert 7011082 on the tip of master in mt puppet dev env, cherry-pick jhesketh's change onto there and then apply from that as an alternate | 19:43 |
| fungi | er, in my puppet dev env | 19:43 |
| fungi | but for testing a one-liner patch that's definitely overlokk | 19:43 |
| clarkb | so my change doesn't fix it either. do /'s match \b? | 19:44 |
| fungi | overkill | 19:44 |
| clarkb | maybe we need jhesketh's change with link instead of html. /me tries that | 19:44 |
| *** sarob has quit IRC | 19:44 | |
| fungi | i never can remember whal characters fall into the \b class | 19:44 |
| fungi | zomg my typing. what happened to my typing? | 19:44 |
| jeblair | fungi: your characters fell | 19:45 |
| fungi | clearly | 19:45 |
| *** johnthetubaguy1 has quit IRC | 19:45 | |
| fungi | jeblair: it's funny how dvorak solves every problem ever | 19:45 |
| jeblair | fungi: afs is much easier to type on dvorak. | 19:47 |
| *** HenryG_ has quit IRC | 19:49 | |
| bknudson | I'm trying to figure out why elastic-recheck thinks a bug is the problem... | 19:50 |
| bknudson | https://review.openstack.org/#/c/62068/ | 19:50 |
| bknudson | things seem to have moved around | 19:50 |
| *** senk1 has quit IRC | 19:51 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Revert "upgrade review-dev from gerrit ver 2.4.x to ver 2.8" https://review.openstack.org/63204 | 19:52 |
| fungi | we can reclaim our gerrit sandbox for now ^ then try again once we figure out why schema migrations aren't working | 19:52 |
| clarkb | bknudson: elastic-recheck has bug fingerprint queries. if a test matches one of those then the bug for that query is blamed | 19:52 |
| clarkb | bknudson: the bug listed should have a query listed in its comments | 19:52 |
| fungi | bknudson: also note that e-r is not infallible and is only as good as the queries people have submitted (some of which might be too loose) | 19:53 |
| *** AaronGR_afk is now known as AaronGR | 19:55 | |
| fungi | bknudson: in that particular instance, a line in one of the test logs matched this pattern... http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries/1259907.yaml | 19:55 |
| *** sdake has quit IRC | 19:55 | |
| *** sdake has joined #openstack-infra | 19:56 | |
| *** sdake has quit IRC | 19:56 | |
| *** sdake has joined #openstack-infra | 19:56 | |
| clarkb | fungi: https://review-dev.openstack.org/#/c/5232/ with link and (^|[\\s.]+)(I?[0-9a-f]{8,40})($|[\\s.]+) is mostly working the match beginning or end of line isn't working | 19:56 |
| clarkb | do I need to escape ^ and $ there? | 19:56 |
| *** davidhadas has joined #openstack-infra | 19:56 | |
| bknudson | fungi: that's the weird thing... I'm not finding that line. | 19:56 |
| clarkb | hmm nope doesn't match a literal $ | 19:57 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Add statsd support https://review.openstack.org/63183 | 19:57 |
| fungi | i'm rereading https://review.openstack.org/Documentation/config-gerrit.html#_a_id_commentlink_a_section_commentlink for possible revelations | 19:59 |
| fungi | i love how their first example there is the horribly busted match we're trying to replace | 20:00 |
| bknudson | ahh, check-grenade-dsvm console.html has the line but it was gate-tempest-dsvm-neutron-large-ops that fialed. | 20:00 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Create a swap partition if needed for new servers https://review.openstack.org/63190 | 20:00 |
| clarkb | bknudson: that would be a bug | 20:01 |
| fungi | bknudson: sounds like a bug in e-r | 20:01 |
| bknudson | I'll file it. | 20:01 |
| clarkb | fungi: well, the horribly busted isn't really a problem for change id's because those don't end up in urls very often | 20:01 |
| fungi | possibly two bugs, since that line appearing in non-failure results means it's not a great indicator of why a job fails | 20:01 |
| clarkb | commit shas however are a popular thing in urls | 20:01 |
| fungi | clarkb: oh, right, their example is missing the ? | 20:02 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Add statsd support https://review.openstack.org/63183 | 20:03 |
| fungi | clarkb: so the fine manual says it's javascript regexp. i'm so far not finding great references which expound on exactly which characters are matched by javascript's word-boundry class | 20:06 |
| fungi | er, boundary | 20:07 |
| clarkb | fungi: \b doesn't work the way we want with the python re module | 20:07 |
| clarkb | and appeared to not work with the GWT javascript madness so I am giving up on \b | 20:07 |
| dhellmann | hey, guys, I'd like to bounce some ideas off of you related to translations. anyone have time for that? | 20:07 |
| *** yolanda has quit IRC | 20:08 | |
| clarkb | may need to use something like [^\\a-zA-z0-9andIhatethisalready] | 20:08 |
| clarkb | dhellmann: sure | 20:08 |
| dhellmann | clarkb: thanks! | 20:08 |
| *** sarob has joined #openstack-infra | 20:08 | |
| dhellmann | I'm finally starting on https://blueprints.launchpad.net/oslo/+spec/log-messages-translation-domain | 20:08 |
| *** thomasem has quit IRC | 20:08 | |
| dhellmann | tl;dr - extract messages for different log levels into separate catalogs so they can be translated with different priorities | 20:09 |
| dhellmann | I haven't used transifex before, so I'm not sure how (if?) that would work with transifex and our existing jobs | 20:09 |
| dhellmann | I found some scripts that pull changes in from transifex and submit change requests | 20:10 |
| dhellmann | I *think* those might need to be updated to account for multiple pot files | 20:10 |
| clarkb | dhellmann: transifex can do it, iirc they are different "resources" | 20:10 |
| clarkb | the scripts would need to be updated for different resources | 20:10 |
| jeblair | one thing to keep in mind is that we plan on moving off of transifex, so we should keep new development there to a minimum to make moving easier | 20:10 |
| clarkb | probably to iterate over each one doing the same steps and skipping if the resource doesn't exist for that particular project | 20:11 |
| dhellmann | jeblair: ok, that's good to know | 20:11 |
| *** hashar has quit IRC | 20:11 | |
| dhellmann | clarkb: does transifex create the .pot files now? or are we doing that, and uploading them somewhere I haven't found? | 20:11 |
| fungi | clarkb: i think part of the \b problem is that it's a zero-width match, and so can match the null string as well | 20:11 |
| clarkb | transifex does not create the .pot files. The scripts do | 20:11 |
| clarkb | dhellmann: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/upstream_translation_update.sh#n32 | 20:13 |
| dhellmann | ok, cool | 20:13 |
| clarkb | dhellmann: so I think we put the bulk of that script in a loop that iterates over the known classes of things | 20:14 |
| dhellmann | I found that script, but was reading it as downloading changes from transifex, not the other way around (the "tx pull") | 20:14 |
| *** Abhishek has joined #openstack-infra | 20:14 | |
| dhellmann | ok, that makes sense | 20:14 |
| clarkb | dhellmann: would need to update the one that pushes to gerrit as well so that we push all of the possible .pot and translations | 20:16 |
| bknudson | Stderr: '/bin/sh: 1: collie: not found\n' -- collie? | 20:16 |
| bknudson | Store sheepdog could not be configured correctly. ?? | 20:16 |
| clarkb | fungi: I switched it to ^(stuff)$ just to go simple and see if that works and it does not, I have no clue what is going on there | 20:17 |
| dhellmann | clarkb: why is upstream_translation_update.sh adding the locale files to git on line 40? | 20:17 |
| dhellmann | just to see if there are changes? | 20:17 |
| clarkb | dhellmann: yes so that the check on line 42 works | 20:17 |
| bknudson | oh, I guess those are normal | 20:17 |
| dhellmann | clarkb: it looks like propose_translation_update.sh is already using find and wildcards, so it would find all of the generated files | 20:18 |
| fungi | clarkb: i'll try out a few alternatives. i suspect line endings aren't treated normally. could be some sort of multi-line pattern space thing | 20:18 |
| clarkb | fungi: I wonder if the text being matched is being passed in with extra html | 20:18 |
| *** CaptTofu has joined #openstack-infra | 20:18 | |
| fungi | also possible | 20:19 |
| dhellmann | clarkb: it's interesting that propose_translation_update.sh also pushes changes to transifex -- keeping them in sync? | 20:19 |
| fungi | clarkb: if you run out of things to try, let me know and i'll take a stab (just don't want to step on your toes) | 20:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: zuul status: allow changes to be collapsed https://review.openstack.org/63018 | 20:20 |
| clarkb | dhellmann: yes, at one time we were going to only let them go in one direction but disabling fuzzy matches fixed the problems we were having instead | 20:20 |
| clarkb | dhellmann: so you can update in git directly and have that show up in transifex | 20:20 |
| dhellmann | clarkb: ah, ok | 20:21 |
| clarkb | fungi: ok, I am checking a few extra things but then will probably give up for lunch | 20:21 |
| fungi | alright | 20:21 |
| dhellmann | clarkb: ok, thanks for this, it was really helpful. I may come back again with more questions. | 20:21 |
| clarkb | dhellmann: np | 20:22 |
| clarkb | fungi: yup there are <p> tags in there | 20:23 |
| * clarkb tries a thing | 20:23 | |
| fungi | might actually be to our advantage, confusing as it is | 20:23 |
| fungi | at least there's something specific to match on (tag brackets) | 20:23 |
| *** AaronGR is now known as AaronGR_lunch | 20:27 | |
| clarkb | fungi: (<p>|[\\s.]+)(I?[0-9a-f]{8,40})(</p>|[\\s.]+) seems to be working | 20:27 |
| clarkb | want to take a look? I am going to find lunch but will update the change with ^ if that looks good | 20:27 |
| fungi | wow! so unnecessarily complex. thanks, gerrit! | 20:28 |
| stevebaker | jeblair, clarkb, I messed up. Could you take a look at https://review.openstack.org/63202 ? | 20:28 |
| fungi | clarkb: i think i'd also like to expand that to 7,40 since short shas are usually written as 7-character abbreviations | 20:29 |
| clarkb | stevebaker: done | 20:29 |
| clarkb | fungi: wfm | 20:29 |
| jeblair | stevebaker: and approved | 20:30 |
| fungi | stevebaker: lgtm. approved | 20:30 |
| fungi | heh | 20:30 |
| jeblair | double approved! | 20:30 |
| stevebaker | \o/ thanks | 20:30 |
| fungi | double-tap to the head | 20:30 |
| *** hashar has joined #openstack-infra | 20:30 | |
| fungi | stevebaker: once that merges you'll probably want to wait about 30 minutes before testing it out | 20:31 |
| stevebaker | fungi: ok, will do | 20:31 |
| jeblair | fungi: why the wait? | 20:33 |
| fungi | jeblair: stevebaker: oh, nevermind. somehow i had it in my head it was a job configuration change | 20:34 |
| fungi | stevebaker: should be usable as soon as it merges | 20:34 |
| sdague | bknudson: yes, glance barfs on itself on startup all the time. They are working on fixing that | 20:34 |
| bknudson | sdague: I think it's swift this time... http://logs.openstack.org/68/62068/5/check/gate-tempest-dsvm-neutron-large-ops/2946b85/logs/screen-s-proxy.txt.gz | 20:35 |
| bknudson | took me a while to track it down. | 20:35 |
| sdague | bknudson: interesting | 20:35 |
| sdague | yep, well that would definitely cause glance issues | 20:36 |
| stevebaker | fungi: so there is no devstack image build thingy to wait for? | 20:36 |
| notmyname | so swift couldn't connect to /dev/log in that case. ie it couldn't connect to syslog. that sounds like an underlying issue with the vm (overloaded or otherwise misconfigured) | 20:37 |
| jeblair | fungi, clarkb: i believe all puppet runs are broken with http://paste.openstack.org/show/55547/ | 20:37 |
| fungi | stevebaker: nope--gets checked out and updated when the job starts | 20:37 |
| *** thingee has quit IRC | 20:37 | |
| clarkb | jeblair: apache problems on the master? | 20:37 |
| fungi | jeblair: saw that a few weeks ago, restart of apache on the puppetmaster fixed it. couldn't find an explanation | 20:38 |
| mordred | jeblair, fungi: blerg | 20:38 |
| fungi | mordred: if you say so | 20:38 |
| clarkb | fungi: cool going to grab lunch if we have a known work around. back in a bit | 20:38 |
| clarkb | also blarg | 20:38 |
| fungi | i will have a look at the apache error logs again | 20:39 |
| jeblair | fungi: k; i don't see anything diagnosable, and am leaning toward restarting now and setting maxrequestsperchild to $lowvalue | 20:40 |
| stevebaker | Is there a whitelist of allowed ERROR log messages that stop tempest failing? Just in case https://bugs.launchpad.net/heat/+bug/1261433 takes a while to be fixed | 20:40 |
| uvirtbot | Launchpad bug 1261433 in heat "heat-tempest-xx already has an action (DELETE) in progress" [High,Triaged] | 20:40 |
| fungi | lots of "...visitor's connection has stalled and couldn't receive the data that Apache is sending..." | 20:40 |
| fungi | all nice and vague | 20:41 |
| fungi | yeah, let's restart apache for now and i'll keep whittling down the error logs looking for a real nugget of fail | 20:42 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Preserve testr temp files https://review.openstack.org/62786 | 20:42 |
| *** rcleere has joined #openstack-infra | 20:43 | |
| *** nicedice has quit IRC | 20:43 | |
| *** mrmartin has quit IRC | 20:44 | |
| *** nicedice has joined #openstack-infra | 20:44 | |
| fungi | maybe a bug in libapache2-mod-passenger 2.2.11debian-2 | 20:45 |
| sdague | stevebaker: it's in the tempest tree | 20:45 |
| * jeblair -> lunch | 20:46 | |
| sdague | etc/whitelist.yaml | 20:46 |
| stevebaker | sdague: ok, I'll take a look | 20:46 |
| *** boris-42 has quit IRC | 20:47 | |
| fungi | [Thu Dec 19 16:25:14 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting | 20:48 |
| *** DinaBelova has joined #openstack-infra | 20:48 | |
| fungi | but that may be a symptom of all the clients hanging waiting for data for way longer | 20:48 |
| *** boris-42 has joined #openstack-infra | 20:50 | |
| *** vkozhukalov has quit IRC | 20:51 | |
| *** melwitt has joined #openstack-infra | 20:51 | |
| *** boris-42 has quit IRC | 20:52 | |
| *** DinaBelova has quit IRC | 20:52 | |
| *** senk1 has joined #openstack-infra | 20:53 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: zuul status: Collapse changes by default https://review.openstack.org/63019 | 20:53 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: zuul status: allow changes to be collapsed https://review.openstack.org/63018 | 20:53 |
| *** mrodden has joined #openstack-infra | 20:54 | |
| *** senk1 has quit IRC | 20:56 | |
| fungi | looks like the first time we saw this i had narrowed it down to starting around 05:30 utc on the 11th, so just over a week ago | 20:58 |
| bknudson | is there a way to clear bugs out of http://status.openstack.org/rechecks/ ? Some of these are obviously mistakes by the rechecker | 21:00 |
| bknudson | or do they just time out after a while? | 21:00 |
| fungi | bknudson: they get downgraded over time as new bugs become "popular" | 21:01 |
| fungi | "frecency" is the term, i believe (it was a new one to me) | 21:01 |
| fungi | some blending of frequency and recentness | 21:02 |
| *** senk1 has joined #openstack-infra | 21:02 | |
| *** thingee has joined #openstack-infra | 21:02 | |
| bknudson | ok, not sure if it would be useful to be able to exclude bugs somehow... it takes a while to look through the list as is. | 21:03 |
| bknudson | maybe a no-recheck tag in the bug | 21:04 |
| *** michchap_ has joined #openstack-infra | 21:04 | |
| *** smarcet has quit IRC | 21:04 | |
| *** CaptTofu has quit IRC | 21:05 | |
| bknudson | e.g., https://bugs.launchpad.net/keystone/+bug/1259426 is obviously not going to cause a recheck since devstack-gate doesn't use ldap. | 21:06 |
| uvirtbot | Launchpad bug 1259426 in keystone "LDAP list group users crash when no member in the group" [Medium,In progress] | 21:06 |
| fungi | bknudson: i think the plan is to extend http://status.openstack.org/elastic-recheck/ to include any remaining uses of the older rechecks page and then retire that system | 21:06 |
| *** michchap has quit IRC | 21:06 | |
| *** AaronGR_lunch is now known as AaronGR | 21:06 | |
| fungi | so that's where development effort is being spent | 21:06 |
| bknudson | that page is much fancier. | 21:07 |
| *** senk1 has quit IRC | 21:07 | |
| fungi | bknudson: and yeah, looks like that 1259907 you were asking about is matching a ton of pass jobs according to its graph on the e-r page | 21:13 |
| *** dolphm_ has quit IRC | 21:13 | |
| fungi | for the past several days | 21:13 |
| *** sarob has quit IRC | 21:13 | |
| fungi | the query's been there for a week according to git, but only just began matching successful jobs a few days ago | 21:14 |
| *** senk has joined #openstack-infra | 21:14 | |
| *** vipul is now known as vipul-away | 21:16 | |
| bknudson | fungi: a comment indicates that sdague changed grenade to not fail for dirtly logs. | 21:17 |
| bknudson | on 12-14 | 21:17 |
| sdague | yes, I did | 21:18 |
| sdague | fungi: I was torn on removing that query, because there were fails for it | 21:18 |
| sdague | hey, so do we have contact users for all the 3rd party CI systems? | 21:20 |
| *** kashyap has left #openstack-infra | 21:20 | |
| sdague | because the DB2 one is useless, and I'd honestly like to revoke their creds | 21:20 |
| sdague | https://review.openstack.org/#/c/62863/ | 21:21 |
| *** vipul-away is now known as vipul | 21:21 | |
| sdague | they are posting +1s on patches, with a comment that their tests actually failed, non voting, and a link which isn't valid | 21:21 |
| *** davidhadas_ has joined #openstack-infra | 21:21 | |
| *** yassine has quit IRC | 21:24 | |
| *** smarcet has joined #openstack-infra | 21:24 | |
| *** davidhadas has quit IRC | 21:24 | |
| fungi | sdague: we have contact info for them, but yes the db2 one was requested by a team in cn who have not been real clear on how this works... i've been suspecting language barrier related problems, but maybe it runs deeper than that | 21:25 |
| jeblair | also, you can find out the email by typing the name into an add reviewer box; IBM DB2 Test <scecd@cn.ibm.com> | 21:26 |
| *** yassine has joined #openstack-infra | 21:26 | |
| jeblair | (could also probably get it with a review ssh query | 21:26 |
| fungi | or the tooltip when you hover over the name on their comments | 21:26 |
| *** yassine has quit IRC | 21:26 | |
| fungi | (though agreed it's a pain to cut/paste out of a tooltip) | 21:26 |
| jeblair | fungi: pretty much any ui paradigm other than what you would expect for "find this user's email address". | 21:27 |
| fungi | yup | 21:27 |
| sdague | yeh, well, if a 3rd party system has screwed up so much that it is +1ing patches without passing any jobs, I want to just shut it off | 21:27 |
| jeblair | fungi: do we have an ongoing thread with them or should we start a new one? | 21:27 |
| *** yassine has joined #openstack-infra | 21:28 | |
| sdague | because right now, it's completely spam | 21:28 |
| fungi | jeblair: there is an ongoing thread... krtaylor may just be able to pipe up here about it though | 21:28 |
| *** smarcet has quit IRC | 21:28 | |
| fungi | since he was the last one to reply on the thread | 21:28 |
| fungi | ArxCruz also seems to have some insight into it | 21:28 |
| fungi | i also love that the comment they're adding on reviews has a completely bogus hyperlink to our logserver | 21:30 |
| jeblair | fungi: did you restart apache? | 21:30 |
| fungi | jeblair: i did--is it still broken? | 21:30 |
| jeblair | nope, just checking | 21:30 |
| *** dtroyer_zz has quit IRC | 21:30 | |
| fungi | jeblair: yeah, i can't find a correlation, though the graphs suggest it ceased working around 16:30 which is right around when we got the maxclientss error in the log | 21:31 |
| fungi | but again, that's probably a symptom, so doesn't help point to the cause | 21:31 |
| jeblair | fungi: ah neat; i wonder if something's sticking around. we might want to enable mod_status and check that out next time. | 21:31 |
| * krtaylor reading scrollback | 21:31 | |
| fungi | krtaylor: the comment from IBM DB2 Test | 21:32 |
| fungi | ...on https://review.openstack.org/62863 pretty much says it all | 21:32 |
| krtaylor | wow, yeah, I wish I had anything to do with that team, I am trying to do what I can though | 21:33 |
| *** dtroyer has joined #openstack-infra | 21:33 | |
| fungi | krtaylor: we appreciate it. it's hard to get them to even reply to e-mail messages, so i end up pestering other people at ibm. sorry about that btw | 21:33 |
| krtaylor | fungi, no problem at all, I think there may be some language barrier issues, but mostly then just need some coaching | 21:34 |
| mtreinish | sdague: +1, I say pull their account | 21:35 |
| fungi | in the short term i can yank the ssh key off of the account, if it's such a nuisance we can't wait for cn daytime to get their attention | 21:36 |
| sdague | fungi: so I'm going to assume we won't get a timely resolution, so I'd really like to put that account in a penalty box | 21:36 |
| krtaylor | actually, I am kinda surprised that it is up and running, well, I use that term loosely | 21:36 |
| sdague | krtaylor: yeh, I think the problem is they probably don't even know they are posting | 21:36 |
| sdague | given the links are all invalid | 21:36 |
| jeblair | fungi: you might try just removing them from the group first | 21:37 |
| krtaylor | yeah, completely possible | 21:37 |
| fungi | jeblair: oh, agreed, that's easier | 21:37 |
| *** yfried1 has joined #openstack-infra | 21:37 | |
| jeblair | fungi: that will probably cause them not to post anything; if i'm wrong about that, then the ssh key. | 21:37 |
| sdague | fungi: seems like we should probably establish some standards about what useful 3rd party CI looks like | 21:37 |
| *** cody-somerville has joined #openstack-infra | 21:37 | |
| *** cody-somerville has quit IRC | 21:37 | |
| *** cody-somerville has joined #openstack-infra | 21:37 | |
| jeblair | sdague: we have them; this does not meet them. | 21:37 |
| sdague | ok :) | 21:37 |
| sdague | then I don't feel bad saying to block them | 21:37 |
| mtreinish | jeblair: heh | 21:38 |
| jeblair | sdague: it is the things you'd expect -- links to useful public logs being the main thing. | 21:38 |
| fungi | krtaylor: this is also not the first time. they originally had it posting false test result messages using shaomei's contributor account (and just not posting verify votes) so it's quite likely they turned that back on with the new creds i issued today | 21:38 |
| krtaylor | there is actually some good documentation on what is needed, per group B | 21:38 |
| krtaylor | fungi, yes, I know, and as an IBMer I am sorry about that, I wish they had asked around first | 21:39 |
| * krtaylor starts writing another internal email... | 21:39 | |
| *** jhesketh_ has joined #openstack-infra | 21:40 | |
| * clarkb is back from lunch | 21:40 | |
| fungi | krtaylor: oh, i wasn't meaning it in a bad way, just suggesting it's not surprising it was up and running since they apparently already had it up and running weeks ago when it happened the first time | 21:40 |
| *** dizquierdo has joined #openstack-infra | 21:40 | |
| *** jhesketh_ has quit IRC | 21:40 | |
| clarkb | fungi: if I s/8/7/ does that regex work for you? | 21:40 |
| fungi | clarkb: yep, i'm good with it | 21:40 |
| *** jhesketh has joined #openstack-infra | 21:40 | |
| *** lcheng__ has quit IRC | 21:41 | |
| fungi | sdague: mtreinish: krtaylor: jeblair: anyway, i've yanked that account out of the external testing tools group so hopefully it silences them for now | 21:41 |
| jeblair | fungi: i think puppetmaster is still not happy. | 21:41 |
| fungi | grar | 21:41 |
| mtreinish | fungi: well they did turn it off after the figured out it was spamming reviews | 21:41 |
| *** yassine has quit IRC | 21:42 | |
| fungi | mtreinish: yep, once fifield got daisy to get their attention, i think it was | 21:42 |
| mtreinish | fungi: haha, I didn't realize that. I assumed they were just watching things. | 21:43 |
| fungi | jeblair: so one other thing i restarted last time was the puppet agent on the master, though that doesn't seem like it would have any bearing on this | 21:43 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Check for whitespace around changeid searches https://review.openstack.org/62523 | 21:43 |
| * fungi tries anyway | 21:43 | |
| jeblair | fungi: ++ | 21:43 |
| krtaylor | fungi, thanks, I'll explain what happened to them | 21:43 |
| ArxCruz | fungi: krtaylor I will try to help them :) | 21:44 |
| *** slong has quit IRC | 21:44 | |
| fungi | jeblair: any better or still timing out for you? | 21:44 |
| fungi | krtaylor: ArxCruz: much appreciated | 21:44 |
| clarkb | fungi: jeblair: I can take a look at puppet master weirdness in a minute if you need extra eyes | 21:44 |
| fungi | clarkb: definitely anything you can find which i missed would be greatly appreciated | 21:45 |
| clarkb | k | 21:45 |
| jeblair | fungi: doesn't seem better | 21:45 |
| *** senk has quit IRC | 21:45 | |
| fungi | last time this happened, i think we had puppet people remarking that the passenger puppetmaster was fairly broken and probably not a great choice | 21:45 |
| *** SergeyLukjanov has quit IRC | 21:46 | |
| jeblair | it wasn't broken for the past 2.5 years. | 21:46 |
| fungi | yeah, no idea why it started a week or so ago | 21:46 |
| clarkb | there are tracebacks in the apache error.log. passenger/apache seem to want to blame the "visitor" (client) | 21:46 |
| clarkb | did a new version of puppet 2.7 just release? | 21:46 |
| sdague | fungi: thanks | 21:47 |
| jeblair | so a quick test run on nodepool worked right after the restart | 21:47 |
| *** crank has quit IRC | 21:48 | |
| fungi | clarkb: grepping the dpkg logs, last updates i see for puppet were in june | 21:49 |
| jeblair | did we change something relating to dashboard reports? | 21:49 |
| jeblair | it seems like the time after the run is finished when it's uploading a report is quite long | 21:50 |
| *** davidhadas_ is now known as davidhadas | 21:51 | |
| jeblair | reports=store, http | 21:51 |
| jeblair | reporturl=http://puppet-dashboard.openstack.org:3000/reports/upload | 21:51 |
| fungi | the conffile containing the reporturl hasn't changed in months | 21:51 |
| jeblair | so, um, if that host is broken, maybe we shouldn't have those lines? | 21:51 |
| clarkb | jeblair: probably not, but it is also reporting to the master | 21:52 |
| clarkb | that is what 'store' means iirc | 21:52 |
| jeblair | clarkb: i think the master forwards it to the url | 21:52 |
| fungi | we did/do have a /etc/hosts entry directing that to anteaya's dev puppet dashboard at 15.185.153.222 which seems to be down (no idea how long that's been down though)\ | 21:52 |
| clarkb | see /var/lib/puppet/reports on the master | 21:52 |
| jeblair | (after storing it) | 21:52 |
| clarkb | jeblair: oh maybe | 21:52 |
| jeblair | fungi: so, to prevent confusion in the future, maybe we should just change the config file and not etc/hosts? | 21:53 |
| *** senk1 has joined #openstack-infra | 21:53 | |
| fungi | jeblair: agreed. we changed /etc/hosts originally to keep puppet agent from undoing that since puppet agent manages the config for the puppet master (which is maybe slightly ouroboros) | 21:54 |
| jeblair | fungi: oh, yeah, but i mean we could have just changed the file in the config repo that i'm patching. | 21:54 |
| clarkb | so I think we undo the /etc/hosts entry then remove the reporturl and reports http settings from puppet.conf? | 21:55 |
| clarkb | jeblair: ya that | 21:55 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Don't report to puppet-dashboard https://review.openstack.org/63220 | 21:55 |
| fungi | jeblair: right. hosts entry is all gone as of seconds ago | 21:55 |
| *** senk1 has quit IRC | 21:55 | |
| *** senk has joined #openstack-infra | 21:55 | |
| clarkb | zaro: are you around today? want to make sure the scp plugin stuff isn't blocking on anything | 21:55 |
| *** lcestari has quit IRC | 21:55 | |
| jeblair | i timed a puppet run; it took 5 seconds for the run and another 70 seconds till it finished | 21:56 |
| fungi | ick. i wonder why the difference in behavior. normally they go fairly quickly for me | 21:56 |
| jeblair | hypothesis: N hosts * 70 seconds of waiting for puppet-dashboard timeout means a lot of occupied apache slots | 21:57 |
| fungi | sounds probable | 21:57 |
| jeblair | which leades to puppet runs timing out | 21:57 |
| fungi | perhaps that host was up (or at least something was there to reject connections rather than time out as unreachable) until a couple weeks ago | 21:57 |
| *** sarob has joined #openstack-infra | 22:00 | |
| *** slong has joined #openstack-infra | 22:04 | |
| *** HenryG has joined #openstack-infra | 22:07 | |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Add dryrun flag to welcome_message.py https://review.openstack.org/61901 | 22:09 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: zuul status: Collapse changes by default https://review.openstack.org/63019 | 22:09 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: zuul status: allow changes to be collapsed https://review.openstack.org/63018 | 22:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't report to puppet-dashboard https://review.openstack.org/63220 | 22:09 |
| *** julim has quit IRC | 22:09 | |
| fungi | did we merge something which broke devstack-gate or the requirements integration job? http://logs.openstack.org/53/60553/2/gate/gate-requirements-integration-dsvm/3dc4ce6/console.html#_2013-12-19_21_31_25_637 | 22:11 |
| fungi | last successful run i see for that job was roughly 48 hours ago | 22:12 |
| zaro | clarkb: yes. had a late start to my day. took a break to do some reviews. still working on the scp thing as well. | 22:13 |
| *** senk has quit IRC | 22:13 | |
| jeblair | fungi: quite possibly, i'll look into it | 22:13 |
| fungi | "bash: /devstack-vm-gate.sh: No such file or directory" | 22:13 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: zuul status: Collapse changes by default https://review.openstack.org/63019 | 22:13 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: zuul status: allow changes to be collapsed https://review.openstack.org/63018 | 22:13 |
| *** sarob has quit IRC | 22:13 | |
| *** sarob has joined #openstack-infra | 22:14 | |
| *** Hefeweizen has quit IRC | 22:14 | |
| *** Hefeweizen has joined #openstack-infra | 22:14 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Update gate script path in requirements job https://review.openstack.org/63224 | 22:15 |
| jeblair | fungi: it seems hazardous to refer to devstack-gate variables without a defined API | 22:16 |
| *** sarob has quit IRC | 22:17 | |
| fungi | jeblair: yay shell variables and their empty defaults | 22:17 |
| *** sarob has joined #openstack-infra | 22:17 | |
| *** yidclare has joined #openstack-infra | 22:18 | |
| *** hashar has quit IRC | 22:18 | |
| *** jcooley_ has joined #openstack-infra | 22:19 | |
| clarkb | zaro: anything I can do to help with the scp thing? | 22:19 |
| fungi | looks like that was the only builder to use $GATE_SCRIPT_DIR at least | 22:20 |
| *** rcleere has quit IRC | 22:20 | |
| fungi | but any of the d-g hooks probably need a carefully managed set of variables which are in-scope for them | 22:20 |
| jeblair | fungi: yep | 22:21 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add graph for gate hit count https://review.openstack.org/61429 | 22:21 |
| *** hogepodge has quit IRC | 22:21 | |
| *** jhesketh__ has joined #openstack-infra | 22:24 | |
| jeblair | releasestatus is broken | 22:26 |
| jhesketh__ | Morning | 22:26 |
| zaro | clarkb: so i'm seeing that an exception occurs on writer.flush(). the pipe seems to get disconnected when that happens and not everything is sent over. | 22:27 |
| clarkb | jhesketh__: hello there | 22:27 |
| zaro | clarkb: the code seems to expect that to happen but when it happened before the pipe probably didn't get disconnected. | 22:28 |
| jeblair | ttx: https://bugs.launchpad.net/openstack-ci/+bug/1262891 | 22:28 |
| uvirtbot | Launchpad bug 1262891 in openstack-ci "releasestatus throws ValueError exception" [Medium,Triaged] | 22:28 |
| bknudson | check-tempest-dsvm-postgres-full just plain takes longer than 90 mins | 22:28 |
| *** eharney has quit IRC | 22:29 | |
| *** esker has quit IRC | 22:29 | |
| clarkb | bknudson: the timeouts seem to happen when there are specific failures | 22:29 |
| clarkb | which I thought jgriffith has sorted out | 22:29 |
| clarkb | jeblair: https://review.openstack.org/#/c/62116/1/modules/jenkins/files/slave_scripts/merge_tags.sh --no-edit is not in my git 1.7.9 man page. Appears in the 1.7.11 man page on git-scm.com | 22:30 |
| clarkb | jeblair: any idea if that option is a thing on precise git? | 22:30 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Replace DEVSTACK_GATE_ZEROMQ for DEVSTACK_GATE_MQ_DRIVER https://review.openstack.org/59993 | 22:30 |
| bknudson | clarkb: I've seen that one but this time everything is 'ok' | 22:31 |
| jeblair | clarkb: i'm running precise and it is not in my 1.7.9.5 man page | 22:31 |
| clarkb | jeblair: I am doing a test locally | 22:32 |
| clarkb | bknudson: have an example? | 22:32 |
| jd__ | all changes to requirements seem to fail, did you see that? http://logs.openstack.org/53/60553/2/gate/gate-requirements-integration-dsvm/3dc4ce6/console.html | 22:32 |
| jeblair | jd__: yes, fix here: https://review.openstack.org/#/c/63224/ | 22:32 |
| fungi | jd__: https://review.openstack.org/#/c/63224/1 | 22:32 |
| bknudson | clarkb: http://logs.openstack.org/68/62068/5/check/check-tempest-dsvm-postgres-full/1c1f5e3/console.html | 22:32 |
| clarkb | jeblair: well git didn't barf when I gave it no-edit locally when doing a fast forward merge. maybe just undocumented? | 22:32 |
| bknudson | for https://review.openstack.org/#/c/62068/ | 22:32 |
| jd__ | jeblair: fungi: thanks! :) | 22:33 |
| bknudson | + setup_host took 10 mins -- that doesn't help | 22:33 |
| jd__ | is there a Jenkins plugin that's used to spawn the VM to run the tests on infra btw? | 22:34 |
| clarkb | bknudson: so that ran on our slowest cloud nodes. we are probably too close to the edge there | 22:34 |
| jeblair | clarkb: it's mentioned in passing in the release notes of 1.7.9.1, so maybe undocumented for merge | 22:34 |
| *** mfer has quit IRC | 22:35 | |
| clarkb | jeblair: ok. seemsto work. I will approve that change | 22:35 |
| bknudson | full develop-inst: /opt/stack/new/tempest -- takes 3.5 mins | 22:35 |
| jeblair | clarkb: in fact, it's mentioned as explicitly being added for merge in the 1.7.9.1 changelog. | 22:35 |
| clarkb | fungi: care to weigh in on https://review.openstack.org/#/c/62122/1 it may address the problem bknudson has pointed out | 22:36 |
| clarkb | fungi: and if it doesn't address it I think we remove new hpcloud from nodepool | 22:36 |
| jeblair | clarkb, fungi: status.o.o is up and running and is working as well as the production vhost according to my local /etc/hosts test | 22:36 |
| jeblair | clarkb, fungi: i'm ready to change dns; do you want to test anything first? | 22:37 |
| bknudson | clarkb: thanks! | 22:37 |
| clarkb | jeblair: I don't have anything specific | 22:37 |
| clarkb | jeblair: I say go for it | 22:37 |
| *** yamahata has quit IRC | 22:37 | |
| fungi | jeblair: what is the ip address? i'll give it a quick once-over | 22:37 |
| *** dstanek has quit IRC | 22:38 | |
| *** nati_ueno has joined #openstack-infra | 22:38 | |
| *** dstanek has joined #openstack-infra | 22:39 | |
| fungi | jeblair: nevermind--got it from nova list | 22:40 |
| fungi | (23.253.71.218 or 2001:4800:7815:0103:3bc3:d7f6:ff04:4603) | 22:40 |
| *** afazekas has quit IRC | 22:40 | |
| jeblair | i copied the last index.html from releasestatus over so it won't look broken | 22:42 |
| jeblair | (it's about 1 day old) | 22:42 |
| *** dangers is now known as danger_fo_away | 22:42 | |
| fungi | jeblair: lgtm | 22:43 |
| fungi | i say go for it | 22:43 |
| *** crank has joined #openstack-infra | 22:44 | |
| clarkb | mtreinish: sdague: (and others) re https://review.openstack.org/#/c/58605/3 is that the sort of thing we should just merge and deal with fallout as it occurs or do we need to cleanup current gate failures before potentially making it much worse? | 22:44 |
| sdague | clarkb: I'd hold until the rest of things got under control | 22:46 |
| clarkb | zaro: are you able to catch and log that exception/step through it with the debugger to see why it is happening? | 22:46 |
| clarkb | sdague: ok | 22:46 |
| *** dizquierdo has quit IRC | 22:48 | |
| jeblair | clarkb, sdague: someone should -1 that then | 22:49 |
| sdague | done | 22:49 |
| sdague | does anyone know what signal screen sends to it's processes on -X quit? | 22:50 |
| jeblair | status.o.o dns record has been changed | 22:50 |
| jeblair | i think the previous ttl was 3600, so we have an hour of transition | 22:51 |
| *** dkranz has quit IRC | 22:53 | |
| jeblair | mordred: not urgent: can you look at https://review.openstack.org/#/c/61498/ when you have a moment? | 22:53 |
| *** jooools has joined #openstack-infra | 22:53 | |
| zaro | clarkb: just about got my debugger setup. had to redo it with this my new laptop. | 22:54 |
| zaro | clarkb: the code seems to indicate that an exception was expected to happen but just ignored and the data still got written to external location. was this the case? | 22:55 |
| *** Ryan_Lane has joined #openstack-infra | 22:55 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Check for whitespace around changeid searches https://review.openstack.org/62523 | 22:56 |
| *** vipul is now known as vipul-away | 22:56 | |
| jhesketh__ | clarkb: would you mind taking another look at the changeid regex please? https://review.openstack.org/#/c/62523 | 22:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove status.o.o vhost from static.o.o https://review.openstack.org/63234 | 22:57 |
| clarkb | zaro: the InterruptedException happens if that thread is interrupted while sleeping | 22:57 |
| clarkb | zaro: we just ignore it and try to do work again | 22:57 |
| clarkb | jhesketh__: sure, why use html over link? | 22:58 |
| clarkb | oh hmm I see the comment now | 22:58 |
| zaro | clarkb: here's the exception http://paste.openstack.org/show/55561/ | 22:58 |
| jhesketh__ | clarkb: because if you match <p>1231231 some text</p> the open <p> tag will disappear | 22:58 |
| clarkb | zaro: so the connection is going away | 22:59 |
| jhesketh__ | using the html replacement lets us put back in the boundary match | 22:59 |
| clarkb | jhesketh__: gotcha | 22:59 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Fix broken HEAT_FETCHED_TEST_IMAGE https://review.openstack.org/63202 | 22:59 |
| *** dcramer_ has quit IRC | 23:00 | |
| clarkb | jhesketh__: give me a minute I will put that on review-dev so we can double check it but I like patchset 4 | 23:00 |
| *** flaper87 is now known as flaper87|afk | 23:00 | |
| jhesketh__ | clarkb: no rush, thanks :-) | 23:00 |
| zaro | clarkb: i'm referring to IOException after writer.close() | 23:00 |
| zaro | clarkb: yes, i believe so. | 23:01 |
| clarkb | zaro: ya the writer is flushing down the scp pipe and the pipe is dead for some reason | 23:01 |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/pbr: package: read a specific Python version requirement file https://review.openstack.org/63236 | 23:02 |
| clarkb | jhesketh__: https://review-dev.openstack.org/#/c/5232/ lgtm | 23:03 |
| openstackgerrit | A change was merged to openstack-infra/config: use parameters from project.yaml https://review.openstack.org/62113 | 23:04 |
| fungi | jhesketh__: i don't think it does lose the whitespace if you use link instead of html... at least our tests didn't seem to https://review-dev.openstack.org/#/c/5232/ | 23:05 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Supply a timeout value to devstack-gate https://review.openstack.org/63238 | 23:05 |
| jhesketh__ | fungi: it depends what it matches on | 23:05 |
| jhesketh__ | but if it matches on the <p> tag that would disappear | 23:06 |
| jhesketh__ | there's a good chance it still renders fine though | 23:06 |
| fungi | ahh, yes okay, i see that | 23:06 |
| zaro | clarkb: it seems like the code is working the same as before, except the pip didn't die on that exception. am i right there? | 23:06 |
| *** thedodd has quit IRC | 23:06 | |
| openstackgerrit | A change was merged to openstack-infra/config: Update gate script path in requirements job https://review.openstack.org/63224 | 23:07 |
| jhesketh__ | clarkb: thanks, I just tried the comment that was breaking previously and that works now: https://review-dev.openstack.org/#/c/5232/ | 23:07 |
| clarkb | zaro: pip? | 23:09 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1262906 https://review.openstack.org/63239 | 23:09 |
| uvirtbot | Launchpad bug 1262906 in neutron "swift unable to start in gate-tempest-dsvm-neutron-large-ops" [Undecided,Confirmed] https://launchpad.net/bugs/1262906 | 23:10 |
| clarkb | zaro: this is definitely new. the scp connection stayed open before long enough to copy the files | 23:10 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 23:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't attempt to open an interactive editor https://review.openstack.org/62116 | 23:12 |
| *** harlowja is now known as harlowja_away | 23:12 | |
| *** b3nt_pin has joined #openstack-infra | 23:13 | |
| *** senk has joined #openstack-infra | 23:13 | |
| *** beagles has quit IRC | 23:14 | |
| *** b3nt_pin is now known as beagles | 23:14 | |
| *** Abhishek has quit IRC | 23:15 | |
| jeblair | clarkb, fungi: logstash is still keeping up; i'm going to shut down two more worker nodes (4 worker processes) | 23:15 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1262906 https://review.openstack.org/63239 | 23:16 |
| uvirtbot | Launchpad bug 1262906 in neutron "swift unable to start in gate-tempest-dsvm-neutron-large-ops" [Undecided,Confirmed] https://launchpad.net/bugs/1262906 | 23:16 |
| clarkb | jeblair: ok | 23:17 |
| *** yfried1 has quit IRC | 23:17 | |
| *** beagles has quit IRC | 23:19 | |
| clarkb | zaro: I think the catch for IOException is there because java forces you to catch exceptions somewhere | 23:22 |
| *** fbo is now known as fbo_away | 23:22 | |
| clarkb | zaro: we can try catching it properly, but I am not sure there is much we can do. would be better to stop it from happening all together | 23:22 |
| *** senk has quit IRC | 23:25 | |
| *** b3nt_pin has joined #openstack-infra | 23:26 | |
| clarkb | jeblair: if you look at the zuul status there are some gate-config-layout jobs that are not running jenkins01/02 appear to have free slaves that can run those jobs. Any idea of what is going on? I suppose looking in gearman is what should happen? | 23:27 |
| jeblair | clarkb: i think they run on bare-precise nodes | 23:29 |
| stevebaker | hmm, any idea what this error means? no tests ran https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-heat-slow/5/console | 23:29 |
| clarkb | jeblair: oohhh | 23:30 |
| jeblair | clarkb: nodepool just marked one ready | 23:30 |
| clarkb | jeblair: thanks that clears it up for me | 23:30 |
| reed | Ryan_Lane, have you had time to work on the wiki's skin to address the issues with the translation extension? | 23:30 |
| clarkb | stevebaker: the test discovery process exited with status code 255 | 23:31 |
| Ryan_Lane | not yet | 23:31 |
| Ryan_Lane | hoping to do that next week | 23:31 |
| clarkb | stevebaker: there was no stderr or stdout which is unfortunate | 23:31 |
| jeblair | clarkb: we should change those to performance nodes; i think they'll build faster | 23:32 |
| clarkb | jeblair: ++ | 23:32 |
| clarkb | stevebaker: if you want to retrigger that job, link me to the build and I can hold the test slave for investigative purposes. Or you can run it locally | 23:32 |
| *** sarob has quit IRC | 23:33 | |
| *** b3nt_pin is now known as beagles | 23:33 | |
| *** nati_uen_ has joined #openstack-infra | 23:34 | |
| *** nati_uen_ has quit IRC | 23:34 | |
| *** nati_uen_ has joined #openstack-infra | 23:35 | |
| *** sarob has joined #openstack-infra | 23:35 | |
| *** dstanek has quit IRC | 23:36 | |
| *** nati_ueno has quit IRC | 23:37 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Switch IAD bare-precise to performance https://review.openstack.org/63242 | 23:38 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Switch to using IAD performance nodes https://review.openstack.org/63243 | 23:38 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Switch ORD bare-precise to performance https://review.openstack.org/63244 | 23:38 |
| jeblair | clarkb: piece of cake ^ :) | 23:38 |
| *** romcheg has quit IRC | 23:38 | |
| clarkb | jeblair: for your config change that sets d-g timeouts, do you have a d-g change for timeouts that you are happy with? | 23:39 |
| clarkb | jeblair: also jenkins says you need to rebase the above changes :) | 23:39 |
| *** romcheg has joined #openstack-infra | 23:39 | |
| jeblair | clarkb: not a final change, but i have a minimally acceptable plan; i don't expect the details of that to affect the jjb changes; having them in place will make d-g changes more self-testing. | 23:42 |
| clarkb | jeblair: ok | 23:42 |
| *** hogepodge has joined #openstack-infra | 23:42 | |
| jeblair | clarkb: (it would be cool if the build timeout plugin set an env var, but that doesn't seem to be the case; we could probably make it a job parameter, but that's a bit clunky) | 23:43 |
| *** hogepodge has joined #openstack-infra | 23:43 | |
| *** prad has quit IRC | 23:43 | |
| *** rongze has joined #openstack-infra | 23:43 | |
| *** jooools has quit IRC | 23:44 | |
| clarkb | ya, this will work for now | 23:44 |
| jeblair | there's no reason those should have a merge conflict...checking zuul logs. | 23:45 |
| jeblair | OSError: [Errno 24] Too many open files | 23:45 |
| jeblair | clarkb: didn't i read something about that being fixed? | 23:45 |
| clarkb | gah, did I not fix that? | 23:45 |
| clarkb | jeblair: ya there was a change to only have Git repo objects when needed whcih should GC the file descriptors | 23:46 |
| clarkb | we even have a test for it | 23:46 |
| jeblair | clarkb: i saved an lsof output; anything else to do before restarting? | 23:47 |
| *** rongze has quit IRC | 23:48 | |
| clarkb | jeblair: I took a threaddump last time too | 23:48 |
| *** slong has quit IRC | 23:49 | |
| clarkb | accoring to lsof just under 2k of those file descriptors are related to git. and about 500 are tcp connections | 23:49 |
| clarkb | geard tcp connections | 23:49 |
| *** dcramer_ has joined #openstack-infra | 23:50 | |
| jeblair | clarkb: geard is a different process | 23:50 |
| jeblair | clarkb: (and 500 connections sounds right) | 23:50 |
| clarkb | jeblair: right but the limit is per user via ulimit | 23:51 |
| *** CaptTofu has joined #openstack-infra | 23:52 | |
| jeblair | clarkb: let's discuss that later; i caused a thread dump; anything else you want? | 23:52 |
| clarkb | jeblair: nope thats it | 23:52 |
| jeblair | unfortunately i can't save a copy of the queues | 23:52 |
| jeblair | i'm digging the current queue out of the debug log | 23:53 |
| *** CaptTofu has quit IRC | 23:54 | |
| *** hogepodge has quit IRC | 23:56 | |
| *** harlowja_away is now known as harlowja | 23:57 | |
| clarkb | jeblair: anyting else I can do to help? | 23:58 |
| *** gothicmindfood has joined #openstack-infra | 23:58 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1262613 https://review.openstack.org/63126 | 23:58 |
| uvirtbot | Launchpad bug 1262613 in tempest "tempest.scenario.test_cross_tenant_connectivity.TestNetworkCrossTenant.test_cross_tenant_traffic[compute,gate,network,smoke] failed" [High,Fix released] https://launchpad.net/bugs/1262613 | 23:58 |
| *** dprince has quit IRC | 23:59 | |
| jeblair | clarkb: nah, i just have some tedious work; the log doesn't have enough info to build the commands, so it's back to gerrit. | 23:59 |
| *** jhesketh__ has quit IRC | 23:59 | |
| *** jhesketh__ has joined #openstack-infra | 23:59 | |
| *** yamahata has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!