Monday, 2014-01-13

« Sunday, 2014-01-12 Index Tuesday, 2014-01-14 »
*** openstack has joined #openstack-qa13:37
*** salv-orlando has quit IRC13:47
openstackrecheckopenstack/tempest change: https://review.openstack.org/58697 failed tempest because of: https://bugs.launchpad.net/bugs/125389613:52
*** sthaha has joined #openstack-qa13:54
*** sthaha has joined #openstack-qa13:54
*** yfried has quit IRC13:56
*** vkozhukalov has joined #openstack-qa13:58
*** salv-orlando has joined #openstack-qa13:58
*** salv-orlando has quit IRC14:00
*** jasondotstar has joined #openstack-qa14:02
openstackrecheckopenstack/python-ceilometerclient change: https://review.openstack.org/66254 failed tempest with an unrecognized error14:06
openstackrecheckopenstack/neutron change: https://review.openstack.org/66310 failed tempest with an unrecognized error14:06
openstackgerritA change was merged to openstack-dev/devstack: Correct Qpid package name in files/apts/neutron  https://review.openstack.org/6568314:09
openstackgerritA change was merged to openstack-dev/devstack: Robustify service shutdown  https://review.openstack.org/6608014:09
*** ewindisch is now known as zz_ewindisch14:09
openstackgerritA change was merged to openstack-dev/devstack: Configuration changes required to support VMware NSX plugin  https://review.openstack.org/6500214:09
*** zz_ewindisch is now known as ewindisch14:11
*** ewindisch is now known as zz_ewindisch14:15
openstackgerritVladimir Kozhukalov proposed a change to openstack/tempest: Initial ironic integration testing scenario  https://review.openstack.org/6442014:15
*** zz_ewindisch is now known as ewindisch14:16
*** ewindisch is now known as zz_ewindisch14:17
*** dims has quit IRC14:22
*** dims has joined #openstack-qa14:25
openstackrecheckopenstack/tempest change: https://review.openstack.org/65234 failed tempest with an unrecognized error14:25
openstackgerritVladimir Kozhukalov proposed a change to openstack-dev/devstack: Some configuration options for tempest scenario  https://review.openstack.org/6440014:27
openstackgerritJerry Cai proposed a change to openstack/tempest: Fix volume metadata validation of identical to request  https://review.openstack.org/6589814:28
openstackgerritjun xie proposed a change to openstack/tempest: Create security group rule with additional args  https://review.openstack.org/6629314:29
*** damnsmith is now known as dansmith14:31
mkodererdkranz: do you have time for a short call?14:34
openstackgerritMarc Koderer proposed a change to openstack/tempest: Autogeneration framework for negative tests  https://review.openstack.org/6473314:38
*** yrabl has quit IRC14:40
openstackgerritA change was merged to openstack-dev/devstack: clean up ubuntu versions  https://review.openstack.org/6483614:40
openstackgerritVladimir Kozhukalov proposed a change to openstack-dev/devstack: Some configuration options for tempest scenario  https://review.openstack.org/6440014:47
*** mordred has quit IRC14:50
*** mordred has joined #openstack-qa14:50
*** psharma has quit IRC14:50
*** malini_afk is now known as malini14:50
*** coolsvap has joined #openstack-qa14:54
dkranzmkoderer: Just got out of a meeting, sure.14:57
*** sthaha has quit IRC14:58
mkodererdkranz: ok just give me a call when you are ready14:59
openstackgerritandrea-frittoli proposed a change to openstack/tempest: Implements official clients with retry  https://review.openstack.org/6197814:59
dkranzmkoderer: ok, in one minute15:00
*** vkozhukalov has quit IRC15:10
*** mrodden has joined #openstack-qa15:11
openstackrecheckopenstack/tempest change: https://review.openstack.org/58697 failed tempest with an unrecognized error15:13
*** rwsu has joined #openstack-qa15:16
*** coolsvap has quit IRC15:21
*** yfried has joined #openstack-qa15:25
openstackrecheckopenstack/tempest change: https://review.openstack.org/60831 failed tempest with an unrecognized error15:30
*** salv-orlando has joined #openstack-qa15:30
openstackrecheckopenstack/python-keystoneclient change: https://review.openstack.org/66302 failed tempest with an unrecognized error15:30
*** mrodden has quit IRC15:32
*** markmcclain has joined #openstack-qa15:33
openstackgerritMalini Kamalambal proposed a change to openstack-dev/devstack: Add Marconi to Tempest config  https://review.openstack.org/6547815:35
openstackgerritMarc Koderer proposed a change to openstack/tempest: Autogeneration framework for negative tests  https://review.openstack.org/6473315:37
*** salv-orlando has quit IRC15:38
*** markmcclain has quit IRC15:41
*** markmcclain has joined #openstack-qa15:42
*** coolsvap has joined #openstack-qa15:42
mkodererdkranz: you should have it on github now15:44
mkodererdkranz: I sent you a mail with a how to merge squash the commits15:44
dkranzmkoderer: Thanks15:46
openstackrecheckopenstack/tempest change: https://review.openstack.org/66293 failed tempest with an unrecognized error15:48
*** tkammer has quit IRC15:49
*** nmagnezi has quit IRC15:51
*** markmcclain has left #openstack-qa15:56
*** mrodden has joined #openstack-qa15:56
*** SergeyLukjanov has quit IRC15:57
*** markmcclain has joined #openstack-qa16:00
*** ruhe is now known as ruhe_16:01
*** ruhe_ is now known as ruhe16:01
*** jlibosva1 has quit IRC16:04
openstackgerritJordan Pittier proposed a change to openstack/tempest: Make endpoint type configurable  https://review.openstack.org/6288716:08
*** Anju1 has joined #openstack-qa16:08
mtreinishdkranz: ping16:12
openstackgerritjun xie proposed a change to openstack/tempest: Create security group rule with additional args  https://review.openstack.org/6629316:14
*** zz_ewindisch is now known as ewindisch16:18
*** afazekas has quit IRC16:19
*** dkorn has quit IRC16:36
dkranzmtreinish: pong16:37
mtreinishdkranz: can you run the meeting on Thursday? I'll be in Montreal and probably won't be able to16:40
dkranzmtreinish: Sure16:41
mtreinishdkranz: ok great thanks16:41
dkranzmtreinish: When does Montreal start?16:41
*** ruhe is now known as ruhe_16:42
mtreinishit's the 15th-17th. I'll be driving up there tomorrow afternoon16:42
openstackgerritandrea-frittoli proposed a change to openstack/tempest: Implements official clients with retry  https://review.openstack.org/6197816:45
*** ruhe_ has quit IRC16:47
openstackrecheckopenstack/requirements change: https://review.openstack.org/66358 failed tempest because of: https://bugs.launchpad.net/bugs/125487216:49
*** andreaf has quit IRC16:49
*** SergeyLukjanov has joined #openstack-qa16:50
*** andreaf has joined #openstack-qa16:51
*** andreaf has quit IRC16:53
*** andreaf has joined #openstack-qa16:53
*** mlavalle has joined #openstack-qa16:56
*** markmcclain has quit IRC16:58
*** markmcclain has joined #openstack-qa17:00
openstackrecheckopenstack/python-neutronclient change: https://review.openstack.org/65924 failed tempest with an unrecognized error17:02
openstackrecheckopenstack/neutron change: https://review.openstack.org/64822 failed tempest with an unrecognized error17:02
*** vkozhukalov has joined #openstack-qa17:03
openstackgerritEmilien Macchi proposed a change to openstack/tempest: network: add metering api operations  https://review.openstack.org/6614317:13
*** hemnafk is now known as hemna_17:24
*** jlibosva has joined #openstack-qa17:26
*** Anju1 has left #openstack-qa17:35
*** fbo is now known as fbo_away17:37
openstackrecheckopenstack/tempest change: https://review.openstack.org/61978 failed tempest with an unrecognized error17:45
openstackgerritA change was merged to openstack-dev/devstack: Setup Keystone catalog information for Ceilometer  https://review.openstack.org/6567817:46
openstackrecheckopenstack/neutron change: https://review.openstack.org/65774 failed tempest because of: https://bugs.launchpad.net/bugs/123543517:48
*** nati_ueno has joined #openstack-qa17:53
*** harlowja_away is now known as harlowja17:59
openstackgerritJoe Hakim Rahme proposed a change to openstack/tempest: Tests object_storage extension listing  https://review.openstack.org/6638418:00
*** amotoki_ has joined #openstack-qa18:01
*** amotoki has quit IRC18:03
openstackrecheckopenstack/tempest change: https://review.openstack.org/64782 failed tempest with an unrecognized error18:06
*** rossella_s has quit IRC18:13
*** gszasz has quit IRC18:18
*** jasondotstar has quit IRC18:18
*** jasondotstar has joined #openstack-qa18:20
dkranzsdague: You might remove your -2 from https://review.openstack.org/#/c/58697/ since I think your objections were addressed.18:20
openstackrecheckopenstack/tempest change: https://review.openstack.org/66143 failed tempest with an unrecognized error18:20
openstackrecheckopenstack/neutron change: https://review.openstack.org/56263 failed tempest with an unrecognized error18:21
*** jpich has quit IRC18:26
*** vkozhukalov has quit IRC18:29
*** ruhe has joined #openstack-qa18:30
openstackgerritA change was merged to openstack-dev/devstack: Fix Heat/Cloud formation catalog template  https://review.openstack.org/6591618:39
openstackgerritA change was merged to openstack-dev/devstack: Enable server-side and client-side logs for libvirt  https://review.openstack.org/6583418:40
openstackgerritMauro Rodrigues proposed a change to openstack/tempest: Add aggregates scenario test  https://review.openstack.org/6261818:40
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: moving readiness checks into stream  https://review.openstack.org/6596118:41
sdaguedkranz: sure18:42
openstackgerritA change was merged to openstack-dev/devstack: gitignore: add .localrc.auto and local.conf  https://review.openstack.org/6543318:50
openstackgerritA change was merged to openstack-dev/devstack: Remove duplicated name services  https://review.openstack.org/6512118:50
openstackrecheckopenstack/heat change: https://review.openstack.org/66265 failed tempest with an unrecognized error18:52
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Update README.rst with details about unit tests  https://review.openstack.org/6639618:55
*** yrabl has joined #openstack-qa18:55
*** jlibosva has quit IRC18:56
*** yrabl_ has joined #openstack-qa19:02
*** yrabl has quit IRC19:05
*** yassine has quit IRC19:08
*** yassine has joined #openstack-qa19:08
*** yassine has quit IRC19:09
*** yassine has joined #openstack-qa19:09
*** yassine has quit IRC19:10
*** RecheckTestBot has joined #openstack-qa19:10
*** RecheckTestBot has quit IRC19:10
*** RecheckTestBot has joined #openstack-qa19:11
*** yassine has joined #openstack-qa19:12
*** jlibosva has joined #openstack-qa19:12
*** yassine has quit IRC19:12
*** yassine has joined #openstack-qa19:12
*** yassine has quit IRC19:13
*** DennyZhang has joined #openstack-qa19:13
*** yassine has joined #openstack-qa19:14
*** RecheckTestBot has quit IRC19:17
*** RecheckTestBot has joined #openstack-qa19:18
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Improve testing of list_extensions for compute  https://review.openstack.org/5880319:19
*** ruhe is now known as ruhe_19:23
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: refactor bot to be based on argparse  https://review.openstack.org/6640119:24
*** openstackgerrit has quit IRC19:31
*** openstackgerrit has joined #openstack-qa19:31
*** RecheckTestBot has quit IRC19:33
*** RecheckTestBot has joined #openstack-qa19:33
*** giulivo has quit IRC19:34
*** ruhe_ is now known as ruhe19:36
*** ruhe has quit IRC19:36
*** RecheckTestBot has quit IRC19:40
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: fix bot to actually use super()  https://review.openstack.org/6640519:42
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: refactor bot to be based on argparse  https://review.openstack.org/6640119:42
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: moving readiness checks into stream  https://review.openstack.org/6596119:42
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: parse the failed jobs in stream  https://review.openstack.org/6578919:42
*** jgrimm has joined #openstack-qa19:43
*** RecheckTestBot has joined #openstack-qa19:43
openstackrecheckopenstack/tempest change: https://review.openstack.org/64476 failed tempest with an unrecognized error19:45
mtreinishsdague: lazy copy and paste: http://git.openstack.org/cgit/openstack-infra/gerritbot/tree/gerritbot/bot.py#n7119:45
mtreinishI'll fix it there too...19:46
sdaguemtreinish: good call19:47
sdagueI seem to not have pep8 entirely in order19:47
sdagueI'll have to fix for that as well19:47
sdaguehonestly patch #2 of that it the complicated one to review19:47
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: fix bot to actually use super()  https://review.openstack.org/6640519:50
openstackrecheckopenstack/glance change: https://review.openstack.org/59725 failed tempest with an unrecognized error19:57
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Convert scenario test tearDown to addCleanup  https://review.openstack.org/6210119:58
sdaguemtreinish: how do I turn down the log level for the gerrit watcher?20:02
*** nati_uen_ has joined #openstack-qa20:03
*** nati_ueno has quit IRC20:04
mtreinishsdague: you need to have a log_config option in your config file20:04
mtreinishthat points to a python logging config file20:05
mtreinishthat specifies the log level20:05
sdagueso I'd like to make the defaults not put the underlying libs into debug20:05
*** markmcclain has quit IRC20:06
*** nati_uen_ has quit IRC20:07
*** nati_ueno has joined #openstack-qa20:07
mtreinishsdague: if I remember correctly I think you'll have to get the logger for that underlying lib and run setLevel on it20:08
mtreinishsince we set the root logger level to debug when we create the BasicConfig object if a file isn't specified20:09
*** DennyZhang has quit IRC20:09
*** DennyZhang has joined #openstack-qa20:09
mtreinishsdague: I'll push a patch changing it which log level did you want to set it to20:11
*** markmcclain has joined #openstack-qa20:12
sdagueI'd like everything that's not our logger to be INFO20:12
sdagueand our namespace to be DEBUG20:12
openstackgerritA change was merged to openstack/tempest: Fix volume metadata validation of identical to request  https://review.openstack.org/6589820:16
openstackgerritMatthew Treinish proposed a change to openstack-infra/elastic-recheck: Set default log level to INFO level for non elastic-recheck loggers  https://review.openstack.org/6641220:17
openstackrecheckopenstack/neutron change: https://review.openstack.org/66388 failed tempest because of: https://bugs.launchpad.net/bugs/126118220:17
mtreinishsdague: ^^^ I think that should do what you're looking for20:19
mtreinishI didn't test it though20:19
sdaguelet me integrated it into this new logging patch20:24
mtreinishjaypipes: I actually don't think that addCleanup patch is going to pass the gate. I really should start testing things before I push them.20:24
jaypipesmtreinish: why, what's up with it?20:25
*** RecheckTestBot_ has joined #openstack-qa20:25
*** RecheckTestBot_ is now known as RecheckTestBot__20:25
mtreinishjaypipes: I keep missing resources somewhere I think. Things were failing before with in-use errors.20:26
jaypipesI see...20:26
*** RecheckTestBot_ has joined #openstack-qa20:26
*** RecheckTestBot_ has quit IRC20:26
mtreinishI rebased and found another 2 addCleanups I missed. But, I'm not sure I got them all20:26
sdaguemtreinish: yeh, that's not quite right20:27
*** RecheckTestBot__ has quit IRC20:27
openstackgerritDean Troyer proposed a change to openstack-dev/devstack: Updates for tools/info.sh  https://review.openstack.org/6641720:27
*** RecheckTestBot has quit IRC20:28
*** malini is now known as malini_afk20:28
*** DennyZhang has quit IRC20:31
*** amotoki has joined #openstack-qa20:34
*** RecheckTestBot has joined #openstack-qa20:36
*** amotoki_ has quit IRC20:36
mtreinishsdague: what did I mess up?20:37
openstackgerritA change was merged to openstack/tempest: Add heat stack action test case  https://review.openstack.org/6524820:37
*** RecheckTestBot has quit IRC20:37
openstackrecheckopenstack/keystone change: https://review.openstack.org/60742 failed tempest with an unrecognized error20:38
*** RecheckTestBot has joined #openstack-qa20:39
sdagueI don't think basic setup hits the other namespaces20:39
sdagueI figured out how to do it20:39
mtreinishsdague, dkranz: can I get eyes on: https://review.openstack.org/#/c/62063/ ? It bounced off the gate because of a merge conflict20:39
mtreinishsdague: ok cool20:40
dkranzmtreinish: looking...20:41
*** RecheckTestBot has quit IRC20:41
*** RecheckTestBot has joined #openstack-qa20:42
sdaguemtreinish: reapproved20:43
dkranzmtreinish: sdague beat me to it20:45
*** RecheckTestBot has quit IRC20:46
mtreinishsdague, dkranz: thanks20:48
sdagueyou know, we need more jobs in the queue, because I don't have enough fails at the moment to know if the bot is working or not :)20:50
*** fbo_away is now known as fbo20:53
*** mkollaro has quit IRC20:54
*** fbo is now known as fbo_away20:54
openstackgerritVadim Rovachev proposed a change to openstack/tempest: Create telemetry client  https://review.openstack.org/5527620:54
*** nati_uen_ has joined #openstack-qa20:55
*** amotoki is now known as amotoki_20:56
*** amotoki has joined #openstack-qa20:56
*** jaypipes has quit IRC20:58
*** nati_ueno has quit IRC20:59
*** fbo_away is now known as fbo21:04
openstackrecheckopenstack/tempest change: https://review.openstack.org/62101 failed tempest because of: https://bugs.launchpad.net/bugs/123543521:05
*** RecheckTestBot has joined #openstack-qa21:09
*** RecheckTestBot has quit IRC21:11
*** RecheckTestBot has joined #openstack-qa21:11
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Convert scenario test tearDown to addCleanup  https://review.openstack.org/6210121:12
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: create more sane logging for the er bot  https://review.openstack.org/6643521:12
mtreinishsdague: ^^^ so it was the inverse of my patch that worked :)21:13
*** RecheckTestBot has quit IRC21:16
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: create more sane logging for the er bot  https://review.openstack.org/6643521:16
*** RecheckTestBot has joined #openstack-qa21:17
*** andreaf has quit IRC21:18
openstackrecheckopenstack/horizon change: https://review.openstack.org/57232 failed tempest with an unrecognized error21:18
sdaguemtreinish: basically :)21:21
sdaguealso... timestamps21:21
*** andreaf has joined #openstack-qa21:21
sdaguekinda useful in logs21:21
rahmumtreinish: I submitted this patch that adds list_extension to the object_storage client https://review.openstack.org/#/c/66384/. The goal is to add it to the config-verification tool21:21
rahmulet me know what you think21:21
mtreinishrahmu: I was just looking at that actually, it looks good after a quick glance21:23
mtreinishI want to test it locally though21:23
rahmumtreinish: great. I'll include to verify_tempest_config as soon as this patch is merged then.21:25
*** andreaf has quit IRC21:25
mtreinishrahmu: well you'll have to wait on https://review.openstack.org/#/c/62063/ too21:25
mtreinishbut that's in the gate queue right now21:25
rahmuyes I assumed so21:27
openstackgerritEric Windisch proposed a change to openstack-dev/devstack: Upload docker images directly into glance  https://review.openstack.org/6644021:29
*** mrodden has quit IRC21:30
openstackrecheckopenstack/horizon change: https://review.openstack.org/56086 failed tempest with an unrecognized error21:32
*** SergeyLukjanov has quit IRC21:37
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: refactor bot to be based on argparse  https://review.openstack.org/6640121:39
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: fix bot to actually use super()  https://review.openstack.org/6640521:39
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: create more sane logging for the er bot  https://review.openstack.org/6643521:39
openstackrecheckopenstack/glance change: https://review.openstack.org/59725 failed tempest with an unrecognized error21:44
*** andreaf has joined #openstack-qa21:48
*** RecheckTestBot has quit IRC21:51
*** RecheckTestBot has joined #openstack-qa21:51
sdaguemtreinish: well, I found at least one terrible bug in my refactor21:55
sdaguelet's see if it sorts things out now21:55
mtreinishsdague: do you think that bug is the cause behind the e-r misses?21:58
sdagueno, I think the e-r misses are because e-s is not getting content quick enough21:58
sdaguerunning the local instance I'm seeing lots of timing out waiting for content21:59
sdaguethe new code logs explicitly when it does that21:59
sdaguewith the current looping, you get 13 minutes for console logs to be uploaded, otherwise we time out22:00
sdagueI don't think ES is in that window at the moment22:00
openstackrecheckopenstack/horizon change: https://review.openstack.org/66424 failed tempest with an unrecognized error22:01
openstackrecheckopenstack/nova change: https://review.openstack.org/63735 failed tempest with an unrecognized error22:01
openstackrecheckopenstack/tempest change: https://review.openstack.org/55276 failed tempest with an unrecognized error22:01
openstackrecheckopenstack/tempest change: https://review.openstack.org/62101 failed tempest because of: https://bugs.launchpad.net/bugs/1251448 and https://bugs.launchpad.net/bugs/123543522:01
mtreinishsdague: that's unfortunate, we used to be at ~30min but you're log filtering helped that alot22:02
mtreinishit used to be very quick on avg22:02
*** jlibosva has quit IRC22:02
sdagueyeh, I think the bayesian analysis might be pushing us over the edge again22:02
openstackrecheckopenstack-infra/devstack-gate change: https://review.openstack.org/65805 failed tempest because of: https://bugs.launchpad.net/bugs/124906522:04
*** jasondotstar has quit IRC22:05
RecheckTestBotopenstack/tempest change: https://review.openstack.org/55276 failed tempest with an unrecognized error22:06
sdaguewell that's promissing22:07
*** amotoki has quit IRC22:07
*** andreaf has quit IRC22:09
mtreinishsdague: that it found one that the infra bot didn't?22:09
sdagueso 62101 is a good instance where the tempest dsvm logs haven't yet posted, but because we aren't job filtered, it matched on whatever was there22:09
sdaguemtreinish: not it found the same one22:09
sdagueor didn't fine the same one22:09
sdague[recheckwatchbot] Console logs not ready for check-tempest-dsvm-full 62101,1322:10
mtreinishsdague: oh, ok I was talking about the irc message about 5527622:11
sdagueyep22:11
sdagueI'm actually thinking that barring publicly accessible logging, we should actually make this spit back out to the channel when it times out22:12
sdaguethen we'd at least know22:12
mtreinishsdague: why can't we dump the logs somewhere public readable?22:13
mtreinishit can't be that large22:13
sdaguesec concerns22:13
sdaguein case we're leaking info we shouldn't be22:13
sdaguesee conversation with fungi about it in -infra22:13
sdaguewhich I understand, and honestly, we could do better with the bot here22:14
mtreinishsdague: I'm just worried about the irc spam level.22:14
mtreinishbut I guess it wouldn't be any worse then a high failure rate day with e-r working well22:15
sdagueright, only do it if we timeout on a fail22:16
sdaguethen at least we'd know that we timed out22:16
sdagueotherwise it's going to be all superstition22:17
sdaguethis particular change is fun, as it failed *all* the tempest jobs22:17
mtreinishsdague: yeah I'm doing patch debug in the gate. I borked the devstack node I was writing it on22:18
mtreinishI'm also not convinced I can get it working with the neutron api22:18
mtreinishI need to spin on it somemore22:18
sdaguemtreinish: use your rax credit :)22:19
mtreinishI've got the internal dev cloud, it's just I'm too lazy to spin up another guest when I've got the working dir already on one.22:19
fungisdague: mtreinish: note i am still going to dig in the bot log for errors here shortly and let you know if there's anything odd22:19
*** andreaf has joined #openstack-qa22:20
RecheckTestBotopenstack/glance change: https://review.openstack.org/58883 failed tempest with an unrecognized error22:20
sdagueok, it caught the log now, processing22:20
RecheckTestBotopenstack/nova change: https://review.openstack.org/61544 failed tempest with an unrecognized error22:20
*** mrodden has joined #openstack-qa22:21
sdagueoh, maybe it timed out22:21
openstackrecheckopenstack/keystone change: https://review.openstack.org/62894 failed tempest with an unrecognized error22:22
sdagueyeh 62101 timed out22:22
fungisdague: one idea for making its log inspectable... the bot is running on a webserver, so maybe we think about having it write nonsensitive info to a location apache can just serve up straight22:22
sdagueit's working on 65805 now22:22
sdaguefungi: or a 2nd irc channel22:23
fungi#openstack-qa-botspam ftw!22:23
sdagueyeh, maybe22:23
*** salv-orlando has joined #openstack-qa22:23
sdaguehonestly, I think we need to just be a little better are normal in channel reporting22:23
fungisdague: looking on status.o.o, i see two elastic-recheck processes running... one has a start time on january 9th, the other yesterday22:26
sdagueinteresting22:26
fungii'm wondering if we had a restart go awry for it22:27
sdaguethough, realistically, that's not the problem22:27
sdagueyeh, I expect so22:27
sdaguehonestly, I don't trust the restart code22:27
fungiistr this coming up before22:27
fungilast time someone asked me to restart it, might have been mtreinish, and i spotted two of them running with different process start times (back when it was still running on the logstash server)22:28
fungido you want one of them killed (or both, and then a fresh start to replace them)?22:28
sdagueso we should definitely eventually fix that, but realistically, it's not really an issue22:28
sdaguebecause it's stateless22:28
fungik22:28
fungiclient.py is throwing ServerNotConnectedError exceptions in the log22:30
sdagueinteresting22:30
fungii assume one of the two bot processes isn't connected to irc though, which would explain that i think22:30
sdagueknow which process that is?22:30
sdagueyeh, that's what I assume22:31
fungiunfortunately it doesn't seem to log its pid22:31
fungifeature request ;)22:31
sdagueheh22:31
sdagueon every log message?22:31
StevenKWrite a pid file out, instead?22:32
sdagueyeh, making the restart code cleaner is really the right way to do it22:32
fungiStevenK: the issue is we have two processes writing to the same log file and don't know which one is generating errors22:32
sdaguelet us say... the bot code is a bit hackish today22:32
sdaguefungi: right, that's why this is really more about restart code22:32
fungiStevenK: unless you meant pidfile as the answer to broken restart code instead22:32
fungiyeah22:33
fungisdague: comparing traceback timestamps with my channel log, it looks like it's once for each message, yes22:34
RecheckTestBotstackforge/python-solumclient change: https://review.openstack.org/66065 failed tempest with an unrecognized error22:35
RecheckTestBotstackforge/solum change: https://review.openstack.org/64458 failed tempest with an unrecognized error22:35
*** dkorn has joined #openstack-qa22:35
sdagueyeh, so we're just timing out22:35
fungisdague: elasticsearch is taking too long to return query results you think?22:36
*** vrovachev1 has joined #openstack-qa22:36
sdague2014-01-13 17:34:52  WARNING  [recheckwatchbot] Console logs not available after 0:13:52.937565s for check-tempest-dsvm-neutron 65805,122:36
sdaguefungi: no, es is not indexing fast enough22:36
*** salv-orlando_ has joined #openstack-qa22:37
*** dkranz has quit IRC22:37
*** mkollaro has joined #openstack-qa22:37
openstackrecheckopenstack/horizon change: https://review.openstack.org/58214 failed tempest with an unrecognized error22:37
sdagueI sent something to the list on it22:38
*** salv-orlando has quit IRC22:38
*** salv-orlando_ is now known as salv-orlando22:39
fungisdague: oh, i still haven't had the luxury of checking my e-mail22:39
sdagueyep, no worries22:39
fungisdague: we have some workers turned off and can scale back out to them i think22:39
sdagueso this would explain the somewhat random behavior we've been getting22:40
fungibasically there's a huge cluster but we only turned on some of them to see how many we realistically needed running22:40
sdagueyeh, well, we need more :)22:40
funginoted22:40
sdaguebecause we aren't keeping our window22:40
sdagueand this would explain why ER got so random recently22:40
fungii'll check the graphs and see if they're hammered22:41
fungiright now we have 16 logstash workers, but i think only like half of them are set to be active for the moment22:41
*** RecheckTestBot has quit IRC22:42
fungicpu utilization on logstash-worker01 was higher last week, not quite as bad today... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1267&rra_id=all22:44
*** RecheckTestBot has joined #openstack-qa22:45
fungi04 looks similar22:45
*** RecheckTestBot has quit IRC22:47
*** RecheckTestBot has joined #openstack-qa22:47
sdaguewell they are definitely not keeping up to the 5 minute mark that we were once running on22:47
openstackgerritSylvain Afchain proposed a change to openstack/tempest: Improve the extra routes test on router.  https://review.openstack.org/6645422:49
fungiyep, so we have 4 out of 16 started right now, looks like22:51
*** rwsu has quit IRC22:52
fungiunfortunately, the activity volume was so low over the holidays we couldn't really gauge the impact of the crm114 addition, so started small with the cluster22:52
fungirather, with the number of active nodes in the new cluster22:52
*** safchain has quit IRC22:53
*** salv-orlando has quit IRC22:53
*** jgrimm has quit IRC22:56
openstackrecheckopenstack/neutron change: https://review.openstack.org/58897 failed tempest with an unrecognized error22:59
sdaguefungi: yep, understood23:01
sdaguejust want to get us back up to steam, because right now we've definitely lost effectiveness23:02
openstackgerritAlexander Ignatov proposed a change to openstack/tempest: Remove copyright from empty files  https://review.openstack.org/6645523:02
*** dims has quit IRC23:03
fungii should have taken better notes when we were discussing how to add them back to the pool, but i can probably muddle through23:03
sdaguewhen do the rest of the troops return?23:04
*** rwsu has joined #openstack-qa23:07
fungii thought they were back today, but i guess not23:08
*** fbo is now known as fbo_away23:09
sdagueI think jeblair might be flying tomorrow, as he wasn't going to make that TC meeting23:10
sdagueor else he assumes he'll be asleep23:10
*** RecheckTestBot has quit IRC23:11
*** RecheckTestBot has joined #openstack-qa23:12
*** andreaf_ has joined #openstack-qa23:12
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Convert scenario test tearDown to addCleanup  https://review.openstack.org/6210123:12
*** andreaf_ has quit IRC23:12
*** andreaf_ has joined #openstack-qa23:13
*** andreaf has quit IRC23:14
*** jaypipes has joined #openstack-qa23:14
*** ChanServ sets mode: +o jaypipes23:14
*** dims has joined #openstack-qa23:18
fungisdague: so, jogging my memory by looking at the logstash workers, each of the 16 servers runs up to 4 worker processes. at the moment each of 01-04 have four workers running, while 05-16 have none. i'm going to add four running workers on 05 and 06 now, which should give us 8 more worker processes (a 50% increase)23:23
fungiand then we can see whether we're keeping up, or still need more23:23
*** andreaf_ has quit IRC23:24
*** openstackgerrit has quit IRC23:24
*** openstackgerrit has joined #openstack-qa23:24
fungiand yeah, i knew he was still away, just didn't realize the others weren't planning to be around today either23:31
*** markmcclain has quit IRC23:31
fungi(not that i bothered to ask... was a silly assumption on my part)23:31
*** dkorn has quit IRC23:31
*** yfried has quit IRC23:33
*** rfolco has quit IRC23:39
*** vrovachev1 has left #openstack-qa23:41
openstackrecheckopenstack/swift change: https://review.openstack.org/60707 failed tempest with an unrecognized error23:44
sdague:)23:51
sdagueyeh, I just don't want to extra swamp you23:51
fungii don't think i'd even notice at this point23:52
sdaguebecause at some point I actually need to pair up with someone on infra root about getting publish pipeline to openstack.org for tc documents23:52
sdaguebut I want you guys to have some breathing room before I pull someone off with that23:52
fungiautopublishing docs from openstack/governance or elsewhere?23:53
sdaguefrom openstack/governance23:53
sdaguejohnathan +1ed the idea of doing it, just need to get sftp link job in infra somewhere23:53
sdaguewhich means secret creds, so needs infra root involved for sure23:54
sdagueok, dinner time23:54
*** mrodden has quit IRC23:56
*** harlowja has quit IRC23:57
RecheckTestBotopenstack/oslo-incubator change: https://review.openstack.org/66458 failed tempest with an unrecognized error23:58
RecheckTestBotstackforge/turbo-hipster change: https://review.openstack.org/66452 failed tempest with an unrecognized error23:59
« Sunday, 2014-01-12 Index Tuesday, 2014-01-14 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!