Thursday, 2014-11-20

clarkbjogo: tempest just started00:03
openstackgerritJoe Gordon proposed openstack-infra/elastic-recheck: Add query for bug 1394408  https://review.openstack.org/13578900:08
* jogo watches00:08
*** Sukhdev has quit IRC00:08
*** rbak has quit IRC00:08
*** oomichi_ has joined #openstack-qa00:09
mtreinishjogo: 2014-11-20 00:08:42.401 | {0} tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestXML.test_live_block_migration [5.044152s] ... FAILED00:10
mtreinishat least it tried :)00:11
jogomtreinish: mtreinish woot!00:11
clarkbthere is a non block one too ya?00:11
jogohmm  SKIPPED: Block Live migration not available00:12
mtreinishjogo: that's the isci one00:12
mtreinishclarkb: no, I think they're all called block migration00:12
mtreinishclarkb: when I looked at the code there were 2 tests both do live migrate on a server00:13
mtreinishthe only difference is that one has a volume attached00:13
jogomtreinish: ohh {USE_BLOCK_MIGRATION_FOR_LIVE_MIGRATION00:13
jogowell if it doesn't work without a volume it won't work with most likely00:14
mtreinishjogo: yeah, that's the other option, but I figured you only wanted to try the first one without a volume00:14
openstackgerritMerged openstack-infra/elastic-recheck: Add query for bug 1394408  https://review.openstack.org/13578900:14
mtreinishthat's all your tempest patch was turning on00:14
jogoyeah00:14
jogonow I just have to wait for the logs to be collected00:14
*** adalbas has quit IRC00:14
jogoif it worked on the first try I wouldn't have believed it anyway00:15
*** dims has quit IRC00:15
*** dims has joined #openstack-qa00:15
*** yamamoto has joined #openstack-qa00:16
*** jecarey has quit IRC00:18
*** openstackrecheck has quit IRC00:20
*** openstackrecheck has joined #openstack-qa00:20
*** wuhg has joined #openstack-qa00:20
*** yamamoto has quit IRC00:20
*** gmann__ has quit IRC00:21
*** gmann has joined #openstack-qa00:22
oomichi_mtreinish: hi, can you take a look at https://review.openstack.org/#/c/135482/ ?00:23
jogomtreinish:  I am a little surprised at how fast that test failed actually00:23
jogo5 seconds00:23
clarkbjogo: failing fast is our speciality00:23
jogoclarkb: well we fail fast and slow and randomly00:24
jogoclarkb: if we just failed fast that would be good00:24
jogoand fail quitely00:24
mtreinishoomichi_: sure, one sec00:24
mtreinishjogo: it probably means it never did anything00:24
mtreinishlike it failed trying to do the first thing00:25
oomichi_mtreinish: thanks!00:25
jogomtreinish: yeah :/00:26
*** adalbas has joined #openstack-qa00:26
mtreinishoomichi_: +A00:28
openstackgerritIan Wienand proposed openstack-dev/devstack: only use lsb_release for distro tags  https://review.openstack.org/11710400:29
mtreinishoomichi_: I'll trade you: https://review.openstack.org/13511500:29
mtreinishjogo: http://paste.openstack.org/show/134987/00:30
mtreinishit's a configuration thing00:31
mtreinishclarkb: ^^^00:31
mtreinishto do live migration they need shared storage00:31
*** dmorita has joined #openstack-qa00:32
clarkbthe VM disks need to be on the same fs?00:32
clarkbso that would be a thin gwe would need to add to the multinode setup. nfs/ceph/bootfromvolume?00:32
jogoclarkb: AFAIK that isn't actually true00:32
mtreinishclarkb: iirc there are 2 ways to do, shared storage for the vm disks is the recommended00:33
jogoclarkb: but I have to double check00:33
mtreinishthere is also a libvirt mechanism to do it which is turned off by default because it's flaky00:33
mtreinishI think, that might just be out of date info00:33
mtreinishclarkb: if we need it, I'd probably just use nfs it would be the simplest to do I think00:34
clarkbexcept we already have cinder00:34
mtreinishclarkb: but can we have a volume shared between 2 servers?00:35
mtreinishI thought you could only have 1 attach at a time00:35
jogomtreinish: AFAIK we still need a test to do live migration of a boot from volume server00:35
clarkbiscsid runs on one host and serves the volume to the compute node00:35
mtreinishclarkb: ah, ok00:35
clarkbso its shared in that its all on the primary node00:36
clarkband you run off of iscsi00:36
jogoclarkb: I think the non shared storage use case is more interesting to test00:36
clarkbwhich may be terrible :)00:36
*** shashankhegde has quit IRC00:37
*** shashankhegde has joined #openstack-qa00:37
oomichi_mtreinish: yeah, I know. the code seems nice, but the commit message is long and difficult for me :-(00:38
oomichi_mtreinish: I prefer the code, +A00:38
* jogo reads http://docs.openstack.org/trunk/config-reference/content/section_configuring-compute-migrations.html00:39
clarkbjogo: how does the non shared stoarge migration work?00:40
*** gfidente has quit IRC00:40
mtreinishclarkb: IIRC it's a libvirt thing00:40
clarkbya so we can do the volume backed thing right now00:40
clarkbwhich is probably a reasonable place to start00:41
*** yamamoto has joined #openstack-qa00:41
mtreinishjogo, clarkb: http://docs.openstack.org/trunk/config-reference/content/section_configuring-compute-migrations.html#true-live-migration-kvm-libvirt00:42
mtreinishthere is also a third option if we were running xenserver00:42
mtreinishwhich looking at the name of the test is probably it's origin00:42
jogomtreinish: yeah looks like I need another nova.conf00:43
mtreinishclarkb: heh, "The Compute service does not use libvirt's live migration by default because there is a risk that the migration process never ends. This can happen if the guest operating system dirties blocks on the disk faster than they can migrated."00:43
jogogonna poke tripleo folks00:43
mtreinishyeah, I think shared storage is the way to go at first00:43
mtreinishwe don't need more flakiness in the gate00:44
clarkbno volume backed00:44
clarkbwe already have that today00:44
clarkband don't need to change anything00:44
clarkbjust boot a cirros node off of a cinder volume00:44
clarkbmigrate node00:44
mtreinishclarkb: that would be a new test case I think, where it's a boot from volume instance started00:45
mtreinishthen we run migrate on that00:45
clarkbsure, but it requires no changes to the underlying infrastructure00:46
clarkbI think new test case is easier than spin up nfs/ceph on nodes and make shared storage for libvirt00:46
mtreinishclarkb: true :)00:46
mtreinishjogo: ^^^ want to write it?00:46
clarkbbut I may be overestimating time to add that atop the multinode stuff00:46
jogoblock_migration_flag=VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_NON_SHARED_INC,VIR_MIGRATE_UNDEFINE_SOURCE00:46
jogois what tripleo tries00:46
jogoI am just gonna try that00:46
jogoand not the shared storage mode00:47
mtreinishjogo: yeah, there's a nice warning in the docs about intermittent failures around doing that00:48
mtreinishsaying it cause things to hang00:48
clarkbwhat coupld possibly go wrong :)00:48
clarkbthough with cirros its probably not a big problem00:48
jogomtreinish: do you remember how to add arbitrary nova.conf settings in devstack00:48
jogoclarkb: that is what I am thinking00:48
clarkbsince it runs not a lot of things to write to memory/disk00:48
mtreinishjogo: I think you put it in extras something00:50
mtreinishthere used to be a separate option for it but that went away00:50
mtreinishdtroyer: ^^^ are you around00:50
mtreinishjogo: I know we go out of our way not to do it in devstack-gate00:50
jogopost-config|$NOV_CONF00:50
jogofound it I think00:51
jogomtreinish: makes sense, but want to do it quick and dirty as  POC first00:51
clarkbya do it and see what happens00:51
mtreinishjogo: yeah I agree, I was just saying that because there won't be any examples there00:51
jogoalthough I can use clarkb's hack to include a different devstack branch00:51
*** hemna is now known as hemnafk00:52
mtreinishclarkb: what do you think: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2385&rra_id=all00:52
mtreinishstart of an un-ending growth curve?00:52
clarkbmtreinish: possibly00:52
clarkbmore time will tell00:52
jogohttp://paste.openstack.org/show/13499300:54
jogois the idea I am thinking of00:54
jogoput it over there so its at the bottom of the localrc file00:54
jogohttp://hackstack.org/x/blog/2013/09/07/devstack-local-config/00:54
jogofound a dtroyer example00:55
mtreinishjogo: heh, yeah that's probably right then :)00:55
openstackgerritYuiko Takada proposed openstack/tempest: Add Ironic driver API tests  https://review.openstack.org/12431300:58
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: Dirty hack to enable non shared storage live migration  https://review.openstack.org/13580200:58
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: Dirty hack to enable non shared storage live migration  https://review.openstack.org/13580201:00
*** suneelb has quit IRC01:01
jogoand running01:02
*** zz_dimtruck is now known as dimtruck01:02
*** yamamoto has quit IRC01:08
*** yamamoto has joined #openstack-qa01:09
*** yamamoto has quit IRC01:10
bknudsonok, time to look at why gate-tempest-dsvm-src-python-keystoneclient-icehouse fails for https://review.openstack.org/#/c/135127/01:12
bknudsonhttp://logs.openstack.org/27/135127/2/check/gate-tempest-dsvm-src-python-keystoneclient-icehouse/20f49d5/logs/devstacklog.txt.gz says01:12
bknudson2014-11-20 00:04:48.502 | 'suds-jurko' is not a global requirement but it should be,something went wrong01:12
bknudsonwhat does that mean.01:12
clarkbbknudson: because oslo.vmware didn't have a stbale branch or something01:12
clarkbits been fixed I am told01:12
mtreinishbknudson: https://bugs.launchpad.net/oslo.vmware/+bug/139440801:13
clarkbbut you were getting master of oslo.vmware which needed a req that wasn't in requirements01:13
clarkbjogo: your bash is bad01:13
* clarkb calls it a day with that :)01:13
bknudsonclarkb: thanks!01:14
mtreinishclarkb: oslo.vmware just added a new req which wasn't in icehouse global req. Icehouse devstack still installs it from git so icehouse devstack blew up because it wasn't in global-req on icehouse01:14
*** JayJ has quit IRC01:14
mtreinishI don't think libs should have stable branches01:15
bknudsonthe fix for https://bugs.launchpad.net/oslo.vmware/+bug/1394408 was merged Nov 12, and it's still failing01:17
bknudsonok, there's a stable/icehouse version too that merged not long ago... https://review.openstack.org/#/c/135692/01:18
mtreinishbknudson: jogo just pasted the wrong link in the bug01:18
mtreinishyeah that;s the right link01:18
*** dims has quit IRC01:24
*** dims has joined #openstack-qa01:24
*** achanda has quit IRC01:25
jogobknudson: woops thanks01:28
* jogo looks01:28
*** melwitt has quit IRC01:28
bknudsonlooks like all requirements changes in stable/juno are failing... 2014-11-18 07:05:46.642 | 'oslo.middleware' is not in global-requirements.txt01:29
* jogo watches tempest run yet again in https://jenkins03.openstack.org/job/check-tempest-dsvm-aiopcpu/8/console01:29
jogobknudson: oh that sounds fun01:29
jogomtreinish: we can't win01:29
mtreinishbknudson: I'm done for the night, I pass the torch to you... :)01:29
mtreinishjogo: 2014-11-20 01:30:32.308 | {3} tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestJSON.test_live_block_migration [5.363019s] ... FAILED01:31
jogovery quick again01:31
mtreinishyeah it probably means it failed fast on the live migration again01:31
jogomtreinish: yeah must be missing another conf option01:31
bknudsonhttps://review.openstack.org/#/c/135807/ is a change to add oslo.middleware to global-requirements.txt in stable/juno... see if it works.01:34
bknudsonoh, looks like https://review.openstack.org/#/c/134727/ was supposed to fix that problem.01:37
openstackgerritMerged openstack-dev/devstack: use a more common rst header hiearchy  https://review.openstack.org/13525201:45
openstackgerritMerged openstack/tempest: Move common compute clients to the setting method  https://review.openstack.org/13548201:45
*** tiantian has joined #openstack-qa01:46
openstackgerritKen'ichi Ohmichi proposed openstack/tempest: Add internal methods for setting volume clients  https://review.openstack.org/13548301:49
openstackgerritKen'ichi Ohmichi proposed openstack/tempest: Add internal methods for setting identity clients  https://review.openstack.org/13548401:49
*** JayJ has joined #openstack-qa01:51
openstackrecheckConsole logs not available after 14:12s for experimental-tempest-dsvm-full-icehouse-dibtest 135768,2,d80494e01:55
openstackrecheckopenstack/tempest change: https://review.openstack.org/135483 failed because of: gate-grenade-dsvm-partial-ncpu: https://bugs.launchpad.net/bugs/136252801:55
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Enable live block migration  https://review.openstack.org/13576801:56
*** yjiang5 is now known as yjiang5_away01:57
*** nosnos has joined #openstack-qa01:59
adam_gjogo, RE all this live migration stuff... you're attempting to get libvirt block migration working for the multi host ds setup?02:03
jogoadam_g: yup02:04
jogoadam_g: have you gotten it working somewhere already?02:04
adam_gjogo, i may be of some help, i did some work in the past automating the deployment of that with other things02:04
jogoadam_g: nice02:05
jogothis is where I am right now https://review.openstack.org/#/c/135768/302:05
jogosee my comments at the bottom02:05
jogohelp would be wonderful02:05
adam_gjogo, cool, ya. was just looking at that02:05
adam_gAFAICS devstack needs to be able to configure libvirtd to listen for connections and setup auth between hosts (either ssh, certificates, etc)02:06
adam_gunless thats been added recently?02:06
jogoadam_g: we set up ssh keys in devstack gate02:07
jogobut don't still need to do libvirtd stuff in devstack02:07
jogoadam_g: want to work on that?02:07
adam_gjogo, sure one sec02:07
openstackgerritSean Dague proposed openstack-dev/devstack: add unit tests for GIT* definitions  https://review.openstack.org/13571102:07
openstackgerritSean Dague proposed openstack-dev/devstack: WIP: test breaking unit tests  https://review.openstack.org/13581702:07
jogoadam_g: awesome!!02:07
openstackgerritMerged openstack-infra/elastic-recheck: Omit ironic agent_ssh jobs from 139099  https://review.openstack.org/13577002:08
*** dkranz has joined #openstack-qa02:19
*** openstackrecheck has quit IRC02:20
*** openstackrecheck has joined #openstack-qa02:20
openstackgerritAdam Gandelman proposed openstack-dev/devstack: Setup libvirt for ssh-based block migration  https://review.openstack.org/13582102:27
adam_gjogo, something like that, if memory serves ^02:27
jogoadam_g: you may be missing something as per http://www.sebastien-han.fr/blog/2012/07/12/openstack-block-migration/02:28
adam_gjogo, is there a TLDR for getting my own multi-host devstack up similar to how it is in the gate?02:29
jogoadam_g: not sure02:29
jogoadam_g: see devstack-gate logs02:29
jogoand open patches02:29
jogoclarkb: ^02:30
adam_gyeah02:30
adam_gfigured as much :)02:30
sdagueadam_g: not really yet, some of that should come back into devstack, and we should mirror it in devstack-vagrant. But it's still such a WIP the replication is ... not there yet02:30
jogonice tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestXML.test_live_block_migration [8.023399s] ... FAILED02:30
jogotook longer to fail this time02:30
clarkbya most of it isnt merged in any way yet either02:31
sdaguejogo: so... did you figure out if live migration is actually even possible on soft qemu?02:31
*** Longgeek has joined #openstack-qa02:33
*** X019 has quit IRC02:33
adam_gjogo, i think whats in that patch is enough, at least it gets me a devstack that i can 'virsh -c qemu+ssh://$foo/system'  to a devstack on $foo, which is all thats needed by libvirt for migration02:34
adam_gim not sure what user nova will attempt to use for ssh tho02:34
adam_gi seem to remember having to allow root logins across systems02:34
adam_gvia ssh keys with no passphrase02:34
jogosdague: yes, mikal told me so02:35
sdaguejogo: ... but has it worked?02:36
jogosdague: he said it worked, he tried it02:36
sdagueadam_g: for devstack is there a reason to make that optional?02:36
adam_gsdague, i suppose it could be the default, sure.02:39
*** Longgeek has quit IRC02:40
*** oomichi_ has quit IRC02:41
sdagueadam_g: yeh, seems like us just turning on live migration facilities in devstack is valid, because it's a dev env. It shouldn't hurt anything else, right?02:41
jogosdague: yeah that logic works for me02:41
adam_gsdague, yea02:43
sdagueeverything time you add a config var to devstack, it kills ... something cute02:43
sdaguesleep deprevation prevents me from figuring out a funny at the end of that02:44
mtreinishsdague: any idea on what the rough count of config options is in devstack?02:44
openstackgerritMerged openstack/tempest: Fix inheritance for BaseTestCase  https://review.openstack.org/13511502:44
sdaguecan I discount neutron?02:44
mtreinishlifeless: ^^^02:44
mtreinishsdague: heh, sure thats probably 2x everything02:45
openstackgerritMerged openstack/tempest: Enable cinder v2 api for volume type extra specs tests  https://review.openstack.org/13521102:45
openstackgerritMerged openstack/tempest: Enable v2 api for volume and snapshot tests  https://review.openstack.org/13519002:45
*** X019 has joined #openstack-qa02:46
sdagueyeh... too many02:46
*** dimtruck is now known as zz_dimtruck02:47
mtreinishsdague: so the tempest fix landed should I propose the unpin for testtool 1.4.0?02:47
sdaguemtreinish: sure, the lastest is still 1.4.0 right?02:47
sdagueso it will test if it works02:47
mtreinishsdague: yeah, 1.4.0 is still the latest02:48
sdaguego for it02:48
sdaguewe'll know in a few minutes if it was the right fix02:48
mtreinishlifeless said that 1.5.0 is pending and that will bring tracebacks on discovery failures02:48
openstackgerritAdam Gandelman proposed openstack-dev/devstack: Setup libvirt for ssh-based block migration  https://review.openstack.org/13582102:48
sdaguemtreinish: ok, but lets do this thing to test that it actually fixes the 1.4.0 issue02:49
lifelesso/02:50
lifelessyes 1.5.0 is pending02:50
lifelessI was trying to get rish tracebacks in02:50
lifelessbut I think it will take a few more days02:50
lifelessso I'll cut 1.5.0 this afternoon probably.02:50
mtreinishsdague: https://review.openstack.org/13582802:50
sdaguemtreinish: cool. If the tempest tests pass I'll +A it02:51
*** shashankhegde has quit IRC02:51
sdaguedtroyer: also, the devstack unit test now has a staleness unit test for the ALL_LIBS bit02:54
*** dkranz has quit IRC02:55
jogoadam_g: if that passes tests +1 from me02:56
openstackgerritEli Qiao proposed openstack/tempest: Add Conflict exception in test_floating_ips_actions case  https://review.openstack.org/13136402:58
jogomtreinish: so this isn't what I expected http://logs.openstack.org/68/135768/3/experimental/check-tempest-dsvm-aiopcpu/7b26557/logs/testr_results.html.gz03:00
jogoclarkb: ^03:00
jogoNoLiveMigrationForConfigDriveInLibVirt: Live migration of instances with config drives is not supported in libvirt unless libvirt instance path and drive data is shared across compute nodes.03:03
clarkbnice03:03
lifelessoh yeah03:03
lifelessuse metadata server03:03
lifelessits better03:03
clarkbtl;dr we should just not support live migration is what I am hearing >_>03:03
lifeless:)03:03
clarkbexcept for when it doesn't work03:03
clarkbwhich is almost always03:03
lifelesswat03:04
* clarkb shakes fist at being unable to boot centos nodes on hpcloud03:04
clarkblifeless: I cannot boot centos nodes in hpcloud at all with my account03:04
clarkbflat doesn't work becaues metadata server fails03:04
clarkbso no ssh key and things03:04
clarkbits fun03:04
lifelessin what way does it fail ?03:04
clarkbcloud init can't get my key03:04
jogohttp://logs.openstack.org/68/135768/3/experimental/check-tempest-dsvm-aiopcpu/7b26557/logs/10.208.167.5/screen-n-cpu.txt.gz?level=TRACE#_2014-11-20_02_29_09_74803:04
clarkbit times out talking to the metadata server over the 169 address03:04
lifelessclarkb: because centos is breaking 169.254.169.25403:04
clarkblifeless: no I think hpcloud is breaking it03:05
clarkbseen it on all OSes but centos is consistent03:05
mtreinishjogo: is there an aiopcpu postgres job?03:05
clarkbI have filed a couple bugs about it with hpcloud before I gave up03:05
lifelessclarkb: is your centos image dib built or ?03:05
clarkbno its hpcloud provided03:05
clarkbbut it failed on precise and trusty too03:05
clarkbjust not as reliably03:06
clarkb(again using the base images provided to me not dib)03:06
lifelessso03:06
jogomtreinish: no :/03:06
clarkbdib sort of solve sthis because you don't need metadata server with dib03:06
jogobut this is devstack-gate patches03:06
lifelessthere's a service, name escaping me rigt now03:06
clarkbsince ou start with an image prekeed03:06
jogoso we can change that there03:06
lifelesswhich mangles the 169.254.169.254 address03:06
lifelessroute03:06
mtreinishjogo: yeah flip that to postgres just to test if it works with the metadata server03:06
lifelessclarkb: do you get a route dump in your console?03:07
sdaguemtreinish: or just set the metadata server flag03:07
lifelessclarkb: found it03:07
mtreinishsdague: oh, yeah03:07
lifelessclarkb: http://docs.openstack.org/image-guide/content/centos-image.html03:07
lifelessDisable the zeroconf route03:08
lifelessFor the instance to access the metadata service, you must disable the default zeroconf route:03:08
*** dims has quit IRC03:08
lifeless# echo "NOZEROCONF=yes" >> /etc/sysconfig/network03:08
lifelessclarkb: worth checking that that is setup witin the image03:08
clarkblifeless: that requires getting on the node...03:08
clarkbwhich I cannot do...03:08
lifelessclarkb: you can't pull it out of glance?03:08
mtreinishjogo: I guess we'll need shared storage if we want to do migration with config drive03:08
clarkblifeless: maybe?03:08
clarkbin any case I don't care anymore03:08
*** dims has joined #openstack-qa03:08
clarkbI just use rax03:08
sdaguewhich is why metadata server is kind of bonghits, because it relies on much magic03:08
clarkbwhich doesn't metadata server03:08
lifelessmtreinish: or fix config-drive :)03:08
*** adalbas has quit IRC03:08
jogomtreinish: maybe we can get ceph set up03:09
jogomtreinish: as that is a common setup03:09
jogowhere we == not us but ceph folks03:09
sdaguejogo: you seriously don't want to have to bite that off with this03:09
lifelesssdague: mmm it has different dependencies03:09
mtreinishjogo: I would just do nfs I think it would be a lot easier03:09
jogosdague: as a follow up down the road03:09
mtreinishjogo: or you can try clarkb's iscsid cinder idea03:09
sdagueso, for this just turn on md03:09
lifelesssdague: but I don't think bonghits is a good summary - its in *extremely* widespread use, as I know you know.03:09
sdaguelifeless: sure :)03:11
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive  https://review.openstack.org/13583103:11
clarkbusing the iso mount thing (is that config drive?) with cloud init seems to be most reliable at least when I derp with things locally03:11
jogoclarkb: yeah iso is configdrive03:11
clarkbI kinda wish that was more common but I guess creating lots of little file sfor mounting is a pita03:11
sdagueclarkb: yeh, it's a ton more reliable, which is why we defaulted it03:11
jogook hacked off config drive03:12
*** krtaylor has quit IRC03:12
jogomtreinish: long run I think we need a live migrate test wtih and without shared file system03:13
jogobut that is long  term03:13
jogorerunning tempest with no config drive, lets see if this fails correctly :)03:14
clarkbjogo: so this is me being pragmatic. why do we need to support 3 different forms of live migration?03:14
jogoclarkb: because legacy03:14
jogoclarkb: and 3?03:14
clarkbseriously would it be such a terrible thing if we put our foot down and made things sane03:14
jogoclarkb: in short I agree nova supports too much03:14
clarkbjogo: shared fs, volume backed, libvirt block03:14
jogoclarkb: someone wants nova to support PXE  booting03:14
*** HenryG has quit IRC03:15
mtreinishclarkb: to be fair we don't have a volume backed test in tempest yet03:15
lifelessjogo: I'm one of those people03:15
jogoclarkb: and someone filed a bug saying volueme backed is broken03:15
sdagueok, train nearly in station. See folks on the flip side.03:15
jogolifeless: you want nova PXE boot?03:15
jogosdague: o/03:15
lifelessjogo: yes03:15
jogolifeless: why :(03:15
jogolifeless: don't we have enough ways to boot things03:16
lifelessjogo: so we can test a lot of tripleo in the regular gate vms03:16
jogolifeless: boot a image with iPXE on it or something03:16
lifelessjogo: erm, rephrease. So we can use VMs from the regular quota of cloud machines.03:16
lifelessjogo: thats a possibility. See the quintupleo spec :)03:17
jogolifeless: why not do this: http://kimizhang.wordpress.com/2013/08/26/create-pxe-boot-image-for-openstack/03:17
mtreinishclarkb: are you seeing any tracebacks in the subunit gearman worker log, I'm really surprised that a single worker is keeping up at all03:17
mtreinishmy sql queries suck and should be slower than this...03:17
jogomtreinish: haha when things appear to work you are suspicious03:17
clarkbmtreinish: I haven't looked. sort of afraid to this late in the day :)03:18
mtreinishclarkb: sure no worries03:18
lifelessjogo: well, we're going to then drop a differen timage on the machine03:18
lifelessjogo: and then reboot it03:18
lifelessjogo: and it should pxe boot again.03:18
*** HenryG has joined #openstack-qa03:18
lifelessjogo: Rackspace already support PXE booting03:18
lifelessjogo: so this is arguably just delta reduction :)03:18
jogolifeless:  there is some Pxe support in xen03:19
lifelessjogo: and kvm03:19
jogoI am fine with adding that support to  kvm03:19
lifelessjogo: or do you mean the driver?03:19
jogolifeless: yeah the nova drivers03:19
lifelessrighto03:19
lifelessso cool :)03:19
*** zz_dimtruck is now known as dimtruck03:19
jogobut that isn't what is being proposed right now03:19
lifelesswhats being proposed right now?03:20
jogolifeless: https://review.openstack.org/13325403:20
mtreinishjogo: I am suspicious, we let it run for a day w/ 4 workers it ate 16GB of mem/swap but kept up with everything03:20
jogowow not bad03:21
mtreinishwe dropped down to 1, the memory footprint has been sitting at <=.5gb and the gearman queue is still 003:21
mtreinishgranted the load has been light since we restarted things03:22
jogomtreinish: nice03:22
lifelessmtreinish: simple schema?03:23
jogoanyway my eyes are starting to glaze over, will continue the road to testing  live migration without shared storage tomorrow03:23
jogolatest job: https://jenkins05.openstack.org/job/check-tempest-dsvm-aiopcpu/7/console03:23
clarkbjogo: still expected to fail right?03:24
lifelessjogo: https://bugzilla.redhat.com/show_bug.cgi?id=112626803:24
*** nosnos has quit IRC03:24
lifelessjogo: mau be related03:24
mtreinishlifeless: more or less, but one table gets very large after a while which really slows down things03:25
lifelessjogo: I'm not sure why redhat bugzilla is cloning specs into bugs03:25
*** krtaylor has joined #openstack-qa03:25
*** dimtruck is now known as zz_dimtruck03:25
lifelessmtreinish: how many million rows?03:25
*** adalbas has joined #openstack-qa03:25
mtreinishI ran it for about 1.5 days locally and it hit ~5.5 million03:25
lifelesstiny03:26
*** yamahata has joined #openstack-qa03:26
mtreinishlifeless: its a row for each individual tempest test executed by ci03:26
mtreinishlifeless: heh, it's the biggest table I've worked with :)03:27
*** rushiagr_away is now known as rushiagr03:27
lifelessjogo: presume you've seen https://bugs.launchpad.net/nova/+bug/1183885 ?03:27
lifelessmtreinish: even sqlite can chew that up and spit it out sub-ms03:27
lifelessmtreinish: assuming appropriate queries and indices03:27
clarkblifeless: the problem is what it looks like after a dev cycle on a poor trove node03:28
clarkbwe will find out :)03:28
mtreinishlifeless: the 2nd half :)03:28
*** Longgeek has joined #openstack-qa03:36
openstackgerritMatthew Treinish proposed openstack/tempest: Unskip test_disassociate_not_associated_floating_ip()  https://review.openstack.org/12157003:39
openstackgerritMatthew Treinish proposed openstack/tempest: Unskip test_list_virtual_interfaces()  https://review.openstack.org/12157103:39
openstackgerritMatthew Treinish proposed openstack/tempest: Unskip test_get_server_diagnostics_by_admin()  https://review.openstack.org/12157203:39
openstackgerritMatthew Treinish proposed openstack/tempest: Unskip test_create_delete_server_group_with_multiple_policies()  https://review.openstack.org/12157403:40
jogoclarkb: yeah expecting to fail03:40
*** Longgeek has quit IRC03:41
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Flush out the db api documenation  https://review.openstack.org/12940503:42
jogolifeless: odd, that explains things03:42
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Reorganize the python api documentation  https://review.openstack.org/12940603:42
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add section on the basic data model  https://review.openstack.org/12940703:42
openstackgerritMatthew Treinish proposed openstack/tempest-lib: Add pretty_tox.sh wrapper script to use subunit-trace  https://review.openstack.org/12941903:43
jogolifeless: do you know how to find redhats plan via bugzilla?03:43
openstackgerritMatthew Treinish proposed openstack/tempest: Sync latest log module from oslo-incubator  https://review.openstack.org/13160103:44
jogooh hehe https://bugzilla.redhat.com/buglist.cgi?component=RFEs&product=Red%20Hat%20OpenStack03:44
jogothat doesn't look like a manually made list03:46
*** nosnos has joined #openstack-qa03:49
lifelessjogo: I just googled03:49
*** yamamoto has joined #openstack-qa03:53
*** rushiagr is now known as rushiagr_away03:53
*** jgrimm is now known as zz_jgrimm03:54
*** patrickeast has quit IRC04:05
*** pcrews has quit IRC04:16
openstackgerritMerged openstack/tempest: Add internal methods for setting volume clients  https://review.openstack.org/13548304:24
openstackgerritKen'ichi Ohmichi proposed openstack/tempest: Add internal methods for setting identity clients  https://review.openstack.org/13548404:26
*** zz_dimtruck is now known as dimtruck04:27
jgriffithjogo: clarkb I have to say I'm one of those people that probably don't agree WRT to live-migration restrictions04:29
jgriffithjogo: clarkb but we should talk some time over a virtual beer or cofee or whatever04:29
jgriffithjogo: clarkb honestly IMO we should just make it "work"04:30
jgriffithjogo: clarkb and screw the options/methods, just figure out what needs done and do it04:30
jgriffithjogo: clarkb and the number one question from customers as of late is in fact "WTF's up with live-migration in Nova"04:31
clarkbya I thibk it should work04:31
clarkband best way for that is to have one option04:32
clarkbthat works04:32
jgriffithclarkb: +100004:32
clarkb3 that dont work is useless04:32
jgriffithclarkb: +1000 * 10000000004:32
jgriffith:)04:32
jgriffithclarkb: setting aside the whole fact that it confuses the hell out of everybody the way it is now anyway :(04:32
*** rushiagr_away is now known as rushiagr04:33
clarkbya I was very confused earlier today04:33
jgriffithclarkb: Yay!!04:34
jgriffithclarkb: Ok, so next week when I get back and start poking it won't be a waste of time04:34
clarkbI have learned so much about openstack doing this multinode stuff :)04:35
jgriffithclarkb: Yeah, it's amazing how silo'd I was in my little devstack all in one world :)04:35
*** dimtruck is now known as zz_dimtruck04:37
*** Longgeek has joined #openstack-qa04:37
*** yamamoto has quit IRC04:39
*** Longgeek has quit IRC04:42
openstackgerritjun xie proposed openstack/tempest: enable cinder v2 api for test_volumes_backup  https://review.openstack.org/13517904:56
*** yamamoto has joined #openstack-qa05:06
*** shashankhegde has joined #openstack-qa05:13
openstackgerritMerged openstack/tempest: Add config options for XML support in Keystone  https://review.openstack.org/12656405:13
*** tiantian has quit IRC05:18
*** yfried has quit IRC05:27
*** achanda has joined #openstack-qa05:38
*** abehl has joined #openstack-qa05:41
openstackrecheckopenstack/cinder change: https://review.openstack.org/133409 failed gate-grenade-dsvm in the gate queue with an unrecognized error05:47
*** chandankumar has joined #openstack-qa05:52
*** bkopilov has joined #openstack-qa05:53
*** Longgeek has joined #openstack-qa06:01
*** Longgeek has quit IRC06:02
*** Longgeek has joined #openstack-qa06:03
*** tiantian has joined #openstack-qa06:11
*** bkopilov has quit IRC06:12
*** shashankhegde has quit IRC06:13
*** yfried has joined #openstack-qa06:17
*** oomichi has joined #openstack-qa06:22
openstackgerritjun xie proposed openstack/tempest: add cinder v2 api tests for volume_services  https://review.openstack.org/13586006:22
*** sridhar has joined #openstack-qa06:23
*** neeti_ has joined #openstack-qa06:27
yfriedsergsh_away: ping06:28
*** harlowja is now known as harlowja_away06:31
*** SridharG has joined #openstack-qa06:31
*** k4n0 has joined #openstack-qa06:33
*** yfried is now known as yfried|afk06:40
*** achanda has quit IRC06:45
*** yfried|afk is now known as yfried06:48
*** nmagnezi has joined #openstack-qa06:53
*** afazekas has quit IRC06:55
*** pcrews has joined #openstack-qa06:55
*** vigneshvar has joined #openstack-qa06:55
openstackgerritBharat Kumar Kobagana proposed openstack-dev/devstack: Deploy GlusterFS server  https://review.openstack.org/13310206:56
*** achanda has joined #openstack-qa06:58
*** achanda has quit IRC06:58
*** achanda has joined #openstack-qa06:59
*** oomichi__ has joined #openstack-qa07:01
*** oomichi has quit IRC07:02
*** ashishg has joined #openstack-qa07:11
*** bkopilov has joined #openstack-qa07:12
*** bkopilov has quit IRC07:13
openstackgerritMerged openstack/tempest: Add internal methods for setting identity clients  https://review.openstack.org/13548407:15
*** pcrews has quit IRC07:16
jogojgriffith: we can do real beer and virtual talk07:20
*** mpaolino has joined #openstack-qa07:22
jogojgriffith clarkb: I agree nova API is terribad07:22
*** neeti_ has quit IRC07:22
jogowell openstack APIs07:22
jogowoot http://logs.openstack.org/31/135831/1/experimental/check-tempest-dsvm-aiopcpu/1109b15/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-11-20_03_49_01_64807:23
jogothat is the failure that I was looking for07:24
*** achanda has quit IRC07:25
*** achanda has joined #openstack-qa07:25
*** achanda has quit IRC07:25
*** ajo has joined #openstack-qa07:28
jogoclarkb: yeah we need a devstack-gate patch for host names libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://devstack-trusty-2-node-rax-dfw-3143753-382.slave.openstack.org/system: Unable to resolve address 'devstack-trusty-2-node-rax-dfw-3143753-382.slave.openstack.org' service '16509': Name or service not known07:29
jogothat is fun07:30
*** e0ne has joined #openstack-qa07:30
*** neeti has joined #openstack-qa07:41
openstackrecheckopenstack/nova change: https://review.openstack.org/123459 failed gate-nova-pep8 in the gate queue with an unrecognized error07:41
*** bkopilov has joined #openstack-qa07:42
*** afazekas has joined #openstack-qa07:45
yfriedoomichi__: ping07:50
*** jpich has joined #openstack-qa07:58
*** markus_z has joined #openstack-qa08:02
*** sergsh_away is now known as sergsh08:08
*** oomichi_ has joined #openstack-qa08:12
*** oomichi__ has quit IRC08:12
sergshyfried: pong08:13
yfriedsergsh: about https://review.openstack.org/#/c/135571/08:14
*** oomichi_ has quit IRC08:14
sergshyeah08:14
yfriedsergsh: you suggested the flag version=Icehose, right?08:14
*** gszasz has joined #openstack-qa08:14
sergshactually I suggested icehouse=False, but your option is better :)08:15
*** b3nt_pin has quit IRC08:15
yfriedsergsh: so, where so you think I can push this flag? it doesn't fit on any of the sections.08:16
*** beagles has joined #openstack-qa08:16
sergshthe "default",I think08:16
*** beagles is now known as Guest2782508:16
sergshbecause it's related to whole tempest testing08:17
yfriedsergsh: how do I edit the default? couldn't find it in config.py. maybe I've missed something?08:18
openstackgerritAdam Gandelman proposed openstack-dev/devstack: Setup libvirt for ssh-based block migration  https://review.openstack.org/13582108:18
openstackgerrithuangtianhua proposed openstack/tempest: Take InstanceType for launchconfig res creation  https://review.openstack.org/13551908:18
sergshyfried: hmm.. seems like this is populated in TempestConfigPrivate as cfg.CONF.set_default08:21
sergshyfried, then maybe service_available_group? or create another group?08:23
*** yamamoto has quit IRC08:29
yfriedsergsh: I don't want to create a whole new group for that08:29
sergshyfried, was there before such problem? it's weird that only now we met such restriction08:31
sergshi'd suggest service_available_group then, but I'd agree any solution :)08:32
yfriedsergsh: what do you mean "such problem"?08:33
sergshof tests that should not run/pass on specific releases08:33
sergshbecause not all changes are backported to previous releases, if I understand right..08:34
*** jlanoux has joined #openstack-qa08:38
*** luqas has joined #openstack-qa08:38
yfriedsergsh: usually this is due to a new feature or change in API so we add a flag (like IPv6 enabled). but this is a bug-fix/behavior-change of regular api08:40
*** avico has joined #openstack-qa08:43
sergshyfried, I see, but it's good idea anyway to have such parameter08:44
yfriedsergsh: not sure, since tempest's supposed to version-less08:45
*** dmellado has joined #openstack-qa08:50
*** sridhar has quit IRC08:54
*** jordanP has joined #openstack-qa08:58
*** yamamoto has joined #openstack-qa08:58
*** safchain has joined #openstack-qa08:59
*** pkoniszewski has joined #openstack-qa09:02
openstackgerritchandan kumar proposed openstack/tempest: Enable cinder v2 api tests in volume quotas  https://review.openstack.org/13488909:12
*** pkoniszewski has quit IRC09:13
*** yfried is now known as yfried|afk09:14
*** yfried|afk is now known as yfried09:15
*** e0ne has quit IRC09:17
*** sridhar has joined #openstack-qa09:20
*** Guest27825 has quit IRC09:27
*** oomichi has joined #openstack-qa09:27
yfriedoomichi: here?09:27
oomichiyfried: hi09:28
yfriedoomichi: sergsh in myself were discussing the skip flag from yesterday09:28
yfriedoomichi: you guys seem to like version=icehouse09:29
oomichiyfried: yes, that seems common way. and maybe we will be able to use it for the other purpose in the future.09:29
*** mwagner_lap has quit IRC09:30
yfriedoomichi: were would you suggest to push it? I can only think of section [default] but I don't know how to add stuff to that09:30
oomichiyfried: OK, I will write it on the gerrit.09:31
*** Longgeek has quit IRC09:32
*** beagles has joined #openstack-qa09:32
*** beagles is now known as b3nt_pin09:33
*** vponomaryov has quit IRC09:34
*** vponomaryov has joined #openstack-qa09:36
oomichiyfried: done09:37
*** yamamoto has quit IRC09:39
*** Longgeek has joined #openstack-qa09:41
*** pkoniszewski has joined #openstack-qa09:44
sergshoomichi: maybe it's better to add "max_version" keyword? and then set it to "icehouse", "juno", etc09:44
*** yfried is now known as yfried|afk09:45
*** JayJ has quit IRC09:50
oomichisergsh: that is an idea, but if some patch are merged icehouse before juno, it cannot cover it. it seems a little overkill for current situation.09:51
*** JayJ has joined #openstack-qa09:51
*** mpaolino has quit IRC09:51
*** mpaolino has joined #openstack-qa09:52
*** pkoniszewski has quit IRC09:53
*** yfried|afk is now known as yfried09:53
openstackgerritKen'ichi Ohmichi proposed openstack/tempest: WIP: Add an option for skipping all XML tests  https://review.openstack.org/13166509:54
openstackgerritKen'ichi Ohmichi proposed openstack/tempest: Move API clients to better places  https://review.openstack.org/13589509:54
openstackgerritSergey Shnaidman proposed openstack/qa-specs: Add tests for Neutron extensions  https://review.openstack.org/13589609:55
yfriedoomichi: so you want the flag to be CONF.network.version? seems like it should be in CONF.default.version09:56
yfriedoomichi: if we want it to be used by other tests as well later09:56
openstackgerritSergey Shnaidman proposed openstack/qa-specs: Add tests for Neutron extensions  https://review.openstack.org/13589609:57
*** JayJ has quit IRC09:57
*** zz_johnthetubagu is now known as johnthetubaguy09:57
sergshPlease, take a look at blueprint: https://review.openstack.org/#/c/135896/209:58
*** JayJ has joined #openstack-qa09:58
oomichiyfried: that is a nice idea.09:58
openstackgerritjun xie proposed openstack/tempest: add cinder v2 api tests for volume_services  https://review.openstack.org/13586009:59
openstackgerritAshish Kumar Gupta proposed openstack/tempest: Security Groups  API Tests Enhancements  https://review.openstack.org/10765410:00
openstackgerritYuiko Takada proposed openstack-dev/devstack: add the kernel/ramdisk id to driver_info  https://review.openstack.org/13589710:00
openstackrecheckopenstack/glance change: https://review.openstack.org/120970 failed because of: gate-glance-docs: https://bugs.launchpad.net/bugs/1298006, gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/135396210:01
openstackrecheckopenstack/horizon change: https://review.openstack.org/135226 failed gate-tempest-dsvm-full, gate-tempest-dsvm-postgres-full, gate-tempest-dsvm-neutron, gate-tempest-dsvm-neutron-heat-slow, check-devstack-dsvm-cells in the gate queue with an unrecognized error10:01
*** om has joined #openstack-qa10:01
*** JayJ has quit IRC10:02
*** JayJ has joined #openstack-qa10:03
*** e0ne has joined #openstack-qa10:06
*** JayJ has quit IRC10:07
*** JayJ has joined #openstack-qa10:08
*** aix has joined #openstack-qa10:08
openstackgerritAshish Kumar Gupta proposed openstack/tempest: Port  API Test Enhancement  https://review.openstack.org/13589810:08
*** jlibosva has joined #openstack-qa10:11
*** mpaolino has quit IRC10:12
*** JayJ has quit IRC10:12
openstackgerritjun xie proposed openstack/tempest: add cinder v2 api tests for volume_services  https://review.openstack.org/13586010:12
*** JayJ has joined #openstack-qa10:13
*** mpaolino has joined #openstack-qa10:17
*** salv-orlando_ has joined #openstack-qa10:17
*** JayJ has quit IRC10:18
openstackgerritRajkumar proposed openstack/tempest: Add Test to Create Port with no security groups  https://review.openstack.org/11590810:18
*** JayJ has joined #openstack-qa10:18
*** salv-orlando has quit IRC10:19
*** salv-orlando_ is now known as salv-orlando10:19
*** JayJ has quit IRC10:23
*** JayJ has joined #openstack-qa10:23
*** raja_vedams has joined #openstack-qa10:26
raja_vedamshow hot swap a volume10:27
*** JayJ has quit IRC10:28
yfriedoomichi: so how do I add stuff to conf.default10:28
*** JayJ has joined #openstack-qa10:28
raja_vedamshi JauJ10:29
*** jordanP has quit IRC10:31
*** JayJ has quit IRC10:33
*** raja_vedams has quit IRC10:33
*** JayJ has joined #openstack-qa10:33
oomichiyfied: that is what I want to know now ;-) one trial is https://review.openstack.org/#/c/131665/, but I'm not sure that is right way..10:34
*** JayJ has quit IRC10:38
*** JayJ has joined #openstack-qa10:38
*** yamamoto has joined #openstack-qa10:39
*** jordanP has joined #openstack-qa10:39
*** JayJ has quit IRC10:43
*** chandankumar has quit IRC10:43
*** JayJ has joined #openstack-qa10:44
*** yamamoto has quit IRC10:44
*** JayJ has quit IRC10:48
*** JayJ has joined #openstack-qa10:49
*** chandankumar has joined #openstack-qa10:50
*** yfried is now known as yfried|afk10:52
*** JayJ has quit IRC10:53
*** JayJ has joined #openstack-qa10:54
*** aalekov has joined #openstack-qa10:58
*** JayJ has quit IRC10:58
*** tosky has joined #openstack-qa10:58
*** JayJ has joined #openstack-qa10:59
*** dims has quit IRC11:01
*** dims has joined #openstack-qa11:01
*** chandankumar has quit IRC11:03
*** JayJ has quit IRC11:03
*** JayJ has joined #openstack-qa11:04
*** chandankumar has joined #openstack-qa11:05
openstackgerritJoseph Lanoux proposed openstack/tempest: Add ServerListMultiStatus test  https://review.openstack.org/13522911:05
*** zz_jgrimm is now known as jgrimm11:06
openstackgerritKen'ichi Ohmichi proposed openstack/tempest: WIP: Add an option for skipping all XML tests  https://review.openstack.org/13166511:07
openstackgerritHugh Saunders proposed openstack/tempest: Add timeout for image operations This patch adds a seperate configuration option for image operations.  https://review.openstack.org/13591111:08
openstackrecheckConsole logs not available after 13:51s for gate-tempest-python27 131665,5,5fa39d411:08
*** JayJ has quit IRC11:08
*** JayJ has joined #openstack-qa11:09
*** tkammer has joined #openstack-qa11:10
*** rfolco has joined #openstack-qa11:11
*** aix has quit IRC11:11
*** yfried|afk is now known as yfried11:13
*** JayJ has quit IRC11:14
*** JayJ has joined #openstack-qa11:14
*** asselin has quit IRC11:16
*** JayJ has quit IRC11:19
*** JayJ has joined #openstack-qa11:19
*** yfried is now known as yfried|afk11:23
*** JayJ has quit IRC11:24
*** JayJ has joined #openstack-qa11:24
*** yamahata has quit IRC11:26
*** rfolco has quit IRC11:27
*** yfried|afk is now known as yfried11:27
*** rfolco has joined #openstack-qa11:28
*** adalbas has quit IRC11:28
*** yamahata has joined #openstack-qa11:28
*** JayJ has quit IRC11:29
*** JayJ has joined #openstack-qa11:29
*** tiantian has quit IRC11:30
*** JayJ has quit IRC11:34
*** JayJ has joined #openstack-qa11:34
*** flwang has quit IRC11:36
*** Longgeek has quit IRC11:39
*** JayJ has quit IRC11:39
*** JayJ has joined #openstack-qa11:40
*** aix has joined #openstack-qa11:42
*** flwang has joined #openstack-qa11:43
*** JayJ has quit IRC11:44
*** JayJ has joined #openstack-qa11:45
openstackgerritUdi Kalifon proposed openstack/tempest: Adding support in the client to create users with a default_project_id. Also adding a test to make sure that such users automatically get a token that is scoped to their project - even when requesting an unscoped token.  https://review.openstack.org/13592211:46
ashishgwhy jenkins server is failing for tempest test cases ?11:48
*** JayJ has quit IRC11:49
*** JayJ has joined #openstack-qa11:50
openstackgerritAbhishek G M proposed openstack/tempest: Test to verify  shared attribute of network  https://review.openstack.org/12499811:53
openstackgerritjun xie proposed openstack/tempest: add cinder v2 api tests for volume_services  https://review.openstack.org/13586011:54
*** nfedotov has joined #openstack-qa11:54
*** JayJ has quit IRC11:54
*** JayJ has joined #openstack-qa11:55
*** mpaolino has quit IRC12:05
openstackgerritchandan kumar proposed openstack/tempest: Enable cinder v2 api tests in volume quotas  https://review.openstack.org/13488912:06
chandankumargmann, jordanP afazekas https://review.openstack.org/134889 Please have a look12:07
jordanPchandankumar, I'd rather wait for jenkins :)12:09
*** rushiagr is now known as rushiagr_away12:15
openstackgerritSalvatore Orlando proposed openstack-infra/devstack-gate: Add support for enabling/disabling service extensions  https://review.openstack.org/11612912:15
*** Longgeek has joined #openstack-qa12:19
*** dmorita has quit IRC12:23
*** adalbas has joined #openstack-qa12:25
*** oomichi has quit IRC12:31
sergshafazekas, sdague: please, take a look in your time: https://review.openstack.org/#/c/134007/12:34
*** chandankumar has quit IRC12:37
*** sergsh is now known as sergsh_away12:38
*** yamamoto has joined #openstack-qa12:41
openstackgerritKen'ichi Ohmichi proposed openstack/tempest: WIP: Add an option for skipping all XML tests  https://review.openstack.org/13166512:41
*** rushiagr_away is now known as rushiagr12:46
*** yamamoto has quit IRC12:47
*** nmagnezi has quit IRC12:49
*** neeti has quit IRC12:52
openstackrecheckopenstack/horizon change: https://review.openstack.org/135251 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/137351312:53
*** tkammer has quit IRC12:58
*** nmagnezi has joined #openstack-qa13:02
*** tosky has quit IRC13:04
*** jhenner has quit IRC13:04
*** gszasz has quit IRC13:05
*** nosnos has quit IRC13:05
*** gfidente has joined #openstack-qa13:06
*** nosnos has joined #openstack-qa13:06
openstackgerritHugh Saunders proposed openstack/tempest: Add timeout for image operations  https://review.openstack.org/13591113:07
*** nmagnezi has quit IRC13:08
*** jhenner has joined #openstack-qa13:08
*** gszasz has joined #openstack-qa13:12
*** afazekas has quit IRC13:13
*** X019 has quit IRC13:14
*** tkammer has joined #openstack-qa13:14
*** pkoniszewski has joined #openstack-qa13:15
openstackgerritLouis Taylor proposed openstack-dev/devstack: Install bash completion for glanceclient  https://review.openstack.org/13594113:16
*** tosky has joined #openstack-qa13:19
*** vigneshvar has quit IRC13:21
*** zz_dimtruck is now known as dimtruck13:21
*** pkoniszewski has quit IRC13:22
*** nmagnezi has joined #openstack-qa13:23
*** markus_z has quit IRC13:25
*** afazekas has joined #openstack-qa13:29
openstackgerritSean Dague proposed openstack-dev/devstack: WIP: test breaking unit tests  https://review.openstack.org/13581713:32
openstackgerritSean Dague proposed openstack-dev/devstack: add unit tests for GIT* definitions  https://review.openstack.org/13571113:32
openstackgerritSean Dague proposed openstack-dev/devstack: add tool for testing project defs  https://review.openstack.org/13594813:32
*** dkranz has joined #openstack-qa13:33
*** k4n0 has quit IRC13:38
*** afazekas has quit IRC13:39
*** bkopilov has quit IRC13:41
openstackgerritHugh Saunders proposed openstack/tempest: Add timeout for image operations  https://review.openstack.org/13591113:43
*** salv-orlando has quit IRC13:47
*** rfolco has quit IRC13:48
*** om has quit IRC13:56
*** yfried has quit IRC13:56
*** mpaolino has joined #openstack-qa13:57
*** abehl has quit IRC13:57
*** weshay has joined #openstack-qa13:58
*** rushiagr is now known as rushiagr_away13:59
*** tkammer has quit IRC14:02
*** rushiagr_away is now known as rushiagr14:07
*** mkollaro has joined #openstack-qa14:11
*** ashishg has quit IRC14:13
*** sergsh_ has joined #openstack-qa14:13
*** mriedem has joined #openstack-qa14:13
*** tkammer has joined #openstack-qa14:14
*** afazekas has joined #openstack-qa14:15
*** rfolco has joined #openstack-qa14:15
*** jhenner has quit IRC14:16
*** dustins has joined #openstack-qa14:19
*** jhenner has joined #openstack-qa14:30
*** rfolco has quit IRC14:32
*** nosnos has quit IRC14:34
*** masayukig has quit IRC14:41
*** toure has joined #openstack-qa14:41
*** jecarey has joined #openstack-qa14:44
*** masayukig has joined #openstack-qa14:45
*** toure has quit IRC14:46
*** toure has joined #openstack-qa14:47
*** yamamoto has joined #openstack-qa14:48
*** markmcclain has joined #openstack-qa14:57
*** salv-orlando has joined #openstack-qa14:57
*** yamahata has quit IRC14:59
*** yamahata has joined #openstack-qa15:00
*** markmcclain1 has joined #openstack-qa15:00
*** markmcclain has quit IRC15:01
*** asselin has joined #openstack-qa15:01
*** nmagnezi has quit IRC15:02
*** markmcclain1 has quit IRC15:05
*** markmcclain has joined #openstack-qa15:05
svashekahi15:10
svashekais there separate ostf irc channel?15:10
*** pcrews has joined #openstack-qa15:17
*** fmccarthy has joined #openstack-qa15:18
openstackgerritAlok Kumar Maurya proposed openstack/tempest: Basic networks Scenario Test Enhancements  https://review.openstack.org/10662015:19
*** yamamoto_ has joined #openstack-qa15:20
*** JayJ has quit IRC15:21
*** bkopilov has joined #openstack-qa15:21
*** yamamoto has quit IRC15:21
*** dustins_ has joined #openstack-qa15:30
*** rbak has joined #openstack-qa15:31
*** dustins has quit IRC15:33
*** dpaterson has joined #openstack-qa15:34
*** aalekov has quit IRC15:34
*** jgrimm is now known as zz_jgrimm15:36
*** X019 has joined #openstack-qa15:37
*** rbak_ has joined #openstack-qa15:42
*** markmcclain has quit IRC15:43
openstackgerritprabhu murthy proposed openstack/tempest: Added neutron cli test case for fwaas  https://review.openstack.org/12363315:44
*** rbak has quit IRC15:45
*** markmcclain has joined #openstack-qa15:46
*** markmcclain has quit IRC15:46
*** markmcclain has joined #openstack-qa15:47
*** jordanP has quit IRC15:48
*** JayJ has joined #openstack-qa15:50
*** ajo has quit IRC15:57
*** jlibosva has quit IRC15:57
*** jlibosva has joined #openstack-qa15:59
*** ajo has joined #openstack-qa15:59
openstackgerritNikita Koshikov proposed openstack-dev/devstack: Coraid cinder driver added  https://review.openstack.org/13603416:00
*** dimtruck is now known as zz_dimtruck16:04
*** SridharG has left #openstack-qa16:04
*** marun has joined #openstack-qa16:05
*** zz_dimtruck is now known as dimtruck16:06
*** sridhar has quit IRC16:07
openstackgerritSean M. Collins proposed openstack-dev/devstack: Documentation: Using Neutron with DevStack  https://review.openstack.org/13120116:08
*** gszasz has quit IRC16:21
*** rbak_ has quit IRC16:22
*** rbak_ has joined #openstack-qa16:23
*** vigneshvar has joined #openstack-qa16:24
*** afazekas has quit IRC16:28
*** zz_jgrimm is now known as jgrimm16:29
openstackgerritSean M. Collins proposed openstack-dev/devstack: Documentation: Using Neutron with DevStack  https://review.openstack.org/13120116:31
*** dustins_ has quit IRC16:35
*** dustins has joined #openstack-qa16:36
*** dustins has quit IRC16:37
*** dmellado has quit IRC16:41
*** jordanP has joined #openstack-qa16:42
openstackgerritMerged openstack/tempest: Support for IPv6 tests for API  https://review.openstack.org/13400716:42
*** Longgeek has quit IRC16:45
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Add query for wrapt stable global reqs issues  https://review.openstack.org/13604916:45
openstackgerritFeodor Tersin proposed openstack/tempest: Use standard assertVolumeStatusWait for volume detachment  https://review.openstack.org/13026216:46
*** mlavalle has joined #openstack-qa16:52
*** rfolco has joined #openstack-qa16:54
sdaguemtreinish: leaking ip addresses - https://jenkins04.openstack.org/job/gate-tempest-dsvm-full/15755/console ?16:54
openstackrecheckopenstack/nova change: https://review.openstack.org/134068 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/1353962, gate-tempest-dsvm-neutron-full: unrecognized error16:55
sdagueactually, that's probably leaking computes16:55
mtreinishsdague: hmm, yeah it's probably servers. I think mriedem was looking at a similar quota issue before16:55
*** anna_ has joined #openstack-qa16:57
*** shashankhegde has joined #openstack-qa16:58
openstackgerritJoseph Lanoux proposed openstack/tempest: Add ServerUsage test  https://review.openstack.org/13522216:59
mriedemfixed ip quotas is a known race bug16:59
*** X019 has quit IRC16:59
mriedemsdague: http://status.openstack.org/elastic-recheck/gate.html#135396217:00
mriedemit's an oddball case where the fixed_ips quota is 99 and it can't allocate anything17:00
*** yfried has joined #openstack-qa17:01
sdaguemriedem: is that the delayed dealocation case?17:01
mriedemby default fixed_ips quota is -1 (it's the only one that is unlimited by default), then we must be hitting that default quota class test that bumps all default quotas + 10017:01
mriedemnot sure what the delayed deallocation case is,17:01
mriedemjogo and i were thinking it was some goofy logic in the quotas code with how it handles per-project quotas and unlimited quotas17:01
*** atiwari has joined #openstack-qa17:02
mriedemas in not handling them correctly17:02
sdagueit looked like quotas are just super sloppy, then there is a throw hands up and just let the periodic jobs recalc last time I looked17:02
mriedemwe got additional logging in nova-network when we hit this and it doesn't make much sense17:02
*** aalekov has joined #openstack-qa17:02
mriedemlet me find the log message17:02
mriedemit's weird17:02
sdaguereading all the quota bugs.... quotas are just massively broken in accounting17:03
mriedemthe quotas code is definitely scary17:03
mriedemthis http://logs.openstack.org/68/134068/2/gate/gate-tempest-dsvm-full/72da861/logs/screen-n-net.txt.gz?level=TRACE#_2014-11-20_15_12_42_98317:03
mriedem0 of 99 are in use or are already reserved.17:03
mriedemwtf?!17:03
sdagueqa meeting17:04
mriedembut i haven't gotten back into that bug since adding that logging and refactoring some of the db api quota_reserve() code so it's actually readable/unit-testable17:04
sdaguedkranz ?17:04
*** markmcclain has quit IRC17:06
*** jlanoux has quit IRC17:07
openstackgerritJordan Pittier proposed openstack-dev/devstack: Remove unused and deprecated SWIFT_CONFIG_DIR variable  https://review.openstack.org/12692417:07
*** avico has quit IRC17:07
*** yamamoto_ has quit IRC17:09
*** rfolco has quit IRC17:10
*** markmcclain has joined #openstack-qa17:11
openstackgerritMerged openstack-infra/elastic-recheck: Add query for wrapt stable global reqs issues  https://review.openstack.org/13604917:12
sdaguedtroyer: test results are back on https://review.openstack.org/#/c/135711/ if you want to +A17:14
*** markmcclain has quit IRC17:15
dtroyersdague: cool, done17:15
*** yamahata has quit IRC17:16
*** sabeen2 has quit IRC17:19
*** sabeen1 has joined #openstack-qa17:20
*** dkranz has quit IRC17:23
*** Viswanath has joined #openstack-qa17:24
*** safchain has quit IRC17:25
*** jpich has quit IRC17:25
*** anna_ is now known as X01917:25
*** shashankhegde has quit IRC17:27
*** openstackrecheck has quit IRC17:27
*** openstackrecheck has joined #openstack-qa17:27
*** Viswanath has quit IRC17:27
*** andreykurilin_ has joined #openstack-qa17:30
andreykurilin_Hi! I'll be happy if somebody review this spec - https://review.openstack.org/#/c/94473/ :)17:32
*** dimtruck is now known as zz_dimtruck17:35
*** dkranz has joined #openstack-qa17:36
*** aix has quit IRC17:51
mkoderersdague: could you give me a hint ... are you refering to https://pypi.python.org/pypi/testresources in https://review.openstack.org/#/c/134472/?17:52
sdaguemkoderer: yeh17:52
*** yfried has quit IRC17:53
mkoderersdague: I saw that we have this lib in requirments.txt... but it is acutally used somewhere?17:53
mtreinishmkoderer: it's not there anymore17:53
mtreinishit got removed yesterday (because nothing was actually using it)17:53
*** dustins has joined #openstack-qa17:54
mtreinishbut you can add it back if we start using it17:54
mkoderermtreinish: hehe ok... ;)17:54
*** aalekov has quit IRC17:55
mkoderermtreinish: I will give it a try... are some other project using it?17:55
mtreinishmkoderer: I'm not sure, I don't think so. oslo.db I think has patches up to start using it17:56
mtreinishsdague: ^^^ you were looking for examples the other day, did you find any?17:57
openstackrecheckopenstack/cinder change: https://review.openstack.org/131886 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/139464717:57
mkoderermtreinish: ok so again pypi beta tester ;)17:57
sdaguethere is an oslo db patch in review17:58
sdaguehonestly, it's a monday thing for me to try to dive into17:58
*** achanda has joined #openstack-qa17:59
mkoderersdague: ok, if I have time I will have a look tomorrow17:59
*** harlowja_away is now known as harlowja18:02
*** jlibosva has quit IRC18:03
*** shashankhegde has joined #openstack-qa18:03
*** e0ne has quit IRC18:03
*** suneelb has joined #openstack-qa18:04
*** rfolco has joined #openstack-qa18:05
*** suneelb has quit IRC18:07
adam_gsdague, i was hoping to flush out this stack of patches for devstack+diskimage-builder. are you still -1 on this? https://review.openstack.org/#/c/122539/18:07
*** suneelb has joined #openstack-qa18:07
*** gfidente has quit IRC18:08
*** yamamoto has joined #openstack-qa18:10
*** mriedem has quit IRC18:10
sdagueadam_g: so, is there a spec for this? Because I feel like I don't have the whole picture in my head about where it's going. In the narrow, it seems weird to add it there.18:10
*** krtaylor has quit IRC18:13
*** melwitt has joined #openstack-qa18:13
adam_gsdague, there isn't a spec but i can put up a short one explaining. in short, that will enable DIB to be installed and used via facilities lib/dib, which (among other things) will allow it to be installed from release/git (like client libs) and point DIB at non-internet resources for image builds18:14
*** yamamoto has quit IRC18:14
*** jordanP has quit IRC18:15
sdagueadam_g: right, so I think all those later things are goodness, I just think it's a thing we can do all the time instead of by flipping a service18:15
*** krtaylor has joined #openstack-qa18:16
sdagueanyway, yeh, sketch down the full story so I can see it, just so we make sure it integrates cleanly18:16
*** mriedem has joined #openstack-qa18:17
*** patrickeast has joined #openstack-qa18:17
adam_gsure18:17
*** rfolco has quit IRC18:22
*** wuhg has quit IRC18:23
*** rbak_ has quit IRC18:31
*** tkammer has quit IRC18:34
*** dkranz has quit IRC18:35
*** rbak has joined #openstack-qa18:35
*** sergsh_ has quit IRC18:37
*** mkollaro has quit IRC18:39
*** markmcclain has joined #openstack-qa18:43
*** rushiagr is now known as rushiagr_away18:45
*** markmcclain has quit IRC18:45
*** suneelb has quit IRC18:47
*** suneelb has joined #openstack-qa18:47
*** markmcclain has joined #openstack-qa18:48
*** dkranz has joined #openstack-qa18:48
*** mpaolino has quit IRC18:48
*** sergsh_ has joined #openstack-qa18:49
*** andreykurilin_ has quit IRC18:52
*** johnthetubaguy is now known as zz_johnthetubagu18:52
*** mkollaro has joined #openstack-qa18:53
*** weshay has quit IRC18:55
*** markmcclain1 has joined #openstack-qa18:56
*** markmcclain has quit IRC18:57
*** Viswanath has joined #openstack-qa19:00
*** Viswanath has quit IRC19:07
*** e0ne has joined #openstack-qa19:09
openstackrecheckopenstack/horizon change: https://review.openstack.org/135962 failed because of: check-devstack-dsvm-cells: https://bugs.launchpad.net/bugs/1365046, gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1365046, gate-tempest-dsvm-postgres-full: https://bugs.launchpad.net/bugs/136504619:11
*** salv-orlando has quit IRC19:12
*** weshay has joined #openstack-qa19:13
*** patrickeast_ has joined #openstack-qa19:16
*** patrickeast has quit IRC19:17
*** patrickeast_ is now known as patrickeast19:17
*** zz_dimtruck is now known as dimtruck19:19
mriedeme-r uncategorized bug % is getting hurt by this super generic issue http://logs.openstack.org/72/123972/2/gate/gate-tempest-dsvm-neutron-heat-slow/be87c42/console.html#_2014-11-17_23_01_33_07819:20
mriedemit's not restricted to heat though19:20
*** vigneshvar has quit IRC19:23
*** Sukhdev has joined #openstack-qa19:24
openstackgerritAdam Gandelman proposed openstack/qa-specs: Improve diskimage-builder in devstack  https://review.openstack.org/13610119:26
adam_gsdague, there ya'r19:27
*** tosky has quit IRC19:29
sdagueadam_g: thanks19:31
*** rwsu has quit IRC19:32
*** rwsu has joined #openstack-qa19:33
*** vigneshvar has joined #openstack-qa19:35
*** e0ne has quit IRC19:37
*** krtaylor has quit IRC19:45
*** tkammer has joined #openstack-qa19:49
*** mpaolino has joined #openstack-qa19:56
*** krtaylor has joined #openstack-qa19:58
*** melwitt has quit IRC20:06
*** markmcclain1 has quit IRC20:08
*** markmcclain has joined #openstack-qa20:08
*** melwitt1 has joined #openstack-qa20:08
*** melwitt1 is now known as melwitt20:09
openstackgerritMerged openstack-dev/devstack: Add IPv6 support for tenant data network  https://review.openstack.org/8798720:09
mtreinishmriedem: that should actually be the testtools issue which should be fixed now20:15
mtreinishtwice over actually20:15
openstackgerritSergey Shnaidman proposed openstack/tempest: DHCPv6 network tests  https://review.openstack.org/11745820:15
mriedemmtreinish: but it keeps showing up20:16
mriedemis there a bug reported?20:16
mriedemwe could at least classify it20:16
mtreinishmriedem: I don't think so, we kinda just rushed through the fix because everything was failing20:16
mtreinishmriedem: at least that was the issue before attrs broke in tempest with testtools 1.4.0 which meant the regex filter had no matches20:17
mtreinishwe blocked that testtools release and I landed a fix in tempest (we merged the revert of the block this morning)20:17
mtreinishmriedem: the way you can tell if it's the same issue is if the full job is running the heat slow tests20:18
mtreinishor grenade runs everything...20:18
*** shashankhegde has quit IRC20:20
mriedemit is a heat slow job, but it's showing up in other jobs also20:23
mtreinishmriedem: do you have a link to the other job?20:23
mriedemmtreinish: this? http://logs.openstack.org/72/123972/2/gate/gate-tempest-dsvm-neutron-heat-slow/be87c42/console.html#_2014-11-17_23_01_33_07820:24
mtreinishactually grenade would run nothing too, I did the regex match backwards...20:24
mtreinishmriedem: that's also heat20:24
mtreinishmriedem: actually isn't that the same link?20:25
mriedemit is20:25
mriedemsec20:25
openstackgerritAdam Gandelman proposed openstack/qa-specs: Improve diskimage-builder in devstack  https://review.openstack.org/13610120:28
openstackrecheckConsole logs not available after 13:25s for check-tempest-dsvm-full 104930,12,e13a31820:30
sergsh_who is interested in networking, please review my blueprint about additional tests in tempest: https://review.openstack.org/#/c/135896/20:34
*** melwitt has quit IRC20:36
*** melwitt has joined #openstack-qa20:37
*** vigneshvar has quit IRC20:38
*** salv-orlando has joined #openstack-qa20:44
*** patrickeast has quit IRC20:48
mriedemmtreinish: sorry, got sidetracked http://goo.gl/k5coaC20:49
*** patrickeast has joined #openstack-qa20:49
mriedemweird that didn't filter on gate20:49
*** MasterPiece has joined #openstack-qa20:50
*** andreykurilin_ has joined #openstack-qa20:50
*** atiwari has quit IRC20:50
mtreinishmriedem: it looks like the only gate tempest job is the heat slow job20:52
mtreinishthere are a bunch of nova functional and unit tests fails20:52
mtreinishand a whole lot of check things (which I'm just going to assume are bad patches)20:52
mriedemmtreinish: yeah that's why i was trying to filter out check queue20:53
mtreinishmriedem: anyway if you want to open a tempest bug for this20:53
mtreinishit was fixed by: https://review.openstack.org/13511520:54
mtreinishbut I guess it'll still be on the unclassified list20:54
mtreinishs/but/because20:54
mriedemmtreinish: ah ok, then that makes more sense20:54
mriedemhttp://status.openstack.org/elastic-recheck/data/uncategorized.html20:54
mriedemyeah that's how i was seeing it20:54
mriedemi'll get a bug opened and closed and tracked20:54
mriedemin e-r i mean20:54
*** yfried has joined #openstack-qa20:59
*** dimtruck is now known as zz_dimtruck20:59
dtroyermtreinish: in the -specs repo, should we do something to sort out the DevStack specs?  Like a subdir or convention of prefixing the files with 'ds-'?  Either of those is fine with me21:05
*** shashankhegde has joined #openstack-qa21:06
*** weshay has quit IRC21:08
*** e0ne has joined #openstack-qa21:10
*** zz_dimtruck is now known as dimtruck21:10
*** yamamoto has joined #openstack-qa21:12
mtreinishdtroyer: if you want to do a subdir that's fine, I was thinking of just adding a section to the template on projects it touches21:13
mtreinishI'd like to keep the rst files matching the lp bp name though so probably not a prefix21:13
dtroyermtreinish: the audiences don't totally overlap, and even if they did I'd expect you would be doing one or the other spec-wide.  I'll do a subdir and we can change it if necessary21:14
dtroyers/wide/wise/21:15
mtreinishok, sounds good21:15
*** david-lyle has joined #openstack-qa21:16
*** yamamoto has quit IRC21:17
*** mkollaro has quit IRC21:24
*** Sukhdev has quit IRC21:24
*** mpaolino has quit IRC21:25
openstackgerritMerged openstack/tempest: Add scenario test for swift-proxy ceilometer middleware  https://review.openstack.org/11597121:30
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for testtools/tempest bug 1394747  https://review.openstack.org/13613121:31
mriedemmtreinish: jogo: ^21:31
jogomriedem: glad that we fail on this now21:32
jogoinstead of passing silently21:32
mtreinishjogo: it was pretty easy to fix21:34
*** weshay has joined #openstack-qa21:34
mtreinishsdague: it looks like tempest-lib from src still isn't quite working: http://logs.openstack.org/18/129418/3/check/gate-tempest-dsvm-src-tempest-lib/232e0b2/logs/devstacklog.txt.gz#_2014-11-20_03_55_13_99121:36
openstackrecheckopenstack/requirements change: https://review.openstack.org/136018 failed gate-tempest-dsvm-full in the gate queue with an unrecognized error21:36
*** toure has quit IRC21:42
mtreinishmriedem, jogo: ^^^ looks like a swift test failure, don't see that very often21:42
mtreinishoh, nm it's just a ceilo failure21:43
jogohaha that makes more sense21:43
notmynamelol21:43
mtreinishjogo: I have a feeling that will be the new top flaky test. Another ceilo notification timeout21:45
jogomtreinish: are you working on the e-r query for it?21:45
mtreinishI wasn't, but I can be21:46
jogomtreinish: cool21:47
*** melwitt has quit IRC21:47
openstackgerritMerged openstack-infra/elastic-recheck: Add query for testtools/tempest bug 1394747  https://review.openstack.org/13613121:49
*** e0ne has quit IRC21:50
*** openstackrecheck has quit IRC21:50
*** openstackrecheck has joined #openstack-qa21:50
*** ajo has quit IRC21:55
*** JayJ has quit IRC21:55
mtreinishjogo: well this log is really useful: http://logs.openstack.org/18/136018/1/gate/gate-tempest-dsvm-full/4c27858/logs/screen-ceilometer-api.txt.gz22:04
jogowow22:05
mtreinishI think I'm just going to log on the console failure, I can't find anything else22:05
jogojust wow22:05
jogonotmyname: why can't your logs be this fun?22:05
*** jgrimm is now known as zz_jgrimm22:06
mtreinishjaypipes: ^^^ you're part of that logging wg thing right. Any thoughts on that one?22:07
*** ajo has joined #openstack-qa22:08
jogoclarkb: so the live migration stuff22:08
jogoclarkb: tempest uses what nova reports back as  the host22:08
*** melwitt has joined #openstack-qa22:09
*** bkopilov has quit IRC22:09
mtreinishjogo: as opposed to it using a host nova doesn't return? :)22:10
jogoas opposed to some IPs somehow22:11
jogomtreinish: it uses what nova returns from 'nova host-list'22:12
mtreinishjogo: oh, ok :) I guess tempest could do a lookup or something, but that would seem weird22:12
jogowhich makes sense, so I think we should just update the host list22:12
jogoclarkb: ^22:12
*** rwsu has quit IRC22:12
jogoright22:12
clarkbjogo: or we can set up name resolution22:12
clarkbit won't be too hard22:12
lifelesshost files 4 eva22:13
*** markmcclain has quit IRC22:13
jogolifeless: you need that on a knuckle tattoo22:14
jogoclarkb: lets do name resolution then22:14
clarkbjogo: basically do like we do for known hosts filling22:14
jogoclarkb: and update the host file?22:14
clarkbask each node for its hostname in turn, put entry in master file for that node, copy resulting file everywhere22:14
clarkbjogo: ya22:14
jogothat is what I meant yeah22:14
lifelessjogo: for this particular one yeah :) - you're traversing a well trodded path :)22:14
jogoI thought you meant DNS22:14
clarkbno dns is flaky in clouds22:15
jogoclarkb: don't we build the clouds ;)22:15
*** rwsu_ has joined #openstack-qa22:15
clarkbjogo: and no one gets dns right22:15
clarkbwe should use WINS22:15
jogoqueue mordred rant22:16
clarkbor netbios whatever that thing is called22:16
mordredDNS22:16
mordredit works22:16
clarkbmordred: not in our clouds it doesn't22:16
mordredlearn the internet22:16
clarkbits very saddening22:16
mordredwell22:16
mordredthat's because our clouds are crazy22:16
clarkbmordred: did you see I added even more checks around that recently22:16
clarkbit seems to have helped22:17
mordredclarkb: sigh22:17
clarkbas the name resolution stuff isn't high on e-r's list anymore22:17
clarkbmordred: basically nodepool checks if the node can resolve git.o.o before making the node ready22:17
clarkbif that fails then it deletes the node22:17
mordred++22:17
jogoclarkb: sounds like a great and sad hack22:17
jogodo you know what the rate of failure is for that22:17
mordredthat's ridiculous, of course22:17
clarkbjogo: e-r did22:17
*** rwsu_ has quit IRC22:19
jogoclarkb: for the hostname file22:22
sdaguemtreinish: yep, wrong call22:22
jogoI should run hostname on each box?22:22
clarkbjogo: ya you need a hosts file with a hostname to ip mapping of ever node22:22
clarkbjogo: because you don't know which way migration will go22:22
clarkbif VM is on primary node it will go to subnode and vice versa22:22
jogoclarkb: or hostname -f22:23
clarkbhrm its whatever nova uses22:23
clarkbjogo: maybe use both to be safe?22:23
lifelessmtreinish: sdague: any concerns about testtools-1.5.0 coming out after my lunch ?22:23
*** weshay has quit IRC22:23
mtreinishlifeless: not really, but if you could push up a d-g patch that installs it from source to test before the release that would be awesome22:24
jogoclarkb: devstack-trusty-2-node-rax-dfw-3143753-382.slave.openstack.org22:24
openstackgerritSean Dague proposed openstack-dev/devstack: fix tempest-lib setup  https://review.openstack.org/13615022:24
mtreinishbut I think we should be fine since the MRO fix landed22:24
sdaguemtreinish: ^^^ that should get you fixed22:25
clarkbjogo: if that is what nova uses you will need the -f22:25
*** dustins has quit IRC22:25
mtreinishsdague: ok cool22:25
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Add query for bug 1394774  https://review.openstack.org/13615122:26
sdagueok, dropping off for the day. Just rechecked all my backports because those got killed in the git storm.22:28
openstackgerritMatthew Treinish proposed openstack/tempest: Add check for too_slow_to_test flag on swift notification test  https://review.openstack.org/13615422:29
mtreinishjogo: ^^^ I think that should fix the failure22:29
* mtreinish still thinks that's the best feature flag in tempest22:30
lifelessmtreinish: I think a bunch of upstream tests would be nice :)22:31
ianwsdague: can you take a look at https://review.openstack.org/#/c/133914/22:32
mtreinishmriedem, jogo: https://review.openstack.org/136151 the only reason the hit count is so low is because icehouse is broken22:32
jogotoo slow to test lolg22:33
openstackgerritDean Troyer proposed openstack/qa-specs: Add references to DevStack specs  https://review.openstack.org/13615622:33
mriedemlooking22:33
mriedemmtreinish: we could pin that by branch and/or project probably22:35
*** gordc has joined #openstack-qa22:35
mtreinishmriedem: it'll hit every project but I can add a stable branch tag22:35
mriedemok22:35
gordcmtreinish: thanks for opening bug... i added a comment to it... seems like a configuration issue which we may or may not want to address22:36
gordcmtreinish: re: ceilometer swift bug22:36
mtreinishgordc: yeah I'm thinking the flag check will be enough it's probably just on icehouse22:37
mtreinishbut we can leave it open for a bit after that tempest change lands and see if it clears up22:37
gordcmtreinish: works for me. seems like a safer fix.22:37
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive  https://review.openstack.org/13583122:37
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full  https://review.openstack.org/13463922:37
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Enable live block migration  https://review.openstack.org/13576822:37
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu  https://review.openstack.org/13615822:37
*** MasterPiece has quit IRC22:37
jogomtreinish clarkb: ok let see what happens22:37
*** andreykurilin_ has quit IRC22:38
jogoI have to go AFK for a bit so if you see a bug  somewhere feel free to -1 and comment or fix it if you like22:38
*** dkranz has quit IRC22:38
jogonote I am expecting this to fail but with a different error now22:38
adam_gjogo, managed to get setup with a multinode setup locally. playing around a bit trying to get ec2 metadata working, what error are  chasing currently?22:38
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Add query for bug 1394774  https://review.openstack.org/13615122:38
mtreinishmriedem: ^^^22:38
mriedemmerged22:39
mtreinishmriedem: cool, thanks22:39
gordcmtreinish: i'm going to leave that bug open for ceilometer in case someone else wants to look at it and correct me tomorrow.22:40
*** mlavalle has quit IRC22:40
mtreinishmriedem: the reason it's only on requirements now is that icehouse jobs are broken everwhere else because of the taskflow reqs issue from this morning :)22:40
mtreinishit's been my reqs fixes that have bounced off of it...22:40
mtreinishgordc: ok cool, thanks22:40
openstackgerritSergey Shnaidman proposed openstack/tempest: DHCPv6 network tests  https://review.openstack.org/11745822:41
openstackgerritDean Troyer proposed openstack/qa-specs: Add references to DevStack specs  https://review.openstack.org/13615622:41
openstackgerritMerged openstack-infra/elastic-recheck: Add query for bug 1394774  https://review.openstack.org/13615122:46
*** openstackrecheck has quit IRC22:51
*** openstackrecheck has joined #openstack-qa22:51
*** suneelb has quit IRC22:54
*** suneelb has joined #openstack-qa22:55
*** dims has quit IRC23:12
*** ajo has quit IRC23:13
*** yamamoto has joined #openstack-qa23:14
openstackrecheckopenstack/requirements change: https://review.openstack.org/136018 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/139477423:15
*** yamamoto has quit IRC23:19
openstackgerritAdam Gandelman proposed openstack-dev/devstack: Set metadata_host in nova.conf  https://review.openstack.org/13617623:20
*** mkollaro has joined #openstack-qa23:26
*** dimtruck is now known as zz_dimtruck23:26
openstackgerritAdam Gandelman proposed openstack-infra/devstack-gate: Enable nova-api-metadata for multi host nova-network  https://review.openstack.org/13617923:31
*** tkammer has quit IRC23:35
*** gordc has quit IRC23:35
*** toure has joined #openstack-qa23:41
*** toure has quit IRC23:42
*** mriedem has quit IRC23:53
*** suneelb has quit IRC23:59
*** suneelb has joined #openstack-qa23:59
*** rbak has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!