Saturday, 2015-08-01

*** mtanino has quit IRC00:00
clarkbmordred: I do like the line length restrictions when I do side to side diff but otherwise its annoying00:00
*** achanda has quit IRC00:01
*** olaph has quit IRC00:01
*** achanda has joined #openstack-infra00:01
*** tsekiyama has quit IRC00:01
*** olaph has joined #openstack-infra00:01
*** tsekiyam_ has quit IRC00:02
*** Sukhdev_ has joined #openstack-infra00:03
*** MarkAtwood has quit IRC00:06
EmilienMclarkb: I might have missed that earlier but zuul queue is growing fast now, up to 170000:08
clarkbEmilienM: its processing the results of some things that were in the gate I think00:09
EmilienMclarkb: ok I was just informing you in case something is wrong00:09
*** sigmavirus24_awa is now known as sigmavirus2400:11
*** theannegentle has quit IRC00:12
EmilienMlol00:13
EmilienMwrong window, sorry00:13
clarkbwhat happens us we merge a nova change which generates a bunch of events to do the merge check00:14
clarkbso as long as the number falls afyer rising we should be ok00:14
*** sigmavirus24 is now known as sigmavirus24_awa00:15
*** alexsyip has joined #openstack-infra00:15
*** tjones has quit IRC00:15
*** megm has quit IRC00:15
*** megm has joined #openstack-infra00:17
*** alexsyip has quit IRC00:18
openstackgerritMerged openstack-infra/project-config: Run gate-dsvm-nodepool-nv on devstack-trusty  https://review.openstack.org/20809000:22
*** yolanda has joined #openstack-infra00:23
*** davideag_ has quit IRC00:24
*** apevec has quit IRC00:25
*** achanda has quit IRC00:25
*** achanda has joined #openstack-infra00:25
*** salv-orlando has quit IRC00:26
*** DericHorn-HP has quit IRC00:26
*** bpokorny has quit IRC00:28
openstackgerritMonty Taylor proposed openstack-infra/shade: Throw an exception on a server without an IP  https://review.openstack.org/20810500:30
*** sdake has joined #openstack-infra00:30
openstackgerritMerged openstack-infra/zuul: Add enqueue-ref command to client  https://review.openstack.org/20809300:31
*** angdraug has quit IRC00:32
*** DericHorn-HP has joined #openstack-infra00:33
*** alexsyip has joined #openstack-infra00:33
*** DericHorn-HP has quit IRC00:33
*** MarkAtwood has joined #openstack-infra00:39
*** MarkAtwood has quit IRC00:41
*** leseb_ has quit IRC00:41
*** Sukhdev_ has quit IRC00:45
*** MarkAtwood has joined #openstack-infra00:50
*** puranamr has quit IRC00:50
*** yamamoto has joined #openstack-infra00:53
*** pvaneck has quit IRC00:54
*** bswartz has joined #openstack-infra00:54
jeblairand when it gets down to 0, i'll restart zuul and trigger the neutron jobs00:56
*** alexsyip has quit IRC00:56
jeblairalso, if anyone wants to review https://review.openstack.org/208049 that would make the zuul logs much more useful again00:58
*** yamamoto has quit IRC00:58
*** jsavak has joined #openstack-infra01:01
*** signed8bit_ZZZzz is now known as signed8b_01:02
*** signed8b_ has quit IRC01:03
jeblairrestarting zuul01:05
pleia2approved01:06
*** tqtran has quit IRC01:08
*** puranamr has joined #openstack-infra01:11
*** btully has quit IRC01:11
*** puranamr has quit IRC01:16
*** jsavak has quit IRC01:17
jeblairokay, the neutron fwaas job is in the release pipelien01:17
jeblairi ran: zuul enqueue-ref --trigger gerrit --pipeline release --project openstack/neutron-fwaas --ref 2015.1.1 --newrev 068ea3674d12dd4243076ecaedf2ec5dce20e22101:18
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Throw an exception when servers don't have IPs  https://review.openstack.org/20809201:18
jeblairand the other two are in there now too01:19
*** armax has joined #openstack-infra01:23
jeblairoh, that probably should have been 'refs/tags/2015.1.1'.  but i think it's harmless -- it look like the right thing happened.01:29
*** puranamr has joined #openstack-infra01:29
*** MarkAtwood has quit IRC01:31
clarkbmordred did you make a new shade release? http://logs.openstack.org/95/207095/2/check/gate-dsvm-nodepool-nv/5033bee/logs/screen-nodepool.txt.gz01:33
clarkbthat was working in my test node earkier today01:33
*** puranamr has quit IRC01:33
clarkbI wobder if tge occ config isnt writteb where we look for it in d-g land01:36
clarkbanyways it did worj on my test machibe so we should be close01:36
*** crc32 has joined #openstack-infra01:38
clarkbhrm it doesnt specify a cloud so I amrealky confused01:39
*** Sukhdev_ has joined #openstack-infra01:44
*** skylerberg has quit IRC01:48
*** banix has joined #openstack-infra01:48
clarkblooks like keystoneclient Auth objects take a tenant_id and we are somehow passing in a project_name which explodes01:50
clarkbprobably because we are not using the cloud specifier and instead declare it directly which isn't getting normalized? though I thought shade's job was to normalize it all01:50
*** puranamr has joined #openstack-infra01:52
mordredclarkb: I did not make a new shade relaese01:52
mordredclarkb: also - hrm indeed - that SHOULD very much be a thing that occ handles01:53
mordredclarkb: I'm going to do lunch - then I will figure it out01:54
*** Sukhdev_ has quit IRC01:54
clarkbmordred: it almost looks like nodepool may require a cloud be set in nodepool.yaml01:54
clarkbmordred: and since we aren't setting one its exploding on passing the args into keystoneclient. BUT this worked fine earlier today so I am suspicious of that01:55
*** baoli has joined #openstack-infra01:55
clarkbOH! I see it in nodepool01:55
clarkblet me write a patch to see if it fixes01:56
clarkbI do find it excellent that we can't seem to agree onproject/tenant to this day01:56
*** puranamr has quit IRC01:57
openstackgerritClark Boylan proposed openstack-infra/nodepool: Use tenant_id for keystone not project_name  https://review.openstack.org/20811302:02
openstackgerritClark Boylan proposed openstack-infra/nodepool: Use OCC cloud specifier in devstack plugin  https://review.openstack.org/20811402:02
clarkbmordred: ^ two stabs at fixes, I think the first is the proper fix and the second is the meh we should use OCC anyways fix02:02
* morganfainberg waves at clarkb02:02
clarkbexcept lol OCC uses project_name so everything should diaf02:02
StevenKclarkb: Do you want to have a glance over https://review.openstack.org/207729 and see if it addresses your concerns?02:02
clarkbmorganfainberg: https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/auth/identity/v2.py#L54 that uses tenant-id02:03
clarkbmorganfainberg: so when we pass it project_name which OCC seems to think is canonical it explodes due to the positional decorator which checks this stuff02:03
morganfainbergoh yeesh02:03
clarkbmorganfainberg: really what keystoneclient should be doing is to stop being a jerk about it02:03
clarkbbecause well this is confusing and not even keystone gets it right apparnetly02:03
morganfainbergwait are you doing (tenant_name=<thing>)02:04
mordredclarkb: so - this is actually hard because we're not using shade for the client stuff02:04
morganfainbergor ... (thing, tenant_name)02:04
mordredmorganfainberg: ^^02:04
morganfainbergmordred: ++02:04
clarkbmordred: no its hard becaus ekeystoneclient is a jerk02:04
mordredno02:05
clarkbWE ARE USING SHADE02:05
mordrednot really02:05
mordredwe are not02:05
mordrednot for thus02:05
clarkblook at the traceback we are02:05
mordredreally?02:05
clarkbyes02:05
clarkb!02:05
clarkbsorry I have very little patience its 101 degrees outside and this is stupid02:05
clarkbshade is trying to create a keystoneclient session that the clinets can use02:05
clarkbso while we aren't using shade to talk to nova we are using shade to auth for nova client02:05
clarkband that breaks because shade says it should be project_name and keystoneclient says it should be tenant-id02:06
mordredclarkb:ok. gotcha. so - this SHOULD be working, I'm sorry it's not02:06
jeblairi recommend we abstract it in a library02:06
*** hogepodge has quit IRC02:06
* jeblair runs away, very quickly02:06
mordredclarkb: shade actually should be translating that to tenant_name for keystone if it's v202:06
morganfainbergjeblair: hah02:06
mordredclarkb: there is code explicity to handle that02:06
clarkbmordred: honestly keystoneclient shouldn't be a jerk here02:06
mordredso - something ha a bug somewhere02:06
clarkbbecause shade is a bug02:06
mordredclarkb: well, sure02:06
clarkband this is a case of shade buggyness02:06
mordredbut let me figure out why our workaround for the jerk part is not working02:06
clarkbso I am going to grump at keystoneclient02:07
morganfainbergclarkb: for the time being, I'd rather fix shade here, in 2.x of keystonelcient we *can* fix this (file a bug?)02:07
morganfainbergand/or bug me when not in aus02:07
clarkbmorganfainberg: you cna fix it in 1.0 too02:07
* morganfainberg makes a note too02:07
clarkbmorganfainberg: just accept both forms02:07
clarkbits forward and backward compatible02:07
morganfainbergclarkb: sortof.02:07
*** sergmelikyan has quit IRC02:07
mordredclarkb: oh... you know what ...02:07
morganfainbergclarkb: i'll need to go harass jamie about the positional thing02:07
mordredclarkb: this is a devstack bug02:07
morganfainbergthats more of the issue than the other thing.02:07
morganfainbergthe name of the kwarf02:07
mordredlet me make patch real quick02:07
clarkbmordred: can you explain?02:07
morganfainbergkwarg02:07
clarkbmordred: should it be using v1 or v3 instead?02:08
mordredclarkb: yes. devstack writes a clouds.yaml, and I believe is likely indicating the wrong version of keystone api02:08
clarkbmordred: we aren't using clouds.yaml yet02:08
mordredI believe I encountered this with manual testing the other day02:08
clarkbmordred: so thats not the issue02:08
*** davideagnello has joined #openstack-infra02:08
clarkbI just pushed a followup change that will use a clouds.yaml so we can check that too, but the current fail is unrelated to that02:08
mordredclarkb: it still could be ... because version detection is not really a thing02:08
clarkbmordred: it explicitly passes in an auth url there02:09
mordredthat's irrelevant02:09
mordredyou have to tell it whether you're using v2 or v302:09
clarkbI don'tunderstand how nodepool uses clouds.yaml when we don't say cloud: devstack02:09
clarkbso I don't see how devstack can be at fault02:09
clarkbalso I hvae no idea how this worked earlier today02:09
mordredme either - I have not released any occ or shade02:10
*** hogepodge has joined #openstack-infra02:11
*** baoli has quit IRC02:12
morganfainbergclarkb: sure, i can do a patch for that behavior in ksc today02:12
morganfainbergand ask on monday for a release02:12
mordredclarkb: uhm ...02:12
morganfainbergclarkb: fyi02:13
mordredclarkb: what version of shade are we using on nodepool? any idea?02:13
*** baoli has joined #openstack-infra02:13
morganfainbergwell and bug keystone-core to merge it02:13
clarkbmordred: pip freeze says 0.11.002:13
openstackgerritMerged openstack/requirements: Updated from generate-constraints  https://review.openstack.org/20774102:13
mordredclarkb: because I checked out the latest tag and that error line is weird02:13
clarkbmordred: http://logs.openstack.org/95/207095/2/check/gate-dsvm-nodepool-nv/5033bee/logs/pip-freeze.txt.gz02:13
mordredoh - wait - reading it wrong02:13
mordredsorry02:14
*** laughterwym has joined #openstack-infra02:14
clarkbI believe the setup here is shade is doing auth using the nodepool.yaml data NOT the clouds.yaml data02:14
clarkbthe nodepool.yaml data can be viewed at http://logs.openstack.org/95/207095/2/check/gate-dsvm-nodepool-nv/5033bee/logs/etc/nodepool/nodepool.yaml.txt.gz02:14
mordredyah. it should be passing those in to OCC as keyword arguments, which should make it all the same02:15
clarkbso it is using v2 via line 108 https://review.openstack.org/#/c/207095/2/devstack/plugin.sh02:15
clarkbmordred: right I believe shade making it all the same is to use project_name02:15
clarkbmordred: I recall explicitly forcing things to use that instead of tenant* or project-id02:16
clarkband I believe that is incompatible with keystoneclient02:16
mordredyes. I understand. but if occ wasn't doing this translation properly nothing would have been working ever02:16
mordredso I'm very confused as to why its not woring _here_02:16
mordredocc on purpose makes you always pass in project_name because asking you to pass in tenant_name sometimes is stupid02:17
mordredbut then, on constructor invocation, it's all supposed to remap things properly02:17
mordredso taht you don't have to know about this02:17
mordredbecause it's stupid02:17
clarkbcould it be a different code path when using clouds.yaml?02:17
clarkband since we clouds.yaml now it just works02:17
clarkb(I suppose my change to use clouds.yaml should shed light on that)02:17
mordredit should not be - but I'm walking through it right now ... also, that would be a bug we shoudl fix, so yay for testing02:18
mordred:)02:18
clarkbalso I notice that we still speak to keystone via port 3535702:18
clarkbmorganfainberg: ^ do we set up apache + wsgi keystone to use that port too ?02:19
mordredyeah. I believe jamielennox|away is working on making the default keystone in devstack less bad02:19
clarkbthat seems like a huge bug imo02:19
clarkbbecause one of the big benefits to keystone in apache was we could stop using that broken port that may not always be free02:19
morganfainbergclarkb: we do, but we are working towards fixing that02:20
*** coolsvap|away is now known as coolsvap02:20
morganfainbergclarkb: ayoung is working on making it /identity on port 80 iirc02:20
morganfainbergclarkb: rather than any random port02:21
clarkbmordred: https://github.com/openstack/os-client-config/blob/2757709f8b84beb550b59bac48f4540a2b06b4ff/os_client_config/config.py#L285 tat seems to indicate tenant-id will not be used02:23
clarkb1.6.0 was released in june so its not new either02:25
mordredclarkb: yah - reading through I'm confused as to why we have not hit this earlier02:25
clarkbmaybe we just v3 everywhere02:27
mordredwell, we can't - because our clouds do not support it02:28
mordredclarkb: I'm doing some synthetic tests against our public clouds right now02:28
clarkbbut it definitely does not accept project_name in the client so I am super confused how it ever worked02:29
mordredkeystoneclient.auth.identity.generic.password.Password <--02:29
mordredthisis what a local script with shade uses02:29
mordredagainst hp02:29
clarkbso is it the v2 specifc url that bites us?02:30
clarkbhttps://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/auth/identity/generic/base.py#L53 generic takes the project_name and all the rest02:30
clarkbso its just the v2 specific path that explodes02:30
clarkb(which is even more confusing as a user)02:30
Guest52591.02:33
*** Guest52591 is now known as morganfainberg_02:33
clarkbmordred: https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/auth/base.py#L58 that seems to select the class02:33
clarkbbut shade is setting the param to that02:33
*** achanda has quit IRC02:34
*** achanda has joined #openstack-infra02:34
*** julim has joined #openstack-infra02:35
morganfainberg_clarkb: I'm having issues loading web pages atm here let me swtich to mobile maybe02:35
mordredclarkb: ok. so ..02:35
mordred(tethered to phone now)02:36
*** salv-orlando has joined #openstack-infra02:36
clarkbwe seemto not be setting an explicit auth_type02:36
mordredclarkb: when I run locally, keystoneclient.auth.identity.generic.password.Password is what is used and that works02:36
*** johnbelamaric has joined #openstack-infra02:36
mordredthat's good- we want that to default to password02:36
mordredso that we get the right thing02:36
mordredbut - I think something in that devstack setup is causing auth_type to get set to v2password02:37
mordredI wonder if something is setting that in an env var somewhere02:37
morganfainberg_oh02:37
clarkbmordred: ok, if that is the case then OCC is leaky02:37
clarkbbecause we aren't telling it to use OCC anywhere02:37
morganfainberg_that might be related to the v3 work?02:37
morganfainberg_some thing about compat02:37
*** ivar-lazzaro has quit IRC02:38
mordredmorganfainberg_: ^^ that's likely02:38
clarkbok OS_AUTH_TYPE=v2password02:38
mordredclarkb: yah. that's borking us02:38
clarkbso if we change that to password it should work?02:38
morganfainberg_clarkb: thats the thought02:38
mordredno - I'll take a note to make sure that doesn't bork us in the future02:38
mordredbut yes. change that to pasword and we'll be fine02:38
mordred(that's an easy occ/shde workaround)02:38
clarkbthis seems really broken in keystoneclient02:39
clarkbif generic works why have a public v2 interface at all?02:39
mordredclarkb: it's a terrible transition - morganfainberg_ and I are meatspace discussing it02:39
clarkband if we have to have a v2 interface why don't we make it compatible with the same interface?02:39
clarkbbecause presumably v2password ~= password02:39
mordredclarkb: yah. I believe that is what they're trying to work through - and we've caught an edge condition in the transition02:40
mordredonce it's done, this should do what you want02:40
mordredAIUI02:40
morganfainberg_clarkb: what mordred said02:40
mordredbut for now - we should force auth_type02:40
mordredin fact ... I'm going to put in a thing to shade that says "if auth_type == v2password: auth_type = password" - because of how we force to v3 words02:41
mordredbecause the break here is confusing02:41
*** davideagnello has quit IRC02:42
morganfainberg_let me see what landed where shortly and see if we can address it [and not be broken]02:42
*** ajmiller has joined #openstack-infra02:42
morganfainberg_but quick fix in shade is going to fix sooner02:42
*** ajmiller has quit IRC02:43
mordredclarkb, morganfainberg_: https://review.openstack.org/20811802:47
*** ajmiller has joined #openstack-infra02:47
mordredclarkb: ^^ that will remove the need for you to work around the problem in teh devstack plugin02:47
*** salv-orlando has quit IRC02:47
*** ddieterly has joined #openstack-infra02:48
ayoungclarkb, mordred, 443  https02:48
mordredclarkb: morganfainberg_ has also found the place it can be fixed in ksc02:48
openstackgerritClark Boylan proposed openstack-infra/nodepool: Use OCC cloud specifier in devstack plugin  https://review.openstack.org/20811402:48
openstackgerritClark Boylan proposed openstack-infra/nodepool: Use password auth_type  https://review.openstack.org/20811902:48
clarkbmordred: ^ something like that should work I hope02:48
ayoungclarkb, the way that devstack setups up apache in in a virtual host.  Need to replace that with code that does something  with the same vhost as horizon runs in02:48
ayoungand in order to do that, we needed the WEBROOT fix, but I think that is now in.02:49
mordredclarkb: yes. that will fix it02:49
*** ajmiller_ has joined #openstack-infra02:50
clarkbhttps://jenkins02.openstack.org/job/gate-dsvm-nodepool-nv/2/console for those follwoing along02:51
*** skylerberg has joined #openstack-infra02:52
*** ajmiller has quit IRC02:53
mordredclarkb: ok- I have verified that the occ fix will fix it locally02:58
mordred(recreated the error and then applied the patch and it fixed it)02:58
*** pal has joined #openstack-infra03:00
mordredclarkb: occ 1.6.1 released with fix applied03:04
*** pal has quit IRC03:07
morganfainbergyay! fixed!03:07
*** pal has joined #openstack-infra03:07
*** fedexo has joined #openstack-infra03:08
*** laughterwym has quit IRC03:10
clarkbhaha changing the auth type in devstack made things fail03:11
clarkbmordred cool so now I shouldnt need to do anything03:11
*** baoli has quit IRC03:13
*** kushal has quit IRC03:13
*** julim has quit IRC03:14
*** coolsvap is now known as coolsvap|away03:18
clarkbhrm now we need a constraints updqte. time to weekend03:20
*** zul has quit IRC03:22
*** dimsum__ has quit IRC03:22
*** armax has quit IRC03:23
*** baoli has joined #openstack-infra03:23
*** dimsum__ has joined #openstack-infra03:23
*** eharney has quit IRC03:24
*** armax has joined #openstack-infra03:24
*** skylerberg has quit IRC03:25
*** harlowja_at_home has joined #openstack-infra03:25
*** shashank_hegde has joined #openstack-infra03:26
*** skylerberg has joined #openstack-infra03:26
openstackgerritlitong01 proposed openstack-infra/project-config: Add new kiloeyes repository for OpenStack monitoring service  https://review.openstack.org/20325903:28
*** armax has quit IRC03:29
*** Sukhdev_ has joined #openstack-infra03:30
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/system-config: Add fuel-library channel  https://review.openstack.org/20031003:34
*** zul has joined #openstack-infra03:37
*** crc32 has quit IRC03:38
*** harlowja_still_a has joined #openstack-infra03:39
*** pal has quit IRC03:40
*** harlowja_at_home has quit IRC03:42
*** skylerberg has quit IRC03:45
*** pal has joined #openstack-infra03:49
*** bhunter71 has quit IRC03:49
*** reed_ has joined #openstack-infra03:50
*** skylerberg has joined #openstack-infra03:50
*** reed_ has quit IRC03:52
*** banix has quit IRC03:53
*** dimsum__ has quit IRC03:58
*** coolsvap|away is now known as coolsvap04:00
*** kushal has joined #openstack-infra04:01
*** johnbelamaric has quit IRC04:05
*** crc32 has joined #openstack-infra04:05
*** isq_ has quit IRC04:07
*** skylerberg has quit IRC04:07
*** isq_ has joined #openstack-infra04:08
*** armax has joined #openstack-infra04:08
*** MarkAtwood has joined #openstack-infra04:10
*** btully has joined #openstack-infra04:15
*** btully has quit IRC04:19
*** salv-orlando has joined #openstack-infra04:28
*** fedexo has quit IRC04:29
*** fedexo has joined #openstack-infra04:30
openstackgerritSirushti Murugesan proposed openstack-infra/project-config: Make mistralclient voting for python3  https://review.openstack.org/20812404:34
*** DericHorn-HP has joined #openstack-infra04:35
*** salv-orlando has quit IRC04:35
*** salv-orlando has joined #openstack-infra04:38
*** ddieterly has quit IRC04:40
*** salv-orlando has quit IRC04:42
*** unicell has joined #openstack-infra04:46
*** sdake has quit IRC04:47
*** shashank_hegde has quit IRC04:51
*** sdake has joined #openstack-infra04:55
*** sdake has quit IRC05:03
*** DericHorn-HP has quit IRC05:05
*** btully has joined #openstack-infra05:06
*** achanda has quit IRC05:08
*** achanda has joined #openstack-infra05:09
*** fawadkhaliq has joined #openstack-infra05:10
*** baoli has quit IRC05:11
*** MarkAtwood has quit IRC05:12
*** e0ne has joined #openstack-infra05:16
*** e0ne has quit IRC05:18
*** harlowja_still_a has quit IRC05:20
*** kushal has quit IRC05:21
*** changbl has joined #openstack-infra05:21
*** fedexo has quit IRC05:25
*** fedexo has joined #openstack-infra05:27
*** Sukhdev_ has quit IRC05:30
*** e0ne has joined #openstack-infra05:30
*** e0ne has quit IRC05:31
*** shashank_hegde has joined #openstack-infra05:32
*** coolsvap is now known as coolsvap|away05:33
*** shashank_hegde has quit IRC05:35
*** DericHorn-HP has joined #openstack-infra05:38
*** fedexo has quit IRC05:38
*** DericHorn-HP has quit IRC05:39
*** salv-orlando has joined #openstack-infra05:41
*** armax has quit IRC05:46
*** crc32 has quit IRC05:48
*** pal has quit IRC05:49
*** fedexo has joined #openstack-infra05:53
*** salv-orlando has quit IRC05:54
*** skylerberg has joined #openstack-infra05:54
*** fawadkhaliq has quit IRC05:56
*** DericHorn-HP has joined #openstack-infra05:56
*** DericHorn-HP has quit IRC05:58
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/20773906:01
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/20812606:06
*** skylerberg has quit IRC06:13
*** sergmelikyan has joined #openstack-infra06:15
openstackgerritMerged openstack-infra/elastic-recheck: Add query for ceilometer meters faultstring 500  https://review.openstack.org/20808506:17
*** mrmartin has joined #openstack-infra06:25
*** pal has joined #openstack-infra06:27
*** DericHorn-HP has joined #openstack-infra06:32
*** DericHorn-HP has quit IRC06:32
*** ddieterly has joined #openstack-infra06:42
*** ddieterly has quit IRC06:46
*** _nadya_ has joined #openstack-infra06:46
*** fedexo has quit IRC06:50
*** salv-orlando has joined #openstack-infra06:53
*** MarkAtwood has joined #openstack-infra07:00
*** _nadya_ has quit IRC07:03
*** _nadya_ has joined #openstack-infra07:03
*** salv-orlando has quit IRC07:05
*** MarkAtwood has quit IRC07:05
*** _nadya_ has quit IRC07:08
*** otter768 has quit IRC07:13
*** fedexo has joined #openstack-infra07:16
*** DericHorn-HP has joined #openstack-infra07:20
*** DericHorn-HP has quit IRC07:26
*** fedexo has quit IRC07:35
*** ddieterly has joined #openstack-infra07:42
*** ddieterly has quit IRC07:47
*** mrmartin has quit IRC07:53
*** btully has quit IRC08:00
*** jlanoux has joined #openstack-infra08:01
*** sergmelikyan has quit IRC08:03
*** salv-orlando has joined #openstack-infra08:04
*** jlanoux has quit IRC08:07
*** puranamr has joined #openstack-infra08:07
*** achanda has quit IRC08:08
*** salv-orlando has quit IRC08:10
*** puranamr has quit IRC08:12
*** pal has quit IRC08:27
*** DericHorn-HP has joined #openstack-infra08:35
*** DericHorn-HP has quit IRC08:39
*** marzif has quit IRC08:40
*** ddieterly has joined #openstack-infra08:43
*** pal has joined #openstack-infra08:46
*** ddieterly has quit IRC08:48
*** MarkAtwood has joined #openstack-infra09:02
*** pal has quit IRC09:03
*** MarkAtwood has quit IRC09:07
*** pal has joined #openstack-infra09:10
*** salv-orlando has joined #openstack-infra09:29
*** mrmartin has joined #openstack-infra09:37
*** salv-orlando has quit IRC09:41
*** salv-orlando has joined #openstack-infra09:44
*** ddieterly has joined #openstack-infra09:44
*** DericHorn-HP has joined #openstack-infra09:45
*** ddieterly has quit IRC09:49
*** salv-orlando has quit IRC09:50
*** salv-orlando has joined #openstack-infra10:03
*** boris-42 has quit IRC10:10
*** vahidh has quit IRC10:19
*** HeOS has quit IRC10:25
*** pal has quit IRC10:27
*** HeOS has joined #openstack-infra10:28
*** puranamr has joined #openstack-infra10:29
*** salv-orlando has quit IRC10:33
*** puranamr has quit IRC10:33
*** vahidh has joined #openstack-infra10:33
*** ddieterly has joined #openstack-infra10:45
*** ddieterly has quit IRC10:50
*** MarkAtwood has joined #openstack-infra10:51
*** dimsum__ has joined #openstack-infra10:51
*** MarkAtwood has quit IRC10:56
*** puranamr has joined #openstack-infra11:28
*** nim has quit IRC11:29
*** puranamr has quit IRC11:32
*** laughterwym has joined #openstack-infra11:35
*** ddieterly has joined #openstack-infra11:46
*** shashank_hegde has joined #openstack-infra11:48
*** ddieterly has quit IRC11:51
*** markvoelker has joined #openstack-infra11:57
*** Nakato has quit IRC12:00
*** shashank_hegde has quit IRC12:02
*** Nakato has joined #openstack-infra12:02
*** pal has joined #openstack-infra12:04
*** weshay has quit IRC12:08
*** weshay has joined #openstack-infra12:10
*** shashank_hegde has joined #openstack-infra12:19
*** shashank_hegde has quit IRC12:30
*** _nadya_ has joined #openstack-infra12:33
*** pal has quit IRC12:36
*** MarkAtwood has joined #openstack-infra12:40
*** MarkAtwood has quit IRC12:45
*** ddieterly has joined #openstack-infra12:46
*** che-arne has joined #openstack-infra12:51
*** ddieterly has quit IRC12:52
*** sergmelikyan has joined #openstack-infra13:04
*** sergmelikyan has quit IRC13:08
*** mriedem has quit IRC13:15
*** mriedem1 has joined #openstack-infra13:15
*** mriedem1 has quit IRC13:23
*** mriedem has joined #openstack-infra13:23
*** mriedem1 has joined #openstack-infra13:25
*** _nadya_ has quit IRC13:26
*** che-arne has quit IRC13:26
*** _nadya_ has joined #openstack-infra13:26
*** mriedem has quit IRC13:27
*** _nadya_ has quit IRC13:30
*** johnbelamaric has joined #openstack-infra13:31
*** ddieterly has joined #openstack-infra13:40
*** olaph has quit IRC13:41
*** teran has quit IRC13:47
*** btully has joined #openstack-infra13:48
openstackgerritGuojian Shao proposed openstack-infra/project-config: test against neutron for osc gate job  https://review.openstack.org/19894813:49
*** btully has quit IRC13:53
*** markvoelker has quit IRC13:53
*** thorst has joined #openstack-infra13:57
funginot really around today, but i've noticed that my nova unit tests on ubuntu-trusty are running about twice as long as on bare-trusty, in the same region in rax during the same time period. i thought at first it might be the extra time to install packages but that's not it (only adds a few seconds). pip installing requirements takes twice as long, the tests themselves take twice as long...13:57
fungii think we might have a performance issue with the ubuntu-trusty minimal builds13:57
openstackgerritTerry Howe proposed openstack/os-testr: Convert file names to regular expressions  https://review.openstack.org/20342413:58
fungiobviously not a statistically significant sample size yet, but as an anecdote it's still pretty suggestive13:58
fungiexample from today: http://logs.openstack.org/44/164444/6/experimental/bindep-nova-python27/71e1395/console.html vs http://logs.openstack.org/44/164444/6/check/gate-nova-python27/fc14e0b/console.html13:59
fungimordred: greghaynes: ^ insights welcome14:00
*** thorst has quit IRC14:01
clarkbfungi likely not pvhvm14:01
clarkbwe should double check the metadata set on the image14:02
fungiooh, good idea14:02
fungialso, clearly your sleep patterns are beyond repair clarkb :/14:02
StevenKThey have been twinned to death?14:05
fungiclarkb: metadata vm_mode hvm14:05
fungithat's the most recent (uhm... from 8 days ago... sigh) image the node would have booted from14:06
*** mrmartin has quit IRC14:06
*** changbl has quit IRC14:07
clarkbI am preparing to sleep all weekend14:08
fungigreat idea14:08
clarkbI was going to go outside but now it is an oven out there14:09
fungii saw, they were calling for 99f on the forecast for your neighborhood. not fun14:10
fungiat least here it's supposed to be in the low 90s14:10
fungiso anyway, comparing the ubuntu-trusty import image and bare-trusty snapshot image in rax-dfw, they're essentially the same14:13
*** johnbelamaric has quit IRC14:14
fungiat least the things that differ don't seem like they should impact anything14:14
fungialso logged into a sample system and confirmed we've got the same flavor of kernel installed on both14:14
fungianyway, if i get time later i might hold a couple of nodes and try running a variety of benchmarks to see if i can discern a particular resource constraint that would explain this14:15
fungibut if somebody else beats me to it, all the better14:15
fungii should say, i compared the _metadata_ for both images and they were effectively the same14:16
*** rguillebert has joined #openstack-infra14:17
*** MarkAtwood has joined #openstack-infra14:26
*** DericHorn-HP has quit IRC14:29
greghaynesclarkb, fungi - phvm seems plausible14:32
greghaynesfungi, clarkb - also it will be less performant because the rootfs is nearly full14:33
greghaynesand sometime it fails due to out of space for me14:33
* greghaynes is on a beach today so also not available14:34
*** puranamr has joined #openstack-infra14:34
greghaynesfungi: oh, so phvm is set...14:34
greghaynesfungi: I wobder if the rootfs could be jt14:34
*** puranamr has quit IRC14:38
greghaynesfungi: also, it was apparently 102 there yesterday14:38
*** armax has joined #openstack-infra14:48
*** btully has joined #openstack-infra14:49
*** mrmartin has joined #openstack-infra14:50
*** mrmartin has quit IRC14:52
*** ddieterly has quit IRC14:52
*** jlanoux has joined #openstack-infra14:52
*** btully has quit IRC14:53
*** markvoelker has joined #openstack-infra14:54
*** davideagnello has joined #openstack-infra14:55
*** johnbelamaric has joined #openstack-infra14:57
*** davideagnello has quit IRC14:57
*** markvoelker has quit IRC14:59
fungigreghaynes: not sure i would call it nearly full... seems to be about 2/3 full http://paste.openstack.org/show/406659/15:00
*** MarkAtwood has quit IRC15:00
fungigranted, some of that 4gb will be taken up installing a few packages and the nova requirements/test-requirements in a venv15:02
fungii've added https://review.openstack.org/208155 to see just how bad it gets15:03
fungihttps://jenkins06.openstack.org/job/bindep-nova-python27/7/console should hopefully print a df at the end when it's finishing15:05
*** jlanoux has quit IRC15:07
greghaynesfungi: ah, I just noticed I was hitting max in some of the tests so I assumed it was perpetually close15:10
greghaynesbut yea, would be nice to see what that oatch shows15:10
fungiyeah, i would expect devstack-based jobs to probably need more than 4gb available space15:10
fungibut nova unit tests less so15:10
fungithat said, this one seems to be going way faster15:11
greghayneshrm15:11
fungii wonder whether we have performance variability of a factor of 2 or greater15:11
funginevermind, i was looking at the wrong line15:12
fungiit's on the pip install now... on a bare-trusty this part seems to go in ~5 minutes vs ~10 on ubuntu-trusty15:12
*** davideagnello has joined #openstack-infra15:13
*** davideagnello has quit IRC15:13
*** MarkAtwood has joined #openstack-infra15:13
fungiyeah, compare to https://jenkins06.openstack.org/job/gate-nova-python27/12192/console which started at about the same time and is well into running tests already15:14
*** davideagnello has joined #openstack-infra15:14
fungithough on hpcloud this time, so less of a valid comparison15:14
*** rossella_s has quit IRC15:16
*** rossella_s has joined #openstack-infra15:16
*** MarkAtwood has quit IRC15:17
*** MarkAtwood has joined #openstack-infra15:17
*** davideagnello has quit IRC15:18
*** mriedem has joined #openstack-infra15:23
*** MarkAtwood has quit IRC15:23
*** mriedem1 has quit IRC15:24
fungigreghaynes: /dev/xvda1       13G  8.8G  3.3G  73% /15:25
*** xnox has quit IRC15:25
fungiso only chews up an additional ~0.7gib15:25
*** xnox has joined #openstack-infra15:27
fungianyway, i have to get back to moving stuff. bbl15:28
*** rguillebert has quit IRC15:30
*** alexsyip has joined #openstack-infra15:33
greghayneshrmmm, ok15:34
*** rguillebert has joined #openstack-infra15:34
*** alexsyip has quit IRC15:34
*** laughterwym has quit IRC15:43
jeblairmordred: retro thinkpad survey 4 is available: http://blog.lenovo.com/en/blog/retro-thinkpad-survey-4-miscellaneous/15:43
*** ddieterly has joined #openstack-infra15:53
*** markvoelker has joined #openstack-infra15:55
*** pcaruana has joined #openstack-infra15:56
*** ddieterly has quit IRC15:58
*** markvoelker has quit IRC15:59
*** skylerberg has joined #openstack-infra16:04
*** johnbelamaric has quit IRC16:06
*** dimsum__ has quit IRC16:13
*** ihrachyshka has joined #openstack-infra16:15
*** _nadya_ has joined #openstack-infra16:27
*** harlowja_at_home has joined #openstack-infra16:27
*** _nadya_ has quit IRC16:31
*** armax has quit IRC16:36
*** armax has joined #openstack-infra16:40
openstackgerritJoshua Harlow proposed openstack-infra/project-config: A few more tweaks to anvil-jobs and usage  https://review.openstack.org/20816616:45
*** armax has quit IRC16:47
*** harlowja_at_home has quit IRC16:48
*** sdake has joined #openstack-infra16:49
*** cburroughs has joined #openstack-infra16:50
*** DericHorn-HP has joined #openstack-infra16:50
*** skylerbe1g has joined #openstack-infra16:51
*** glauco_ has joined #openstack-infra16:52
*** skylerberg has quit IRC16:52
*** glauco_ has quit IRC16:54
*** ddieterly has joined #openstack-infra16:54
*** ddieterly has quit IRC16:59
*** dimsum__ has joined #openstack-infra17:00
*** skylerbe1g has quit IRC17:07
*** dimsum__ has quit IRC17:08
*** dimsum__ has joined #openstack-infra17:09
*** cburroughs has quit IRC17:15
*** ParsectiX has joined #openstack-infra17:18
*** davideagnello has joined #openstack-infra17:19
*** ddieterly has joined #openstack-infra17:20
*** davideagnello has quit IRC17:24
*** asahlin has quit IRC17:25
*** asahlin has joined #openstack-infra17:26
*** dimsum__ has quit IRC17:27
openstackgerritPaul Michali proposed openstack-infra/devstack-gate: Remove q-vpn service for DSVM tests  https://review.openstack.org/20816717:28
*** Somay has joined #openstack-infra17:29
*** DericHorn-HP has quit IRC17:29
greghaynesfungi: welp, I am kinda stumped since that image is apparently hvm...17:31
greghaynesfungi: we are actually booting the same flavor, right?17:32
*** otter768 has joined #openstack-infra17:33
*** pal has joined #openstack-infra17:37
*** ihrachyshka has quit IRC17:41
*** otter768 has quit IRC17:43
*** ashleighfarnham has quit IRC17:45
*** otter768 has joined #openstack-infra17:46
*** DericHorn-HP has joined #openstack-infra17:48
*** sergmelikyan has joined #openstack-infra17:49
*** sdake has quit IRC17:52
*** tjones has joined #openstack-infra17:53
*** ParsectiX has quit IRC17:54
*** markvoelker has joined #openstack-infra17:56
*** ParsectiX has joined #openstack-infra18:00
*** markvoelker has quit IRC18:00
*** sergmelikyan has quit IRC18:07
*** DericHorn-HP has quit IRC18:09
*** Somay has quit IRC18:18
*** tjones has quit IRC18:18
*** DericHorn-HP has joined #openstack-infra18:29
*** Somay has joined #openstack-infra18:32
*** ig0r_ has joined #openstack-infra18:32
*** Somay has quit IRC18:33
fungigreghaynes: i believe so. at least the ram and cpu count match18:35
fungigreghaynes: next thing to do is probably hold a couple of them and do some more precise benchmarking18:36
*** achanda has joined #openstack-infra18:39
*** pal has quit IRC18:42
*** ig0r_ has quit IRC18:42
*** DericHorn-HP has quit IRC18:50
*** Hal has joined #openstack-infra18:56
greghaynesfungi: also, when you hold you can dmesg | grep Xen to see if its actually hvm18:56
*** Hal is now known as Guest345418:56
*** glauco_ has joined #openstack-infra19:03
clarkbgood idea its possible the image metadata is insufficient or buggy19:07
*** pal has joined #openstack-infra19:08
*** Guest3454 has quit IRC19:10
fungiBooting paravirtualized kernel on Xen HVM19:13
fungigreghaynes: clarkb: ^19:14
*** glauco_ has quit IRC19:16
*** rossella_s has quit IRC19:16
*** rossella_s has joined #openstack-infra19:16
morganfainbergjeblair: nice (re: survey).19:17
*** thorst has joined #openstack-infra19:17
greghaynesyea, so seems like it is hvm19:19
greghaynesnow I am sufficiently stumped19:20
*** mordred has quit IRC19:20
*** thorst has quit IRC19:21
*** jlanoux has joined #openstack-infra19:24
*** ParsectiX has quit IRC19:31
*** e0ne has joined #openstack-infra19:39
*** armax has joined #openstack-infra19:39
*** Somay has joined #openstack-infra19:44
*** mrmartin has joined #openstack-infra19:47
openstackgerritSomay Jain proposed openstack-infra/jenkins-job-builder: Support for externaljob project-type for monitoring external jobs  https://review.openstack.org/20404719:48
*** sanjayu has quit IRC19:48
*** mrmartin has quit IRC19:49
*** MarkAtwood has joined #openstack-infra19:56
*** sergmelikyan has joined #openstack-infra19:56
*** markvoelker has joined #openstack-infra19:57
*** sergmelikyan has quit IRC20:01
*** e0ne has quit IRC20:02
*** markvoelker has quit IRC20:02
*** armax has quit IRC20:07
*** mordred has joined #openstack-infra20:09
*** MarkAtwood has quit IRC20:10
*** MarkAtwood has joined #openstack-infra20:13
*** Somay has quit IRC20:15
*** pal has quit IRC20:15
mordredgreghaynes: I do not find it amusing that there is a performance issue20:25
mordredmorganfainberg_: aren't you up a little early?20:25
*** julim has joined #openstack-infra20:25
greghaynesI wonder if rax is going "oh, you want to boot a custom image? you can go run on the mac minis in the closet."20:26
*** jlanoux has quit IRC20:28
mordredgreghaynes: why do I havea  hunch that there is a new piece of metadata were not setting20:30
greghaynesmeta turbo_button true?20:31
*** julim has quit IRC20:31
clarkbother idea, the disk image format is less performant than what a snapshot produces20:31
mordredgreghaynes: RATS. we forgot tubo_button20:31
mordredclarkb: that would be sad making20:32
greghaynes:(20:32
* fungi presses the turbo button and witnesses the leds on the front leap from 25 to 50 mhz20:35
* fungi rubs turbo button blood on ubuntu-trusty metadata20:35
mordredI do not see anything special in teh glance metadata for the rax provided images20:36
*** sdake has joined #openstack-infra20:39
*** markvoelker has joined #openstack-infra20:58
*** fawadkhaliq has joined #openstack-infra21:01
*** markvoelker has quit IRC21:02
*** sanjayu has joined #openstack-infra21:07
*** Adri2000 has quit IRC21:14
openstackgerritEmilien Macchi proposed openstack-infra/project-config: jenkins: implement copy-puppet-logs publisher  https://review.openstack.org/20669321:14
*** Adri2000 has joined #openstack-infra21:15
morganfainbergmordred: early, late, all the same travelling.21:17
*** dimsum__ has joined #openstack-infra21:17
mordredmorganfainberg: :)21:20
*** teran has joined #openstack-infra21:20
*** armax has joined #openstack-infra21:22
*** sdake has quit IRC21:27
jeblairi would have made a turbo button joke, but my computer was too slow21:31
greghaynesfungi: do the hp ubuntu-trusty images perform comprable to the snapshot ones?21:34
greghaynesso we can rule out it being an in-image issue21:35
greghaynesI'm guesing they must or we would have noticed by now21:35
*** salv-orlando has joined #openstack-infra21:38
*** achanda has quit IRC21:38
greghaynesfungi: funny idea, I wonder if you snapshot a node you make from the dib image if that magically fixes the perf issue (per clarkb's hypothesis)21:49
*** fawadkhaliq has quit IRC21:52
*** fawadkhaliq has joined #openstack-infra21:53
openstackgerritMerged openstack-infra/shade: Remove meta.get_server_public_ip() function  https://review.openstack.org/20713921:55
*** fawadkhaliq has quit IRC21:57
*** armax has quit IRC22:00
*** dmellado has quit IRC22:00
*** dmellado has joined #openstack-infra22:02
*** armax has joined #openstack-infra22:05
*** yamahata has joined #openstack-infra22:36
*** achanda has joined #openstack-infra22:44
*** johnbelamaric has joined #openstack-infra22:45
*** achanda has quit IRC22:48
*** johnbelamaric has quit IRC22:48
*** johnbelamaric has joined #openstack-infra22:50
*** sdake has joined #openstack-infra22:53
*** armax has quit IRC22:58
*** markvoelker has joined #openstack-infra22:58
*** btully has joined #openstack-infra22:58
*** achanda has joined #openstack-infra23:02
*** btully has quit IRC23:03
*** markvoelker has quit IRC23:03
*** johnbelamaric has quit IRC23:07
*** btully has joined #openstack-infra23:07
*** rossella_s has quit IRC23:16
*** rossella_s has joined #openstack-infra23:16
*** davideagnello has joined #openstack-infra23:21
*** salv-orlando has quit IRC23:24
*** davideagnello has quit IRC23:26
*** thorst has joined #openstack-infra23:33
*** banix has joined #openstack-infra23:34
*** thorst has quit IRC23:34
*** doude has quit IRC23:37
openstackgerritMerged openstack/requirements: Updated from generate-constraints  https://review.openstack.org/20812623:38
*** doude has joined #openstack-infra23:41
*** sdake has quit IRC23:49
*** mkj_wk has quit IRC23:50
*** skylerberg has joined #openstack-infra23:53
*** mkj_wk has joined #openstack-infra23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!