Wednesday, 2014-02-05

openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Fix os-svc-restart to exit correctly without -n.  https://review.openstack.org/7115600:02
StevenKlifeless: The fact that the top of those scripts looks pretty identical makes me sad00:02
*** ftcjeff has quit IRC00:03
lifelessStevenK: yes00:03
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Fix use of os-svc-restart in cinder.  https://review.openstack.org/7115700:05
*** edmund has quit IRC00:12
SpamapSinteresting00:16
SpamapSso 1hour of purging only gets about 300,000 tokens00:16
SpamapSok, overcloud finally successfully deleted.. starting tripleo-cd00:18
lifelessSpamapS: it won't deploy until the fixes I just pushed are landed00:23
lifelessSpamapS: the wait condition won't fire00:23
*** CaptTofu has joined #tripleo00:24
*** matsuhashi has joined #tripleo00:25
*** CaptTofu has quit IRC00:28
SpamapSlifeless: cripes, you've built a tangled web of dependencies00:33
lifelessSpamapS: blame lack of reviews00:34
StevenKlifeless: ... or that you've pushed up 16,000 patches in 5 minutes?00:37
SpamapSlifeless: https://review.openstack.org/#/c/70873/ has a legitimate -1 on it00:40
lifelessStevenK: several days00:42
*** jcoufal has quit IRC00:42
lifelessSpamapS: I'm working -1's atm00:42
*** ccrouch has joined #tripleo00:42
*** rbrady-backlater is now known as rbrady00:42
*** jcoufal has joined #tripleo00:42
*** ccrouch1 has quit IRC00:43
SpamapSlifeless: the problem is that https://review.openstack.org/#/c/71157/ fixes the cloud, all the other stuff does not.00:45
lifelessmeh00:45
lifelessgimme a sec, whinge whinge whinge00:45
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Fix use of os-svc-restart in cinder.  https://review.openstack.org/7115700:46
openstackgerritA change was merged to openstack/tripleo-image-elements: Fix use of os-svc-restart in cinder.  https://review.openstack.org/7115700:46
*** ccrouch has quit IRC00:46
SpamapSlifeless: :-D00:46
SpamapSlifeless: I promise I'll review all your stuff very soon. :)00:46
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Allow more time for overcloud bring-up.  https://review.openstack.org/7117000:49
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Fix chattr +C of VM images.  https://review.openstack.org/7107500:49
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Support flat-networks for the overcloud.  https://review.openstack.org/7088000:49
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Don't source devtest_end.sh.  https://review.openstack.org/7086100:49
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Reintegrate devtest actions with docs.  https://review.openstack.org/7086000:49
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Long overdue change to use multiple -P's with heat  https://review.openstack.org/7087900:49
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Attempt to make devtest_testenv docs work better.  https://review.openstack.org/7087300:49
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Fix os-svc-restart to exit correctly without -n.  https://review.openstack.org/7115600:50
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Add a script to deploy the ci-overcloud.  https://review.openstack.org/7115400:50
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: os-svc-enable should error when no svcname given.  https://review.openstack.org/7115500:50
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: os-config-applier->os-apply-config for Neutron  https://review.openstack.org/7064000:50
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Use new cleaner setting for OVERCLOUD_LIBVIRT_TYPE  https://review.openstack.org/7087800:50
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Permit injecting entries into /etc/hosts.  https://review.openstack.org/7030700:50
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Permit setting ml2 flat networks in Neutron.  https://review.openstack.org/7064500:50
lifelessSpamapS: you better, or my pushing is going to be > 50% of channel traffic00:50
*** peoplemerge has quit IRC00:53
*** nosnos has joined #tripleo00:57
*** nosnos has quit IRC01:01
*** nosnos has joined #tripleo01:01
*** ccrouch has joined #tripleo01:15
*** mestery has quit IRC01:15
*** cody-somerville has quit IRC01:17
*** markmcclain has quit IRC01:30
SpamapSgah01:30
SpamapSrevoked token list still takes 5 seconds01:30
SpamapSrange queries are the suck :-P01:31
*** CaptTofu has joined #tripleo01:32
StevenKSpamapS: Or you need better indices01:33
SpamapSStevenK: no, there's nothing more we can do in this case.01:33
SpamapSStevenK: the range query is isolating to 11,000 rows from the b-tree01:33
StevenKSchema change01:33
SpamapSbut the filter is too complex for the b-tree so we still have to apply some where logic to those 11,00001:34
StevenKAs a pithy-surely-that-will-fix-everything remark01:34
SpamapSStevenK: the schema is fine, the query is entirely covered by indexes where it can be. We shouldn't ever allow 1.6 million tokens for one tenant.01:34
SpamapSStevenK: in fact I just recently fixed the query so it was entirely covered by indexes. :)01:35
lifelessSpamapS: its a shame its only running mysql :P01:35
SpamapSlifeless: is there something else that would handle this better? :)01:36
SpamapSDrizzle maybe? :)01:36
greghaynesJust run mongodb, im sure itll work out :p01:36
SpamapSits webscale01:36
lifelessSpamapS: btree down to 11000 rows then in-engine logic? postgresql would do that subsecond for sure.01:36
lifelessSpamapS: anyhow, you -know- I'm trolling :)01:37
StevenKHahaha01:37
SpamapSlifeless: I missed a digit, 11516801:37
SpamapShttp://paste.openstack.org/show/6249801:37
lifeless100K - ah thats more reasonable to take 5 seconds01:37
SpamapSactually...... hmmmm01:38
SpamapSIt appears the individual field keys are taking precedence to the compound one01:38
lifelessSpamapS: but surely those checks are all indexable01:38
lifelessSpamapS: query planners, gotta love em01:38
SpamapSlifeless: yeah I was looking at a different query. This is entirely covered by ix_token_expires_valid01:38
SpamapSsome migration must have added the other two back in01:38
SpamapSAhhh.01:38
SpamapSthey landed the fix for the query... but not the migration to drop the old indexes01:39
SpamapSlifeless: yes this is where MySQL 5.6 has now caught up to PostgreSQL in query planning apparently.01:39
SpamapSand MariaDB _WANTS_ to catch up ;)01:39
StevenKSpeaking as a MySQL noob, EXPLAIN ANALYZE in MySQL is far different from Postgres?01:40
SpamapSthere is no explain analyze AFAIK01:41
greghaynes:o01:41
StevenKSpamapS: WAT01:41
lifelessStevenK: postgresql's explain is the awesome01:41
SpamapSWhat you see is what you get.01:41
* greghaynes is also pgsql'er but mysql noob01:41
*** morazi has quit IRC01:41
lifelessSpamapS: so nothing at all ? :P01:42
* StevenK glares at mordred's "postgres gives us nothing" statement01:43
StevenKClearly, it gives us EXPLAIN01:43
SpamapSyou get explain01:43
SpamapSand it tells you what indexes it is using01:44
SpamapShow it is using them01:44
SpamapSwhat happens after the indexes01:44
SpamapSWhat you don't get is all the magic of the optimizer exposed to you.01:44
lifelessSpamapS: you don't get that in pg either01:45
lifelessSpamapS: what you get is a trace of the request01:45
lifelessSpamapS: there's a teeny VM in postgresql (like sqlite actually :)) that has things like 'walk a btree' or 'probe a hash'01:46
lifelessSpamapS: so the planner actually compiles a program in that VM language01:46
SpamapSyeah.. not so much in mysql :)01:46
lifelessSpamapS: and explain decompiles that01:46
lifelessSpamapS: explain analyze runs it with instrumentation to give timing data01:46
lifelessSpamapS: if you haven't used it in 9.1 or newer, you -really- should.01:47
lifelessSpamapS: its mind blowingly good01:47
SpamapSah so the thing that is causing the 'Using where' in my pastebinned explain is token.user_id01:47
SpamapSBecause we have no compound index on that, when we delete all the tokens for a user, we have to comb through _all_ tokens basically.01:47
SpamapSlifeless: I've only supported one production pgsql db.. it was 7.2 I think.01:48
StevenKSpamapS: Twitch01:48
lifelessStevenK: yeah, totally different01:48
SpamapSand it was the bane of all of our existence.. but only because the devs just didn't know how to deal with pgsql.01:48
StevenKIMO, postgres didn't get good until 7.401:49
SpamapShm01:49
StevenKI've supported production databases from 8.2 until 9.101:49
SpamapSI kind of wonder if we should submit a change to keystone to add an index for this delete01:49
SpamapSbecause even if we do keep on top of the token maintenance... :-P01:50
StevenKSpamapS: But as lifeless says, if you've not used EXPLAIN with >=9.1, do so, it is totally awesome01:50
SpamapSone thing that kind of drives me nuts is these orm driven queries01:52
StevenKSpamapS: You can't say anything bad about sqla.01:52
StevenKBecause it isn't Storm.01:52
SpamapSthey just encourage bad practices01:53
SpamapSlike "go get me the entire object". when all you need is the id01:54
StevenKSpamapS: I don't think that's the ORM's fault.01:54
StevenKI think that could be better solved by documentation and training.01:54
lifelesswithout the ORM01:56
lifelessthere is no default :)01:56
*** TravT has quit IRC02:04
greghaynessqla should be smarter than that02:07
greghaynesSo is the way to start grokking these heat templates to dive into cloud formation tutorial?02:08
SpamapSgreghaynes: there's a template repo on git.openstack.org (openstack/heat-templates)02:09
greghaynesooo02:10
*** ccrouch has quit IRC02:16
*** AaronGr is now known as AaronGr_Zzz02:21
*** coolsvap_away has quit IRC02:34
*** spzala has quit IRC02:36
*** jcoufal has quit IRC02:42
*** jcoufal has joined #tripleo02:43
StevenKlifeless: *prod* re 7023902:44
lifelessSpamapS: there is another init bug02:45
StevenKlifeless: I've also twiddled inject_partition to -2, but I'm not sure how to test it02:45
lifelessStevenK: git review, if it succeeds, it worked.02:46
lifelessStevenK: what do you mean by 'also twiddled' - that patch changes the default..02:46
StevenKlifeless: You changed inject_key to false, which will stop injecting the ssh public key02:47
StevenKSetting inject_partition to -2 disables the entire injection mechanism02:47
lifelessStevenK: the patch did both things02:49
StevenKlifeless: The patch on gerrit only touches inject_key, not inject_partition02:49
lifelesswaaat02:49
lifelessI messed uppatch set 3 apparently02:50
StevenKAh yeah02:51
StevenKYour line counts halve02:51
StevenKlifeless: Shall I git review what I have, to save you context switching, or you've already paid the context switching price?02:52
lifelessStevenK: please get it up there02:52
*** CaptTofu has quit IRC02:53
*** vkozhukalov has joined #tripleo02:53
*** CaptTofu has joined #tripleo02:55
*** jcoufal has quit IRC02:59
*** morazi has joined #tripleo03:12
*** edmund has joined #tripleo03:43
*** cwolferh has quit IRC03:57
*** cwolferh has joined #tripleo03:59
*** CaptTofu has quit IRC04:09
*** CaptTofu has joined #tripleo04:09
*** CaptTofu has quit IRC04:14
*** ccrouch has joined #tripleo04:15
*** killer_prince has joined #tripleo04:19
*** killer_prince has quit IRC04:31
*** killer_prince has joined #tripleo04:37
*** akuznetsov has joined #tripleo04:56
*** marun has quit IRC05:02
*** marun has joined #tripleo05:03
*** tzumainn has quit IRC05:16
*** jcooley_ has quit IRC05:19
*** killer_prince has quit IRC05:20
*** morazi has quit IRC05:26
*** mestery has joined #tripleo05:38
*** mestery has joined #tripleo05:39
*** killer_prince has joined #tripleo05:41
*** markmcclain has joined #tripleo05:41
*** mestery_ has joined #tripleo05:44
*** markmcclain has quit IRC05:45
*** mestery has quit IRC05:46
*** jcooley_ has joined #tripleo05:48
*** mestery_ has quit IRC06:00
*** weshay has quit IRC06:02
*** killer_prince has quit IRC06:03
*** killer_prince has joined #tripleo06:04
*** e0ne has joined #tripleo06:04
*** killer_prince has quit IRC06:05
*** killer_prince has joined #tripleo06:05
*** killer_prince has quit IRC06:05
*** killer_prince has joined #tripleo06:08
*** e0ne has quit IRC06:26
*** rlandy has joined #tripleo06:35
*** markmcclain has joined #tripleo06:50
*** boris-42_ has quit IRC07:05
*** uvirtbot has joined #tripleo07:09
*** coolsvap has joined #tripleo07:09
*** jcooley_ has quit IRC07:12
*** jcooley_ has joined #tripleo07:12
*** jcooley_ has quit IRC07:17
*** pblaho has joined #tripleo07:30
openstackgerritGregory Haynes proposed a change to openstack/tripleo-heat-templates: Allow scaling of notCompute nodes  https://review.openstack.org/7119807:36
*** cwolferh has quit IRC07:45
*** jprovazn has joined #tripleo07:54
*** jistr has joined #tripleo08:08
*** CaptTofu has joined #tripleo08:11
*** CaptTofu has quit IRC08:16
*** jtomasek has joined #tripleo08:18
*** hashar has joined #tripleo08:19
*** mrunge has joined #tripleo08:20
*** jtomasek has quit IRC08:20
*** jtomasek has joined #tripleo08:22
*** markmcclain has quit IRC08:22
*** bauzas has joined #tripleo08:26
*** akuznetsov has quit IRC08:28
*** akuznetsov has joined #tripleo08:31
*** ifarkas has joined #tripleo08:40
*** ifarkas_ has joined #tripleo08:42
*** ifarkas has quit IRC08:44
*** lsmola has quit IRC08:46
*** matsuhashi has quit IRC08:54
*** matsuhashi has joined #tripleo08:55
*** jcoufal has joined #tripleo08:55
*** e0ne has joined #tripleo08:56
*** lsmola has joined #tripleo08:59
*** gcha has joined #tripleo09:03
*** derekh has joined #tripleo09:05
*** fungi has quit IRC09:12
*** dtantsur has joined #tripleo09:14
*** lucasagomes has joined #tripleo09:16
*** fungi has joined #tripleo09:16
*** ChanServ sets mode: +v fungi09:16
*** lsmola has quit IRC09:19
lifelessand now var/log/upstart/nova-api.log:2014-02-05 09:19:49.834 3889 ERROR nova.api.openstack [req-61a0a806-76e9-4835-a761-387055d50e59 1d4dbedad355438888dcb1e1af82f64d 962e421a84504665b6bc4e3603ab2805] Caught error: Connection to neutron failed: Maximum attempts reached09:22
lifelesssigh09:22
*** boris-42_ has joined #tripleo09:23
*** gcha_ has joined #tripleo09:29
*** gcha has quit IRC09:29
*** d0ugal has quit IRC09:29
*** d0ugal has joined #tripleo09:29
*** gcha_ is now known as gcha09:30
*** lsmola has joined #tripleo09:32
lifelessgnnnar /dev/mapper/lifelesshp-root  168G  164G  2.7G  99% /09:33
StevenKlifeless: Did you get woken by C?09:33
lifelessStevenK: its only 223309:33
lifelessStevenK: I'm still fighting with deploying 3 nodes09:34
StevenKlifeless: Ah, I thought it was 233309:34
lifelessits amazing how bad btrfs is09:34
StevenKlifeless: I don't trust it yet.09:35
StevenKPlus the 'btrfs df' garbage makes me sigh09:35
StevenKlifeless: You saw markwash's mail yesterday about the mid-cycle?09:36
lifelessno09:37
derekhlifeless: so we have no ci-overcloud, is that what your looking at or do you want me to see if I can help out?09:38
*** e0ne_ has joined #tripleo09:38
lifelessderekh: yeah, I am - I'm trying to fix overcloud deploys first - found a stack of bugs.09:39
lifelessderekh: so - making sure you can deploy a 2-compute node overcloud would be a great step09:39
derekhlifeless: ok, will got it a whirl09:40
lifelessderekh: either locally or in the DC, I've been working locally09:40
lifelessderekh: I've pushed the beginnings of a script to automate ci-overcloud deployment09:40
lifelessderekh: amongst other things09:40
lifelessderekh: I suggest poking around all my pending reviews first :P09:40
lifelessderekh: did I give you a copy of the TODO for ci-overcloud deploy automation?09:41
derekhlifeless: ok will do, will work locally at first and see how far I get,09:41
derekhlifeless: I don't think so09:41
*** e0ne has quit IRC09:41
lifelessderekh: dumped it in the bottom of tripleo-test-cluster09:42
*** julim has joined #tripleo09:42
lifelessderekh: ok, just checked everything I'd committed is pushed up (more or less :))09:44
lifelessof course, nova being painfully slow didn't help, nor running out of disk space...09:45
lifelessso more bug fixes09:45
derekhlifeless: ok, running a master devtest at the moment, will kick off another with your current branches and see how far I get09:45
lifelessderekh: there's only about 20 outstanding patches :P09:45
lifelessplus https://review.openstack.org/71214 when I get time to polish it09:45
lifelessor someone takes me up on my offer to get more nova cred :P09:46
openstackgerritJiri Tomasek proposed a change to openstack/tuskar-ui: Updated Nodes Overview page  https://review.openstack.org/6938009:51
*** jcoufal has quit IRC09:54
lifelessand for more fun https://review.openstack.org/7121910:00
*** max_lobur_afk is now known as max_lobur10:04
*** cody-somerville has joined #tripleo10:05
*** CaptTofu has joined #tripleo10:12
lifelessStevenK: ^ those nova patches (214 and 219 above) might be good things for you to see a bit more of the terrible terrible plumbing at work10:15
lifelessStevenK: if you have spare slots to poke at stuff10:16
lifelessStevenK: (and nothing better to do :)10:16
*** CaptTofu has quit IRC10:16
StevenKlifeless: Hm, I have a test failure to fix tomorrow morning. Trying to get the injection test actually passing when the default is false.10:20
lifelessStevenK: you will need to change the test to enable the feature10:21
lifelessStevenK: there is glue in the unittests for setting config options10:21
lifelessStevenK: and a fixture that resets things afterwards mostly automatically10:21
lifelessstab neutron stab10:22
lifelessrighto, night all10:23
*** jcoufal has joined #tripleo10:23
openstackgerritA change was merged to openstack/tripleo-image-elements: Permit injecting entries into /etc/hosts.  https://review.openstack.org/7030710:26
*** ava1 has joined #tripleo10:27
*** ava1 is now known as Guest4115410:27
*** boris-42_ has quit IRC10:28
*** markmc has joined #tripleo10:31
*** Guest41154 has quit IRC10:32
lifelessSpamapS: I probably need to get a fix to https://bugs.launchpad.net/heat/+bug/1276517 to get around this ridiculous slow-IO problem10:34
uvirtbotLaunchpad bug 1276517 in heat "failure to poll for server creation treated as failure to create" [Undecided,New]10:34
lifelessSpamapS: suggestions on where in the code to tackle it much appreciated10:34
* lifeless really goes to bed now10:34
*** e0ne_ has quit IRC10:35
*** e0ne has joined #tripleo10:35
*** jistr has quit IRC10:40
*** boris-42_ has joined #tripleo10:40
*** martyntaylor has joined #tripleo10:42
*** nosnos has quit IRC10:53
*** akrivoka has joined #tripleo10:57
*** pblaho has quit IRC10:58
*** jistr has joined #tripleo11:00
openstackgerritJan Provaznik proposed a change to openstack/tripleo-heat-templates: Add rabbit.cookie attribute  https://review.openstack.org/7123311:01
openstackgerritJan Provaznik proposed a change to openstack/tripleo-heat-templates: Add rabbit.cookie attribute  https://review.openstack.org/7023811:04
*** jtomasek has quit IRC11:05
*** jtomasek has joined #tripleo11:05
openstackgerritAna Krivokapic proposed a change to openstack/tripleo-incubator: Make sure the path of default storage pool is used  https://review.openstack.org/7105211:24
*** panda___ has quit IRC11:32
openstackgerritA change was merged to openstack/tripleo-image-elements: os-config-applier->os-apply-config for Neutron  https://review.openstack.org/7064011:32
openstackgerritA change was merged to openstack/tripleo-image-elements: Permit setting ml2 flat networks in Neutron.  https://review.openstack.org/7064511:35
*** panda___ has joined #tripleo11:45
*** lsmola has quit IRC11:46
*** gcha has quit IRC11:57
*** lsmola has joined #tripleo12:01
*** CaptTofu has joined #tripleo12:07
*** hashar has quit IRC12:24
*** coolsvap has quit IRC12:33
*** CaptTofu has quit IRC12:36
*** CaptTofu has joined #tripleo12:36
*** cody-somerville has quit IRC12:39
*** CaptTofu has quit IRC12:41
*** lucasagomes is now known as lucas-hungry12:43
*** dtantsur has left #tripleo12:49
*** lsmola has quit IRC12:53
*** pblaho has joined #tripleo12:53
*** dtantsur has joined #tripleo12:55
*** lsmola has joined #tripleo12:56
*** lblanchard has joined #tripleo13:00
lsmolaslagle: hello, are you around?13:05
*** hashar has joined #tripleo13:08
slaglelsmola: i am13:11
*** dprince has joined #tripleo13:11
lsmolaslagle: hello13:12
slaglehi, what's up?13:12
lsmolaslagle: just wanted to ask, do you still have architecture images from this email?13:12
lsmolaslagle: http://lists.openstack.org/pipermail/openstack-dev/2013-September/014754.html13:12
lsmolaslagle: I would like to use it for presentation :-)13:12
slaglei suspect i do somewhere13:13
lsmolaslagle: excellent13:15
*** gcha has joined #tripleo13:17
*** CaptTofu has joined #tripleo13:17
slaglelsmola: ok, those links should work now13:18
lsmolaslagle: excellent, thank you very much13:18
slaglelsmola: bear in mind, the images are a little dated at this point. probably still applicable for high level overview though13:18
*** cody-somerville has joined #tripleo13:18
lsmolaslagle: yep, high level is great :-)13:18
*** michchap has quit IRC13:21
*** michchap has joined #tripleo13:22
openstackgerritDirk Mueller proposed a change to openstack/diskimage-builder: Update openSUSE package mappings for OpenStack  https://review.openstack.org/7017413:25
*** michchap has quit IRC13:26
*** panda___ is now known as panda13:27
*** panda is now known as Guest6112313:27
*** Guest61123 is now known as panda13:28
*** jdob has joined #tripleo13:30
openstackgerritDirk Mueller proposed a change to openstack/tripleo-incubator: initialize Keystone database on openSUSE  https://review.openstack.org/7126513:31
Ngmorning13:42
*** CaptTofu has quit IRC13:46
*** lucas-hungry is now known as lucasagomes13:48
*** CaptTofu has joined #tripleo13:50
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Install syslinux in nova-barmetal  https://review.openstack.org/7071813:53
*** peoplemerge has joined #tripleo13:54
*** tzumainn has joined #tripleo14:01
openstackgerritA change was merged to openstack/python-tuskarclient: Add bindings for overcloud API entrypoints  https://review.openstack.org/6931214:03
*** ftcjeff has joined #tripleo14:12
akrivokamy devstack timed out while waiting for overcloud to become ready14:12
akrivokahere's some output from commands ran on the undercloud: http://paste.openstack.org/show/62587/14:13
akrivokaany tips on how to proceed debugging this? where can I find the logs on the undercloud VM?14:13
*** mrunge has quit IRC14:14
shadowerakrivoka: I seem to remember a discussion regarding nova timeouts on yesterday's meeting14:14
* shadower tries to dig up the relevant info14:14
shadowerakrivoka: yea I think it's the same issue. marios can you confirm? ^14:16
shadowerif that's the case, lifeless is looking into it iirc14:16
ifarkas_akrivoka, can you find anything useful in the overcloud machine log?14:19
akrivokaifarkas_: where should I look?14:22
ifarkas_akrivoka, journalctl is good start14:22
*** michchap has joined #tripleo14:23
openstackgerritJan Provaznik proposed a change to openstack/tripleo-image-elements: Update openstack config files to support rabbitmq cluster  https://review.openstack.org/6839314:27
openstackgerritJan Provaznik proposed a change to openstack/tripleo-image-elements: Fix rabbitmq env file name and log path  https://review.openstack.org/7023614:27
openstackgerritJan Provaznik proposed a change to openstack/tripleo-image-elements: Add cluster support for rabbitmq element  https://review.openstack.org/6839214:28
*** michchap has quit IRC14:28
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: openSUSE package install support for neutron  https://review.openstack.org/7072114:28
*** matty_dubs|gone is now known as matty_dubs14:29
*** jprovazn has quit IRC14:30
*** boris-42_ has quit IRC14:31
*** weshay has joined #tripleo14:34
akrivokaifarkas_: found this, the time stamp coincides with the timeout from heat event-list http://paste.openstack.org/show/62599/14:35
*** jayg|g0n3 is now known as jayg14:36
openstackgerritJay Dobies proposed a change to openstack/tuskar: Resource category model changes for Icehouse.  https://review.openstack.org/7127514:51
*** mestery has joined #tripleo14:52
*** mestery has quit IRC14:53
*** mestery has joined #tripleo14:53
*** jprovazn has joined #tripleo14:54
*** morazi has joined #tripleo14:59
openstackgerritJay Dobies proposed a change to openstack/tuskar: Resource category model changes for Icehouse  https://review.openstack.org/7127515:01
*** mestery has quit IRC15:10
*** matsuhashi has quit IRC15:17
*** dprince has quit IRC15:18
*** matsuhashi has joined #tripleo15:18
*** dprince has joined #tripleo15:19
*** michchap has joined #tripleo15:24
openstackgerritDougal Matthews proposed a change to openstack/tuskar: Update the HTTP 500 JSON structure to match oslo  https://review.openstack.org/7128815:24
openstackgerritDougal Matthews proposed a change to openstack/tuskar: Update the HTTP 500 JSON structure to match oslo  https://review.openstack.org/7128815:25
*** hewbrocc` has joined #tripleo15:26
*** hewbrocca has quit IRC15:28
*** michchap has quit IRC15:28
*** coolsvap has joined #tripleo15:29
*** boris-42_ has joined #tripleo15:32
*** matsuhashi has quit IRC15:34
*** morazi has quit IRC15:38
*** hewbrocc` has quit IRC15:45
openstackgerritA change was merged to openstack/tuskar: Resource category model changes for Icehouse  https://review.openstack.org/7127515:48
derekhlifeless: ran your branches and both nova compute nodes on overcloud died with same error, CRITICAL nova [req-47c8e8be-995b-4df9-9e02-ca264a74a8c8 None None] UnboundLocalError: local variable 'e' referenced before assignment15:48
derekhlifeless: I got no other traceback and starting it a second time seems to go ok.... will see what I can find out15:49
openstackgerritDougal Matthews proposed a change to openstack/tuskar: Update the HTTP 500 JSON structure to match oslo  https://review.openstack.org/7128815:49
*** max_lobur is now known as max_lobur_afk15:49
*** morazi has joined #tripleo15:51
*** pblaho has quit IRC15:57
*** rlandy has quit IRC16:01
SpamapShmmm16:11
SpamapSI just realized..16:11
SpamapSwe should cache tokens in os-collect-config16:11
SpamapSoh, we don't use tokens16:13
*** hewbrocca has joined #tripleo16:17
*** mestery has joined #tripleo16:18
*** pblaho has joined #tripleo16:21
*** pblaho has quit IRC16:21
*** jcooley_ has joined #tripleo16:21
*** pblaho has joined #tripleo16:21
*** michchap has joined #tripleo16:23
*** michchap has quit IRC16:28
*** jcooley_ has quit IRC16:28
*** AaronGr_Zzz is now known as AaronGr16:29
*** weshay is now known as weshay_bbiab16:30
*** jcooley_ has joined #tripleo16:33
*** mestery has quit IRC16:40
SpamapSwoot, landed a fix in cliff so it can escape double quotes in the shell formatter16:40
SpamapSgo me16:40
*** bauzas has quit IRC16:40
openstackgerritDougal Matthews proposed a change to openstack/tuskar: Update the HTTP 500 JSON structure to match oslo  https://review.openstack.org/7128816:45
*** taps has joined #tripleo16:46
*** cwolferh has joined #tripleo16:46
*** jcoufal has quit IRC16:47
*** CaptTofu has quit IRC16:48
*** pblaho has quit IRC16:53
*** pblaho has joined #tripleo16:54
openstackgerritLadislav Smola proposed a change to openstack/tuskar-ui: Adding nova-baremetal, heat and nova API  https://review.openstack.org/6897316:56
*** athomas has joined #tripleo16:59
*** markmcclain has joined #tripleo17:01
*** markmcclain has quit IRC17:02
*** markmcclain has joined #tripleo17:03
*** e0ne has quit IRC17:03
*** michchap has joined #tripleo17:09
*** ifarkas_ has quit IRC17:10
*** bogdando has quit IRC17:11
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Avoid adding qpid-python twice  https://review.openstack.org/7072317:11
fungianybody happen to have updated details on the tripleo-ci cloud rebuild status? just wondering whether it's safe to reenable it in our nodepool yet17:12
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Fix python-ceilometerclient package install  https://review.openstack.org/7113617:13
*** michchap has quit IRC17:14
*** julim has quit IRC17:15
*** markmc has quit IRC17:19
*** jcooley_ has quit IRC17:22
*** jcooley_ has joined #tripleo17:23
*** jistr has quit IRC17:26
*** funzo has joined #tripleo17:33
*** athomas has quit IRC17:33
*** CaptTofu has joined #tripleo17:33
*** funzo_ has quit IRC17:34
*** jcooley_ has quit IRC17:34
*** jcooley_ has joined #tripleo17:34
openstackgerritDougal Matthews proposed a change to openstack/tuskar: Update the HTTP 500 JSON structure to match oslo  https://review.openstack.org/7128817:37
*** jcooley_ has quit IRC17:41
*** peoplemerge has quit IRC17:42
*** jcooley_ has joined #tripleo17:42
*** peoplemerge has joined #tripleo17:42
*** peoplemerge has quit IRC17:42
* Ng -> kids/dinner/bedtime duties, back in a couple of hours17:46
openstackgerritDirk Mueller proposed a change to openstack/diskimage-builder: Update openSUSE package mappings for OpenStack  https://review.openstack.org/7017417:48
*** hashar has quit IRC17:53
*** matty_dubs is now known as matty_dubs|lunch17:53
openstackgerritAna Krivokapic proposed a change to openstack/tripleo-image-elements: Account for amd64 architecture in snmpd element  https://review.openstack.org/7017117:56
*** pblaho has quit IRC17:57
*** akuznetsov has quit IRC17:58
*** derekh has quit IRC18:00
*** funzo_ has joined #tripleo18:01
*** funzo has quit IRC18:02
SpamapS2014-02-05 17:20:50,183.183 4763 ERROR nova.api.openstack [req-5e4a374b-90e7-4c1c-b22e-577428cd7b60 2de96ce5da994575a08447c93bcafd53 4956c533154c476799c688eda7ed65ab] Caught error: Connection to neutron failed: Maximum attempts reached18:03
*** cody-somerville has quit IRC18:07
*** michchap has joined #tripleo18:10
*** michchap has quit IRC18:10
*** michchap has joined #tripleo18:11
*** michchap has quit IRC18:15
*** jcooley_ has quit IRC18:19
*** lucasagomes has quit IRC18:21
*** jcooley_ has joined #tripleo18:21
*** jcooley_ has quit IRC18:23
*** jcooley_ has joined #tripleo18:23
*** hewbrocca has quit IRC18:25
*** gcha has quit IRC18:27
*** weshay_bbiab is now known as weshay18:36
*** peoplemerge has joined #tripleo18:40
*** hewbrocca has joined #tripleo18:42
*** akuznetsov has joined #tripleo18:43
*** matty_dubs|lunch is now known as matty_dubs18:44
*** bauzas has joined #tripleo18:48
*** jprovazn has quit IRC18:50
*** lynxman has quit IRC18:51
*** lynxman_ has joined #tripleo18:51
*** lynxman_ is now known as lynxman18:51
*** lynxman has quit IRC18:51
*** lynxman has joined #tripleo18:51
*** gcha has joined #tripleo18:54
*** markmcclain has quit IRC18:55
openstackgerritJay Dobies proposed a change to openstack/tuskar: Renamed resource category to overcloud role  https://review.openstack.org/7133618:59
*** weshay has quit IRC19:07
*** jcooley_ has quit IRC19:07
*** jcooley_ has joined #tripleo19:08
*** edmund has quit IRC19:09
*** michchap has joined #tripleo19:11
*** cd-undercloud has joined #tripleo19:11
cd-undercloud************** overcloud complete status=1 ************19:11
*** cd-undercloud has quit IRC19:11
*** michchap has quit IRC19:11
*** michchap has joined #tripleo19:11
*** coolsvap is now known as coolsvap_away19:15
*** michchap has quit IRC19:16
openstackgerritJiri Tomasek proposed a change to openstack/tuskar-ui: Updated Nodes Overview page  https://review.openstack.org/6938019:20
*** weshay has joined #tripleo19:24
*** ftcjeff has quit IRC19:28
openstackgerritDirk Mueller proposed a change to openstack/diskimage-builder: Update openSUSE package mappings for OpenStack  https://review.openstack.org/7017419:32
*** markmcclain has joined #tripleo19:40
openstackgerritA change was merged to openstack/tuskar-ui: Updated Nodes Overview page  https://review.openstack.org/6938019:41
*** akuznetsov has quit IRC19:43
*** spzala has joined #tripleo19:43
*** akrivoka has quit IRC19:47
*** edmund has joined #tripleo19:48
*** e0ne has joined #tripleo19:55
*** edmund has quit IRC19:56
*** shakayumi has joined #tripleo20:00
*** shakayum_ has joined #tripleo20:02
lifelessdprince: hey, today is a public holiday for me20:03
hewbroccalifeless day?20:03
lifelessdprince: so I'll be missing the hangout probably20:03
lifelesshewbrocca: sadly now, Waitangi day20:03
* hewbrocca was all set to be very impressed20:04
dprincelifeless: cool, I think we'll take this one off anyways20:04
*** shakayumi has quit IRC20:05
lifelessSpamapS: did you get my meeting handoff mail request?20:05
lifelesstwitch : /dev/mapper/lifelesshp-root  168G  166G  437M 100% /20:06
Nglifeless: on the assumption that a holiday day means you're probably not reading work mail - are we go for sunnyvale flights yet? :)20:06
lifelessNg: thats pending cody-somervilles arrangement of group rates20:07
Nglifeless: k, just wanted to check I hadn't missed a gogogo20:08
lifelessnonono20:08
lifelesswoawoawoa20:08
NgI'd say "my flights aren't going to be cheap", but I imagine we have to sell a couple of divisions to cover yours ;)20:09
lifelessLOL20:09
lifelessNg: actually you or me -> sunnyvale is about equal20:09
lifelessdistance wise20:09
*** julim has joined #tripleo20:11
*** michchap has joined #tripleo20:11
*** martyntaylor has left #tripleo20:12
*** michchap has quit IRC20:12
*** michchap has joined #tripleo20:12
Nglifeless: yeah20:13
NgI peeked at options the other day and it was $1600-$3200 depending on the times ;o20:13
*** jtomasek has quit IRC20:14
openstackgerritDan Prince proposed a change to openstack/tripleo-incubator: Add -m memory option to setup-seed-vm  https://review.openstack.org/7137020:14
openstackgerritDan Prince proposed a change to openstack/tripleo-incubator: Add -c cpus option to setup-seed-vm.  https://review.openstack.org/7137120:14
*** markmcclain has quit IRC20:15
*** michchap has quit IRC20:17
*** cd-undercloud has joined #tripleo20:22
cd-undercloud************** overcloud complete status=1 ************20:22
*** cd-undercloud has quit IRC20:22
lifelessok, as I said, pub holiday here - gotta go do family stuff20:40
*** vkozhukalov has quit IRC20:40
dprincelifeless: go get your Waitangi skirt on!20:41
* dprince wishes he had one20:41
*** hashar has joined #tripleo20:43
*** akrivoka has joined #tripleo20:45
*** rlandy has joined #tripleo20:50
lifelesswin 5321:00
*** dprince has quit IRC21:05
*** CaptTofu_ has joined #tripleo21:06
*** TravT has joined #tripleo21:07
*** CaptTofu has quit IRC21:09
*** michchap has joined #tripleo21:12
SpamapSdurn where's dprince when you need him21:12
*** funzo_ is now known as funzo21:12
*** michchap has quit IRC21:13
SpamapSsitting here trying to run through devtest with real hardware21:13
*** michchap has joined #tripleo21:13
*** rlandy has quit IRC21:14
*** michchap has quit IRC21:17
*** cwolferh has quit IRC21:19
*** rollerj has joined #tripleo21:23
*** blamar has joined #tripleo21:25
*** blamar has quit IRC21:26
*** blamar has joined #tripleo21:26
*** gcha has quit IRC21:27
*** cd-undercloud has joined #tripleo21:31
cd-undercloud************** overcloud complete status=1 ************21:31
*** cd-undercloud has quit IRC21:31
*** gcha has joined #tripleo21:33
*** rbrady is now known as rbrady-afk21:35
*** gcha has quit IRC21:37
*** markmcclain has joined #tripleo21:43
*** markmcclain1 has joined #tripleo21:43
*** markmcclain has quit IRC21:47
*** hashar has quit IRC21:48
*** edmund1 has joined #tripleo21:51
ccrouchSpamapS: are you "releasing" all the things this week? Or was that shadower?21:53
ccrouchi think i quoted the wrong phrase there21:53
ccrouchreleasing "all the things"21:54
*** mestery has joined #tripleo21:54
*** EntropyWorks has left #tripleo21:54
*** EntropyWorks has joined #tripleo21:55
*** jdob has quit IRC22:07
*** cody-somerville has joined #tripleo22:10
*** michchap has joined #tripleo22:13
*** michchap has quit IRC22:13
*** michchap has joined #tripleo22:14
openstackgerritDirk Mueller proposed a change to openstack/tripleo-incubator: initialize Keystone database on openSUSE  https://review.openstack.org/7126522:16
*** bauzas has quit IRC22:17
*** Shrews has quit IRC22:18
*** michchap has quit IRC22:19
*** michchap has joined #tripleo22:22
*** markmcclain1 has quit IRC22:23
*** matty_dubs is now known as matty_dubs|gone22:25
*** mestery has quit IRC22:26
openstackgerritA change was merged to openstack/tripleo-image-elements: Use new cleaner setting for OVERCLOUD_LIBVIRT_TYPE  https://review.openstack.org/7087822:27
*** e0ne has quit IRC22:28
openstackgerritA change was merged to openstack-infra/tripleo-ci: Get logs for seed vm  https://review.openstack.org/6982522:30
*** markmcclain has joined #tripleo22:34
openstackgerritA change was merged to openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/7096422:34
*** markmcclain1 has joined #tripleo22:36
*** markmcclain1 has quit IRC22:36
*** markmcclain1 has joined #tripleo22:36
*** markmcclain1 has joined #tripleo22:37
*** markmcclain1 has quit IRC22:37
*** markmcclain has quit IRC22:39
*** lblanchard has quit IRC22:43
*** CaptTofu_ has quit IRC22:45
*** cd-undercloud has joined #tripleo22:46
cd-undercloud************** overcloud complete status=1 ************22:46
*** cd-undercloud has quit IRC22:46
SpamapSccrouch: I will do the release thing tonight or tomorrow (next 24h)22:47
ccrouchSpamapS: ok cool, not trying to rush you :-)22:49
openstackgerritA change was merged to openstack/diskimage-builder: Add mysql mapping for Fedora  https://review.openstack.org/7008122:50
*** shakayum_ has quit IRC22:54
* Ng ejects22:57
*** jayg is now known as jayg|g0n323:02
*** weshay has quit IRC23:06
SpamapShm23:07
SpamapSI'm trying to bridge a seed VM onto a network with some machines23:08
*** lblanchard has joined #tripleo23:17
*** lblanchard has quit IRC23:19
*** morazi has quit IRC23:19
*** CaptTofu has joined #tripleo23:27
*** jhurlbert has joined #tripleo23:31
*** e0ne has joined #tripleo23:40
*** julim has quit IRC23:42
*** e0ne has quit IRC23:44
*** cd-undercloud has joined #tripleo23:53
cd-undercloud************** overcloud complete status=1 ************23:53
*** cd-undercloud has quit IRC23:53
*** epim has joined #tripleo23:58
*** jhurlbert has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!