Friday, 2013-11-22

*** michchap_ has joined #tripleo00:01
SpamapSlifeless: deleted baremetal node that pointed to it. It was being chosen "next" all the time.. then failing, then the thing would move on to the next that presumably works better. Not definitive proof yet, but the machine does look hosed00:02
*** michchap has quit IRC00:03
*** toci-bot has joined #tripleo00:04
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_FnRhGRy/00:04
*** toci-bot has quit IRC00:04
*** asalkeld has joined #tripleo00:05
*** csd has quit IRC00:08
lifelessSpamapS: have you pdated the machine-inforamtion-tab.txt file for it?00:08
SpamapSlifeless: not yet no still investigating00:09
*** csd has joined #tripleo00:10
lifelessSpamapS: please do - just a 'am looking' so that it doesn't get added back in or anything00:11
SpamapSyeah I left a note in the in use by00:11
lifelessthanks00:13
SpamapSlifeless: running into "EOD stage 1" .. but i am hopeful. :)00:15
SpamapS"Waiting for link-up on net0" ... slooooowww.00:16
SpamapSand now freecloud0033 is getting the PXE, but failing trying to load pxelinux.cfg00:23
SpamapSTrying to load: pxelinux.cfg/01-78-e7-d1-23-94-15                   ok00:23
SpamapSBoot failed: press a key to retry, or wait for reset..00:23
lifelessone thing to check is the mac address00:29
lifelessmake sure the port is registered in neutron00:29
lifeless(78-e7-d1-23-94-15)00:29
SpamapSit is..00:30
SpamapSand in fact we get the right DHCP address00:30
lifelessthe pxe config file isn't available?00:31
SpamapSit is available, and it gets downloaded ^^^ "    ok"00:31
*** csd has quit IRC00:33
SpamapSI think maybe ilo textcons was confusing me00:33
*** csd has joined #tripleo00:33
SpamapStrying conman00:33
SpamapSbut.. out of time. :(00:33
lifelessohbai00:33
SpamapSoh and now it booted00:36
SpamapSlifeless: I still wonder about flakiness00:36
lifelessI am strongly suspicious of the mellanox driver situation00:39
*** cd-undercloud has joined #tripleo00:44
cd-undercloud************** overcloud complete status=2 ************00:44
*** cd-undercloud has quit IRC00:44
*** jayg is now known as jayg|g0n300:50
asalkeldhi guys, do you still have auto generated cfntools guest images?00:50
asalkeldwhat's the new version of: http://jenkins.tripleo.org:8080/job/autobuilt-images/elements=fedora%20vm%20heat-cfntools/lastSuccessfulBuild/artifact/fedora-vm-heat-cfntools.qcow200:51
lifelessasalkeld: we don't at the moment00:52
asalkeld:(00:52
*** michchap_ has quit IRC00:56
*** michchap has joined #tripleo00:59
*** rongze has joined #tripleo01:23
*** rongze has quit IRC01:27
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface  https://review.openstack.org/5777601:33
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface  https://review.openstack.org/5777601:35
*** boris-42 has joined #tripleo01:36
*** nosnos has joined #tripleo01:40
*** arata has joined #tripleo01:40
*** nosnos has quit IRC01:40
*** nosnos has joined #tripleo01:41
*** cd-undercloud has joined #tripleo01:52
cd-undercloud************** overcloud complete status=1 ************01:52
*** cd-undercloud has quit IRC01:52
*** boris-42 has quit IRC01:55
*** boris-42 has joined #tripleo01:56
*** boris-42 has quit IRC01:58
*** morazi has quit IRC02:02
*** rongze has joined #tripleo02:03
*** csd has quit IRC02:10
*** csd has joined #tripleo02:11
*** ccrouch has quit IRC02:11
*** ccrouch has joined #tripleo02:12
*** ccrouch has quit IRC02:16
*** boris-42 has joined #tripleo02:17
*** victor_lowther has quit IRC02:32
*** edmund has quit IRC02:44
*** cd-undercloud has joined #tripleo02:52
cd-undercloud************** overcloud complete status=2 ************02:52
*** cd-undercloud has quit IRC02:52
*** boris-42 has quit IRC02:57
*** edmund has joined #tripleo03:07
*** ccrouch has joined #tripleo03:20
*** arata has quit IRC03:22
*** CaptTofu_ has quit IRC03:26
*** CaptTofu has joined #tripleo03:26
lifelessjamezpolley: oh, and devtest is special - it gets turned into docs.03:30
lifelessjamezpolley: dig around in-tree a little to see that03:30
*** cd-undercloud has joined #tripleo03:34
cd-undercloud************** overcloud complete status=1 ************03:34
*** cd-undercloud has quit IRC03:34
*** edmund has quit IRC03:46
*** edmund has joined #tripleo04:01
*** julim has joined #tripleo04:13
*** toci-bot has joined #tripleo04:14
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_zqdhVcF/04:14
*** toci-bot has quit IRC04:14
*** arata has joined #tripleo04:18
*** julim has quit IRC04:23
*** cody-somerville has quit IRC04:27
SpamapSlifeless: ever dig through these a bit: http://www.mellanox.com/related-docs/prod_software/Mellanox_EN_for_Linux_Release_Notes_v2_0-3_0_0.pdf ?04:29
*** cd-undercloud has joined #tripleo04:40
cd-undercloud************** overcloud complete status=2 ************04:40
*** cd-undercloud has quit IRC04:40
*** cody-somerville has joined #tripleo04:42
SpamapSOk I think status=2 is always this failure:04:42
SpamapSAuthorization Failed: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP Unable to establish connection t04:42
SpamapSo http://138.35.77.4:5000/v2.0/tokens)04:42
lifelessSpamapS: not so much.04:51
*** rongze has quit IRC04:53
SpamapSlifeless: when you've unloaded/reloaded the mlx driver.. did you leave notes somewhere?04:54
SpamapSlifeless: I"m seeing tons of really odd intermittent network behavior04:54
SpamapSlifeless: crazy thought - bad cable maybe? :-P04:54
*** arata has quit IRC05:00
*** matsuhashi has joined #tripleo05:06
lifelessSpamapS: unload the driver, reload, remove from bridge, add to bridge05:08
lifelessSpamapS: it's all in the console session history05:09
*** akuznetsov has joined #tripleo05:15
*** matsuhashi has quit IRC05:16
SpamapSlifeless: "console session" ?05:20
SpamapSlifeless: remember that I am screen stupid05:20
SpamapSlifeless: so if there is some screen somewhere, I have no clue how to attach to it. :-/05:20
*** matsuhashi has joined #tripleo05:23
*** rongze has joined #tripleo05:24
lifelesson the bastion05:24
lifelessscreen -ls05:24
*** vkozhukalov has joined #tripleo05:24
*** edmund has quit IRC05:25
lifelessthen screen -r 15804.ilos or 14980.ilos (they are different ilo sessions screens)05:25
*** rpodolyaka1 has joined #tripleo05:32
*** rongze has quit IRC05:33
SpamapSlifeless: ty05:37
SpamapSlifeless: so if that clears it up.. I think we should consider migrating our undercloud to a new machine...05:37
SpamapSlifeless: perhaps one with saucy05:38
*** michchap has quit IRC05:40
*** michchap has joined #tripleo05:40
*** cd-undercloud has joined #tripleo05:44
cd-undercloud************** overcloud complete status=1 ************05:44
*** cd-undercloud has quit IRC05:44
*** arata has joined #tripleo06:06
*** arata has quit IRC06:07
lifelessSpamapS: no reason to believe other machines will be better06:08
lifelessSpamapS: and AIUI saucy doesn't have the fix... IMBW06:08
*** vkozhukalov has quit IRC06:18
*** michchap has quit IRC06:19
SpamapSlifeless: is there an explicit fix? Something in the back of my brain is saying yes.06:24
lifelessSpamapS: I linked stuff in trello06:24
* SpamapS knew there was something06:25
*** michchap has joined #tripleo06:26
SpamapSlifeless: its not so much that I think other hardware will be better. It is that if other hardware is not better, that is _data_.06:26
lifelessSpamapS: ack; thing is, we don't really have a facility for moving the undercloud around :(06:28
lifelessSpamapS: if you recall we deployed perfectly for the first 3-4 days06:29
SpamapSYes I do recall that.06:29
lifelessSpamapS: we could try a reboot, but I'm skeptical.06:29
lifelessSpamapS: so, I guess, it is data.06:30
lifelessSpamapS: I'm worried about achieving it06:30
SpamapSlifeless: http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&mrequest=downloads&mtype=ofed&mver=MLNX_OFED-2.0-3.0.0&mname=MLNX_OFED_LINUX-2.0-3.0.0-ubuntu12.04-x86_64.iso right?06:33
lifelessSpamapS: I guess06:33
lifelessSpamapS: I just have the pointer to JIRA same as you06:33
SpamapSah I think I was booted from JIRA after the HPCC move06:34
lifelessyou should have been reinstated06:36
lifelessif not, there is a clicky clicky thing to get back in06:36
*** rongze has joined #tripleo06:37
*** matsuhashi has quit IRC06:44
*** matsuhas_ has joined #tripleo06:49
SpamapSlifeless: so looking at the driver source in that iso..06:49
SpamapSlifeless: there are some fixes that look like "zero out previously used memory"06:49
*** rpodolyaka1 has quit IRC06:51
SpamapSand one thing where the old one waits for a timeout then moves on, where the new one waits for a timeout, if it is reached, warns and retries, and then waits unconditionally for the same thing the other one would skip if the timeout was reached06:51
SpamapSseems like it might be worth trying the new driver06:52
* SpamapS checks to see if saucy already has this06:53
* SpamapS should probably check trusty too06:53
*** matsuhas_ has quit IRC06:56
lifeless6 techcon papers to go06:57
*** cd-undercloud has joined #tripleo06:58
cd-undercloud************** overcloud complete status=2 ************06:58
*** cd-undercloud has quit IRC06:58
lifelessit's a little worrying that every LXC in Ubuntu was broken recently (AFAICT)06:59
*** jprovazn has joined #tripleo07:01
*** matsuhashi has joined #tripleo07:03
*** sdake_ has quit IRC07:08
*** akuznetsov has quit IRC07:09
jprovaznlifeless, you still around?07:18
lifelessjprovazn: somewhat07:19
jprovaznphew07:19
jprovaznquestion about this:07:19
jprovaznhttps://etherpad.openstack.org/p/tripleo-icehouse-scaling-design07:19
jprovaznlifeless, it seems that next step is to add logstash and/or collectd to monitor undercloud07:20
jprovaznlifeless, anybody is on it?07:20
lifelessjprovazn: not at the moment07:22
lifelessjprovazn: we're working from the MVP in trello07:22
lifelessjprovazn: something that needs doing right now is migrating elements to store state in /mnt/state07:22
*** cwolferh has quit IRC07:23
jprovaznlifeless, by state you mean any state and any element? e.g. for yum/apt save package database in /mnt/state?07:23
lifelessany files we want to be modifying in a running image-update image07:24
lifelessso not yum/apt07:24
lifelessbut nova.conf07:24
lifelesssshd host keys07:24
lifelessmysql database files07:24
lifelessnova instance image files07:24
jprovaznah, thanks07:30
jprovaznlifeless, when working on an element, should a task in trello be created?07:31
*** rushiagr has joined #tripleo07:31
lifelessjprovazn: only if you start to run into coordination issues; tere is already a card for this overall thing I think07:31
lifelessjprovazn: yeah, third down in the current MVP column07:32
lifelessjprovazn: just add your name to the names on the card, and folk will know you're also working on that broad theme.07:32
lifelessjprovazn: there are lots of elements to update, likely there won't be any conflicts :)07:32
jprovaznlifeless, ok07:32
jprovaznlifeless, and completely different question - how about HA? https://etherpad.openstack.org/p/tripleo-icehouse-ha-production-configuration07:33
lifelessjprovazn: noone's working on it yet - its just a little over the horizon07:33
jprovaznlifeless, this is in status of finding out what should be done or is there already cleaner idea?07:33
lifelessthe HA spec outlines what we'll do when we get to that point - and we'll need that to get CI/CD fully live so it's also quite important.07:34
openstackgerritA change was merged to openstack/tripleo-incubator: Add the nova V3 api endpoint  https://review.openstack.org/5753307:34
lifelessbasically I'm delighted to see any work that takes us towards the full CI/CD story :)07:34
lifelessjprovazn: I would most like to see folk hacking on the /mnt/state stuff for now.07:35
lifelessas rpodolyaka and jog0 are hacking on the nova rebuild support07:35
jprovaznlifeless, great, I will be happy to help with /mnt/state right now, but I'd like also put some attention into HA at the momments I'm blocked/wait for something07:36
lifelessjprovazn: fair enough!07:37
GheRiveromorning TripleO07:38
rpodolyakamorning all07:38
jprovaznmorning07:40
*** akuznetsov has joined #tripleo07:41
*** jtomasek has joined #tripleo07:51
*** akuznetsov has quit IRC07:51
*** vkozhukalov has joined #tripleo07:52
*** ifarkas has joined #tripleo07:52
*** ifarkas has quit IRC07:53
*** ifarkas has joined #tripleo07:54
openstackgerritA change was merged to openstack/tripleo-image-elements: Fix PATH in pypi-mirror cron job  https://review.openstack.org/5772407:56
*** arata has joined #tripleo08:14
*** cd-undercloud has joined #tripleo08:14
cd-undercloud************** overcloud complete status=1 ************08:14
*** cd-undercloud has quit IRC08:14
SpamapShm ok saucy does not have the latest mellanox either08:20
SpamapSok, 3.12.0 doesn't either08:27
lifelessthat would be too easy, right?08:28
SpamapSyeah08:28
SpamapSI guess it is time to learn how to use dkms08:29
SpamapSthat or ask them to plug in 04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)08:29
lifelessnoo08:30
lifelessdkms :)08:30
*** nosnos_ has joined #tripleo08:31
*** nosnos has quit IRC08:35
*** cody-somerville has quit IRC08:44
*** akuznetsov has joined #tripleo08:44
*** cody-somerville has joined #tripleo08:45
SpamapSlifeless: FYI, screen says those ilos sessions are not shared08:45
*** athomas has joined #tripleo08:46
lifelessSpamapS:         15804.ilos      (10/07/2013 03:00:54 PM)        (Multi, attached)08:46
*** jtomasek has quit IRC08:46
lifelessSpamapS: 'Multi'08:46
SpamapS$ screen -r robertc/15804.ilos08:46
SpamapSMust run suid root for multiuser support.08:46
SpamapS$ sudo screen -r robertc/15804.ilos08:47
SpamapSAccess to session denied.08:47
lifelesshu08:47
lifelesstry08:47
lifelessscreen -r 15804.ilos ?08:47
SpamapS$ screen -r 15804.ilos08:47
SpamapSThere is no screen to be resumed matching 15804.ilos.08:47
lifelessnuts08:48
lifelesstry screen -Ax robertc/15804.ilos08:48
SpamapSdenied08:49
lifelessboth with and without sudo ?08:49
SpamapSisn't there a ctrl-A you have to do to give me access?08:49
SpamapSw/o sudo it whines about suid08:49
lifelessI dunno, Ng set this up originally08:50
lifelesshuh08:50
lifelessyou're not in the users gorup08:50
lifeless*group*08:50
*** jtomasek has joined #tripleo08:50
lifelessadd yourself to that, log out and in, try agin08:50
SpamapSdenied08:52
SpamapSalso tried making screen suid root08:52
SpamapSnow denied as my regular user08:52
lifelessuhm, no there was no need to chmod the screen binary08:52
lifelessyou'll want to put that back to unbreak it08:53
SpamapSwas just testing08:53
lifelesssigh08:54
lifelessrobertc@bm-aw1az2-freecloud0001:/home/shared$ screen -Ax ilos08:54
lifelessDirectory '/var/run/screen' must have mode 755.08:54
lifelessI wish you hadn't done that test08:54
SpamapSlifeless: hence... conman. :-/08:54
lifelesson a box where it's known to work for everyone else08:54
SpamapSlifeless: really? that borkens things?08:54
lifelessSpamapS: it may have been suid already08:54
SpamapSbrittle little thing :-P08:54
SpamapSit was not08:54
SpamapSI checked08:54
SpamapSit was sgid08:54
SpamapSand is now08:55
lifelessok, I raced with you08:55
lifelessits ok now08:55
SpamapSand /var/run/screen is the same as other working boxes08:55
lifelessok, try screen -Ax ilos now08:56
lifelessno sudo08:56
SpamapS-ls shows none08:59
SpamapS-Ax says no sockets found in /var/run/screen/S-clint08:59
*** jistr has joined #tripleo09:00
lifelesstry with ilos2?09:01
SpamapSnothing. :(09:02
lifelessrobertc/ilos2?09:02
SpamapS"Must run suid root for multiuser support"09:04
lifelessbear with me a sec09:04
*** nosnos_ has quit IRC09:04
SpamapSjust building modules09:05
*** nosnos has joined #tripleo09:05
lifelessSpamapS: ok, try screen -Ax robertc/ilos109:08
SpamapS"Must run suid root for multiuser support.09:08
lifelessok, try sudo that09:08
SpamapSAccess to session denied.09:09
lifelessok, acess denied09:09
lifelesstry now09:09
SpamapS\o/09:09
SpamapSso glad I could use your time to get that done09:10
lifelesstry ilos2, no sudo ?09:11
SpamapSstill same09:11
lifelessok, w/sudo should work09:12
SpamapSso, the ofed thing actually came with a dkms package already09:12
SpamapSlifeless: thoughts on installing the newer driver?09:12
lifelessSpamapS: test it in a baremetal image first :)09:12
SpamapSah now that's thinking :)09:13
SpamapSok, will attack that tomorrow09:13
SpamapSofed source is in ~heat-admin/mlnx-ofed-kernel-2.0 if you get bored09:13
SpamapSdebuild binary produces the dkms package09:13
* SpamapS heads to bed09:13
lifelessSpamapS: check iols2 fist?09:14
lifelessilos2 first?09:14
*** derekh has joined #tripleo09:15
*** cd-undercloud has joined #tripleo09:15
cd-undercloud************** overcloud complete status=1 ************09:15
*** cd-undercloud has quit IRC09:15
*** marun has joined #tripleo09:16
Ngmornint09:18
derekhovercloud-notcompute-5myoxsyixxnk stuck in BUILD state ...09:18
derekhon cd-overcloud09:19
Nglifeless: SpamapS: for talking to the ilos I just apply a giant hammer of awfulness and chmod my pts so I can su to the user who owns the screen session and attach09:20
Nglifeless: SpamapS: but if you just want to talk to serial console for shell, remember conman is there and works :)09:20
*** akuznetsov has quit IRC09:20
lifelessNg: ah, I thought you had done something nice09:20
lifelessNg: I've added root to the acls09:20
Ngcool09:20
lifelessso if anyone does screen -S ilo1 -c ilo_sreenrc109:21
Ngideally we'd have a role user to run the screen sessions as, I think09:21
lifelessand again with 1/209:21
lifelessthen sudo screen -Ax user/ilos109:21
lifelesswill work09:21
*** marun has quit IRC09:22
*** marun has joined #tripleo09:23
*** lucasagomes has joined #tripleo09:23
lifelessderekh: we suspect flaky drivers09:27
lifelessderekh: SpamapS has about 1/2 through getting a new mellanox driver together09:27
derekhlifeless: ok09:27
*** boris-42 has joined #tripleo09:29
*** matsuhashi has quit IRC09:29
lifelessderekh: I'm just resetting the firmware now09:30
*** matsuhashi has joined #tripleo09:30
lifelessbut I'm off to bed after that09:30
derekhlifeless: ok, if there is anything I can help out with let me know (although I know pretty much nothing about mellanox drivers)09:31
lifelesswill do09:31
lifelessrmmod has hung... sure sign that this was the issue09:32
lifeless[3893443.151304] mlx4_core 0000:05:00.0: mlx4_cmd_post:cmd_pending failed09:32
lifeless[3893451.633588] INFO: task ovs-vswitchd:9303 blocked for more than 120 seconds.6e:0609:32
*** akuznetsov has joined #tripleo09:32
lifeless[3893451.633659] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"09:33
lifelessSpamapS: ^09:33
Ngouch09:33
*** boris-42 has quit IRC09:34
lifelessstill going09:34
*** jtomasek has quit IRC09:40
*** martyntaylor has joined #tripleo09:41
*** jtomasek has joined #tripleo09:43
lifelessstill doing it's thing on the ilo textcnos09:46
lifelessI'm going to leave it churning away09:46
lifelessit may come back, it may not - if not check br-ctlplane has the right ports et09:46
*** arata has quit IRC09:48
derekhtempest getting close10:06
derekhRan 1236 tests in 3675.426s10:06
derekhFAILED (id=0, failures=5, skips=64)10:06
derekhalthough it took an hour10:06
*** pblaho has joined #tripleo10:11
openstackgerritMarios Andreou proposed a change to openstack/tripleo-heat-templates: Make merge.py invokable from another script  https://review.openstack.org/5694710:17
*** marun has quit IRC10:45
*** cody-somerville has quit IRC10:46
*** marun has joined #tripleo10:47
*** cody-somerville has joined #tripleo10:48
*** pblaho has quit IRC10:52
*** marun has quit IRC10:56
*** marun has joined #tripleo10:56
*** marun has quit IRC10:58
*** cody-somerville has quit IRC11:02
*** athomas has quit IRC11:07
*** cody-somerville has joined #tripleo11:07
*** rdo has quit IRC11:07
*** athomas has joined #tripleo11:15
*** max_lobur_afk has quit IRC11:30
*** max_lobur has joined #tripleo11:33
openstackgerritImre Farkas proposed a change to openstack/tripleo-incubator: Add support to register horizon as an endpoint  https://review.openstack.org/5790611:54
*** cody-somerville has quit IRC11:58
*** matsuhashi has quit IRC12:09
*** matsuhashi has joined #tripleo12:10
*** matsuhashi has quit IRC12:15
*** ifarkas has quit IRC12:18
*** ifarkas has joined #tripleo12:19
*** akuznetsov has quit IRC12:21
*** michchap has quit IRC12:23
*** michchap has joined #tripleo12:23
*** CaptTofu has quit IRC12:28
*** CaptTofu has joined #tripleo12:28
*** lucasagomes is now known as lucas-hungry12:29
openstackgerritDan Prince proposed a change to openstack-infra/tripleo-ci: Set OVERCLOUD_IP correctly for all-in-one.  https://review.openstack.org/5791212:34
*** michchap has quit IRC12:45
*** michchap has joined #tripleo12:46
ifarkasmarios, ping12:56
*** rdopieralski has joined #tripleo12:58
*** bauzas has quit IRC12:58
mariosifarkas: hey13:00
*** marun has joined #tripleo13:00
*** akuznetsov has joined #tripleo13:01
ifarkasmarios, hey, regarding the horizon endpoint patch, I am working on it to add the overcloud horizon url to tuskar-ui. The patch is more general as it just adds support to register the horizon service, so it's definitely not only an overcloud endpoint blueprint specific change. What else do you think I should add to the commit message?13:01
*** CaptTofu has quit IRC13:02
*** akuznetsov has quit IRC13:02
*** CaptTofu has joined #tripleo13:03
mariosifarkas: heh, i don't know you could write any number of things. just thought it looked odd to not have at least one sentence there explaining the commit for sake of posterity13:04
marios"right now horizon isn't registered as an endpoint, so services like <foo> (tuskar/ui) can't bla"13:06
mariosexample13:06
ifarkasmarios, ok13:07
openstackgerritImre Farkas proposed a change to openstack/tripleo-incubator: Add support to register horizon as an endpoint  https://review.openstack.org/5790613:11
*** lsmola has quit IRC13:11
mariosifarkas: thanks13:15
ifarkasmarios, than you for your hint ;-)13:15
ifarkas*thank13:15
*** bauzas1 has joined #tripleo13:18
openstackgerritImre Farkas proposed a change to openstack/tripleo-incubator: Add support to register horizon as an endpoint  https://review.openstack.org/5790613:19
mariosifarkas: hehe, thanks :)13:19
ifarkasmarios, thanks for the review. I think it's better to fix this small issue ;-)13:20
*** pblaho has joined #tripleo13:33
*** jdob has joined #tripleo13:33
*** lucas-hungry is now known as lucasagomes13:34
*** nosnos has quit IRC13:38
*** akuznetsov has joined #tripleo13:39
*** julim has joined #tripleo13:47
*** CaptTofu has quit IRC13:53
*** akuznetsov has quit IRC13:53
*** CaptTofu has joined #tripleo13:53
*** dprince has joined #tripleo13:55
*** marun has quit IRC13:57
*** marun has joined #tripleo13:58
*** michchap has quit IRC13:58
*** michchap has joined #tripleo14:01
*** tzumainn has joined #tripleo14:02
*** rdopieralski has quit IRC14:04
*** morazi has joined #tripleo14:07
*** akuznetsov has joined #tripleo14:07
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: Adding a geard element  https://review.openstack.org/5792714:09
mariosderekh: s/borker/broker14:11
*** bauzas1 is now known as bauzas14:11
derekhmarios: thanks fixing14:11
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: Adding a geard element  https://review.openstack.org/5792714:12
*** boris-42 has joined #tripleo14:13
*** julim has quit IRC14:13
*** michchap has quit IRC14:14
*** michchap has joined #tripleo14:14
*** julim has joined #tripleo14:20
*** jayg|g0n3 is now known as jayg14:21
*** lsmola has joined #tripleo14:23
rpodolyakaI'm a bit worried about "The server is not designed for production use under load" line :)14:24
*** jdob has quit IRC14:31
*** jdob has joined #tripleo14:31
*** boris-42 has quit IRC14:32
*** boris-42_ has joined #tripleo14:32
*** cody-somerville has joined #tripleo14:34
*** rushiagr has quit IRC14:41
*** shakayumi has joined #tripleo14:44
derekhrpodolyaka: ya I hear ya, but from what I can gather its what infra use for all the gating stuff14:44
derekhrpodolyaka: https://github.com/openstack-infra/gear14:45
rpodolyakayeah, just wanted to say I'm totally ok with that as it doesn't seem we have too much choice here14:47
dkehnmorning all14:49
*** martyntaylor has quit IRC14:52
*** jdob has quit IRC14:52
*** jdob has joined #tripleo14:52
*** jtomasek has quit IRC14:53
*** bnemec is now known as beekneemech14:53
*** boris-42_ has quit IRC14:58
NobodyCamGood Morning TripleO15:03
*** lucasagomes has quit IRC15:06
*** pblaho has quit IRC15:06
*** marun has quit IRC15:08
*** marun has joined #tripleo15:09
*** jtomasek has joined #tripleo15:19
*** lucasagomes has joined #tripleo15:21
*** shakayumi has quit IRC15:22
*** edmund has joined #tripleo15:23
*** shakayumi has joined #tripleo15:23
*** marun has quit IRC15:29
*** pblaho has joined #tripleo15:32
*** rongze has quit IRC15:45
*** ifarkas has quit IRC15:47
*** shakayumi has quit IRC15:48
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: Adding a geard element  https://review.openstack.org/5792715:49
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Update nova-kvm to destroy the default libvirt net  https://review.openstack.org/5776315:50
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Update nova-kvm to destroy the default libvirt net  https://review.openstack.org/5776315:52
*** rongze has joined #tripleo16:04
*** UtahDave has joined #tripleo16:06
*** akuznetsov has quit IRC16:08
*** vkozhukalov has quit IRC16:17
*** rongze_ has joined #tripleo16:32
*** rongze has quit IRC16:32
*** max_lobur has quit IRC16:33
*** jtomasek has quit IRC16:36
*** akuznetsov has joined #tripleo16:38
*** csd has quit IRC16:44
*** dkehn has quit IRC16:59
*** derekh has quit IRC17:00
*** dkehn has joined #tripleo17:00
*** julim has quit IRC17:05
*** athomas has quit IRC17:06
*** julim has joined #tripleo17:06
*** rpodolyaka has quit IRC17:12
*** athomas has joined #tripleo17:12
openstackgerritPetr Blaho proposed a change to openstack/python-tuskarclient: [WIP] Adds help for subcommands  https://review.openstack.org/5625717:16
*** arata has joined #tripleo17:17
*** arata has left #tripleo17:17
*** pblaho has quit IRC17:20
*** jistr has quit IRC17:27
*** athomas has quit IRC17:36
*** rushiagr has joined #tripleo17:36
*** hewbrocca has quit IRC17:37
*** lsmola has quit IRC17:41
openstackgerritDan Prince proposed a change to openstack/tripleo-heat-templates: Use merge.py for the undercloud templates.  https://review.openstack.org/5799417:50
*** rongze has joined #tripleo17:57
* Ng out for dinner17:58
*** rongze_ has quit IRC18:01
*** akuznetsov has quit IRC18:02
*** UtahDave has quit IRC18:09
*** akuznetsov has joined #tripleo18:11
openstackgerritDan Prince proposed a change to openstack/tripleo-incubator: Make undercloud-vm.yaml.  https://review.openstack.org/5799718:23
*** rushiagr has quit IRC18:27
openstackgerritDan Prince proposed a change to openstack/tripleo-incubator: Make undercloud-vm.yaml.  https://review.openstack.org/5799918:30
openstackgerritDan Prince proposed a change to openstack-infra/tripleo-ci: Make undercloud-vm.yaml.  https://review.openstack.org/5800018:31
*** rushiagr has joined #tripleo18:33
*** rongze has quit IRC18:42
*** lucasagomes has quit IRC18:43
SpamapSlifeless: when you're around.. it isn't clear to me what happened with your rmmod.18:43
dprinceSpamapS: does that mean the CD cloud is down?18:49
SpamapSdprince: it is turned off right now18:51
SpamapSdprince: we were going to try a new mellanox driver actually18:51
dprinceSpamapS: I'm a bit out of the loop on the mellanox situation. Last I used mellanox it was for IB... Is there a tl;dl?18:52
SpamapSdprince: our hardware has Mellanox 10Gb ethernet cards that we use for connection to the network. We've had serious driver flakiness.18:56
SpamapSdprince: rmmod/insmod clears them for a while.18:56
dprinceSpamapS: we do this on the undercloud?18:56
SpamapSdprince: yeah, that one seems to be the one with the problem.19:00
SpamapSdprince: only because it has been running for so long we suspect19:01
*** sdake_ has joined #tripleo19:10
*** sdake_ has quit IRC19:10
*** sdake_ has joined #tripleo19:10
lifelessSpamapS: hi19:18
lifelessSpamapS: when I did rmmod mlx4_en mlx4_core; modprobe mlx19:18
lifeless4_en; sleep 1; ip address del 10.10.16.169/26 dev eth2; ovs-vsctl del-port br-ct19:18
lifelesslplane eth2; ovs-vsctl add-port br-ctlplane eth219:18
lifelessSpamapS: on a console session19:18
lifelessSpamapS: it spat out that cmd error for 20m before I said enough and went to bed19:18
*** vkozhukalov has joined #tripleo19:25
*** jdob has quit IRC19:28
*** jdob has joined #tripleo19:32
SpamapSlifeless: ok, so you killed it, it never actually rmmodded?19:32
*** rpodolyaka has joined #tripleo19:38
*** rongze has joined #tripleo19:43
dprincelifeless: got time for a question?19:44
*** rongze has quit IRC19:48
*** dprince has quit IRC19:52
*** rpodolyaka has quit IRC19:55
*** toci-bot has joined #tripleo20:04
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_qInnh5J/20:04
*** toci-bot has quit IRC20:04
*** akuznetsov has quit IRC20:06
*** akuznetsov has joined #tripleo20:11
*** jprovazn has quit IRC20:12
*** cwolferh has joined #tripleo20:13
*** akuznetsov has quit IRC20:16
*** rushiagr has quit IRC20:18
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface  https://review.openstack.org/5777620:25
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface  https://review.openstack.org/5777620:58
*** vipul is now known as vipul-away21:29
*** vipul-away is now known as vipul21:29
*** toci-bot has joined #tripleo21:31
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_4YFSr4g/21:31
*** toci-bot has quit IRC21:31
openstackgerritClint Byrum proposed a change to openstack/tripleo-image-elements: In tripleo-cd log hardware to instance map  https://review.openstack.org/5774921:32
*** jistr has joined #tripleo21:33
*** vipul is now known as vipul-away21:37
*** vipul-away is now known as vipul21:37
openstackgerritClint Byrum proposed a change to openstack/tripleo-image-elements: Log pypi-mirror cron job output to syslog  https://review.openstack.org/5772521:41
NgSpamapS: very +1 capturing cron output with logger :)21:47
SpamapSNg: it's one of those "why doesn't cron just do this?" things.21:47
NgSpamapS: I can kinda see an argument that such a default would mean dumping giant pages of text into syslog, but, I agree21:48
SpamapSdefault no21:50
SpamapSbut just.. do it21:50
SpamapSlike,  OUTPUT=syslog or something21:50
Ngthere is still my vomitous option of procmailling into logger ;)21:51
*** vkozhukalov has quit IRC21:51
openstackgerritA change was merged to openstack-infra/tripleo-ci: Set OVERCLOUD_IP correctly for all-in-one.  https://review.openstack.org/5791221:57
*** julim has quit IRC21:59
*** jistr has quit IRC22:10
SpamapSNg: wtf are you doing up?22:10
SpamapSOh wait it's only 2200 there22:11
SpamapSI'm seriously going to buy a globe that rotates a sun-like light bulb and paste all of your pictures on the places you live to see what time it is22:11
* SpamapS may have just invented the next booth schwag for the openstack summit :22:12
clarkbSpamapS: google earth/maps can do realtime sunlight and weather if you zoom out far enough22:12
clarkbbut physical device is much more interesting22:12
SpamapSright22:12
SpamapSI could put earth on the one screen I don't really use..22:13
SpamapSIf it can actually show me peoples' G+ pictures in their current location (or at least profile location) that would be money.22:13
NgSpamapS: you don't need the globe thing, just add timezones for all the people you care about the wakefulness of, to the datetime dropdown at the top right of your screen :)22:15
NgI typically have Pacific, Easter, UTC, Australia and New Zealand in there22:16
*** jdob has quit IRC22:18
Nghmm, do not like that gerrit lists a review as having a +2, but when I go into it, I see that it has a more recent -1 from a core person22:19
*** morazi has quit IRC22:23
clarkbit is biggest value wins where -2 is biggest value :)22:23
openstackgerritA change was merged to openstack/tripleo-image-elements: Update nova-kvm to destroy the default libvirt net  https://review.openstack.org/5776322:25
Ngclarkb: yeah I'm not sure how it should really display, maybe some icon indicating dissent in the ranks ;)22:26
Ngis there an emoji of a mob fighting?22:26
openstackgerritDan Prince proposed a change to openstack-infra/tripleo-ci: Make undercloud-vm.yaml.  https://review.openstack.org/5800022:33
openstackgerritDan Prince proposed a change to openstack/tripleo-incubator: Make undercloud-vm.yaml.  https://review.openstack.org/5799922:34
SpamapSNg: I have it. I don't want to click anywhere :)22:37
Ngmaybe now that we have bendable LCD someone could make you a globe that displays xearth :)22:37
SpamapSNg: I disagree actually. You still should be reviewing things with -1's... at least you should be reviewing said -1 to see if you agree. -1 is not "nobody else review this". It is "I don't like it."22:38
NgSpamapS: yeah it's not so much because I want to just skip it, more that if I can see there is dissent I can go and check it out22:39
lifelessSpamapS: it did eventually I think22:40
Ngbut realistically I'm going to go in anyway, so it's no biggie22:40
SpamapSlifeless: ah ok22:40
lifelessSpamapS: did it come good ?22:40
SpamapSNg: right, I actually think the "last touched" queue is more important. As in, look at things that have been touched least recently.22:40
SpamapSlifeless: tripleo-cd was still off, so don't know for sure22:44
SpamapSlifeless: going to watch it run now22:44
SpamapSI do wonder if we have a race with init-keystone and o-r-c somehow in the result=2 cases.22:44
jamezpolley@SpamapS I was more confused by you being online last night when it was ~2000 my time22:45
lifelessSpamapS: remember init-keystone is synced by a wait condition22:47
lifelessSpamapS: so, no.22:47
SpamapSjamezpolley: indeed, I have to _EARN_ my sleep time. :)22:48
SpamapSlifeless: unless after the wait condition is fired, we run orc again and are mid-restarting22:48
SpamapShypothesis .. not findings22:48
*** UtahDave has joined #tripleo22:52
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface  https://review.openstack.org/5777622:52
*** sdake_ has quit IRC23:04
*** toci-bot has joined #tripleo23:07
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_3mrvTb5/23:07
*** toci-bot has quit IRC23:07
*** cd-undercloud has joined #tripleo23:28
cd-undercloud************** overcloud complete status=1 ************23:28
*** cd-undercloud has quit IRC23:28
SpamapSUnable to find network with name 'ext-net'23:39
SpamapSand yet, ext-net exists now23:40
SpamapSlifeless: is there any reason we actually build the user image?23:40
SpamapSlifeless: couldn't we just upload base ubuntu?23:41
*** noslzzp has quit IRC23:41
*** edmund has quit IRC23:42
lifelessSpamapS: we could today; We'll want a heat enabled image to check heat in the cloud though23:43
SpamapSok so this might be that asynchronous network schedling thing23:44
SpamapSscheduling23:44
SpamapSthe command that failed was neutron floatingip-create ext-net --port-id "${PORT//[[:space:]]/}"23:45
SpamapSwhich works fine23:45
SpamapSI've heard this before23:45
SpamapS"Neutron is liar" was the title of a lightning talk slide. :)23:45
SpamapS| fault                                | {u'message': u'Unexpected vif_type=binding_failed', u'code': 500, u'created': u'2013-11-22T19:05:22Z'} |23:48
SpamapSactually that was the problem23:48
SpamapSbut.. no idea what it actually means23:48
SpamapSlifeless: seems that our network driver problems were some of the issues, but we do have something really breaking the overcloud23:49
lifelesshttp://www.gossamer-threads.com/lists/openstack/dev/3318423:49
lifelessSpamapS: thus my point about treating this like production even though we don't have state - keeps bitrot low :)23:50
*** edmund has joined #tripleo23:50
lifelessSpamapS: neutron agent-list?23:50
lifelesshttp://www.gossamer-threads.com/lists/openstack/dev/33218#3321823:51
SpamapSlifeless: doh, tripleo-cd wiped it out again :(23:51
SpamapSshould have stopped it23:51
lifelessnvm23:51
SpamapSI added the bridge_mapping in the ml2_plugin.ini and now everything works23:52
lifelessSpamapS: so - look at neutron agent logs next time I think23:52
* SpamapS checks23:52
SpamapSshould have been quotes on that last thing23:52
SpamapSwe don't even have an ml2_plugin.ini23:53
lifelessSpamapS: where are you looking?23:53
SpamapSok apparently it is usually configured in ovs_neutron_plugin.ini which we do23:54
SpamapSso maybe neutron.ovs.bridge_mappings wasn't set right23:54
lifelesselements/neutron/os-config-applier/etc/neutron/plugins/ml2/ml2_conf.ini23:54
*** UtahDave has quit IRC23:55
SpamapS_conf isn't _plugin ..?23:55
* SpamapS should probably just RTFM on bridge_mapping and see if we are doing it right23:55
SpamapS          bridge_mappings: ''23:56
*** jayg is now known as jayg|g0n323:58
SpamapSlifeless: http://docs.openstack.org/network-admin/admin/content/app_demo_single_router.html <-- we are that right?23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!