Thursday, 2013-11-21

*** epim has joined #tripleo00:00
*** epim has quit IRC00:01
*** CaptTofu has quit IRC00:03
*** CaptTofu has joined #tripleo00:03
derekhAre we to refrain from approving for the moment ?00:06
*** derekh has quit IRC00:09
*** CaptTofu has quit IRC00:14
*** CaptTofu has joined #tripleo00:15
*** epim has joined #tripleo00:16
*** boris-42 has joined #tripleo00:32
*** matsuhashi has joined #tripleo00:32
*** julim has joined #tripleo00:37
*** arata has joined #tripleo00:42
*** CaptTofu has quit IRC00:43
*** sdake_ has quit IRC00:44
*** CaptTofu has joined #tripleo00:44
*** julim has quit IRC00:46
*** cody-somerville has quit IRC00:48
*** epim has quit IRC00:54
*** CaptTofu has quit IRC00:55
*** CaptTofu has joined #tripleo00:55
*** matsuhashi has quit IRC01:02
*** cody-somerville has joined #tripleo01:02
*** matsuhashi has joined #tripleo01:02
*** epim has joined #tripleo01:06
*** matsuhashi has quit IRC01:07
*** morazi has quit IRC01:11
*** matsuhashi has joined #tripleo01:15
*** boris-42 has quit IRC01:20
*** asalkeld has quit IRC01:25
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Refactor reset-db so we don't sync by default.  https://review.openstack.org/5746601:29
*** nosnos has joined #tripleo01:30
*** arata has left #tripleo01:35
*** matsuhashi has quit IRC01:45
*** matsuhashi has joined #tripleo01:46
lifelesshttps://lists.ubuntu.com/archives/foundations-bugs/2013-June/152224.html is apparently key for saucy ramdisk support01:50
lifelessSpamapS: so, did you get time to poke at tripleo-cd ?02:09
*** michchap_ has joined #tripleo02:11
*** michchap has quit IRC02:14
*** epim has quit IRC02:19
*** vkozhukalov has joined #tripleo02:19
*** michchap has joined #tripleo02:21
*** michchap_ has quit IRC02:24
*** boris-42 has joined #tripleo02:24
*** sdake_ has joined #tripleo02:30
SpamapSlifeless: no I got distracted by reviews... realized I hadn't done any in over a week.02:38
SpamapSlifeless: will look tonight and tomorrow morning though.02:38
*** sdake_ has quit IRC02:56
openstackgerritTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Use baremetal node id for node detail page  https://review.openstack.org/5754903:01
*** cwolferh has quit IRC03:02
*** sdake_ has joined #tripleo03:04
*** matsuhas_ has joined #tripleo03:05
*** matsuhashi has quit IRC03:08
*** sdake_ has quit IRC03:21
*** CaptTofu has quit IRC03:54
*** CaptTofu has joined #tripleo03:55
*** michchap has quit IRC03:55
*** boris-42 has quit IRC03:57
*** boris-42 has joined #tripleo03:57
*** matsuhas_ has quit IRC04:00
*** matsuhashi has joined #tripleo04:02
*** matsuhashi has quit IRC04:14
*** matsuhashi has joined #tripleo04:19
*** coolsvap has joined #tripleo04:21
*** coolsvap has quit IRC04:27
*** coolsvap_ has joined #tripleo04:27
*** coolsvap_ has quit IRC04:28
*** sdake_ has joined #tripleo04:38
*** rushiagr has joined #tripleo04:54
*** akuznetsov has joined #tripleo04:58
*** michchap has joined #tripleo05:01
*** matsuhashi has quit IRC05:02
*** rushiagr has quit IRC05:49
*** tzumainn has quit IRC06:00
SpamapSpulling/updating diskimage-builder06:02
SpamapSerror: Unable to find b9348447b9ed82b738856b7c105857dde1026fca under https://git.openstack.org/cgit/openstack/diskimage-builder06:02
SpamapSCannot obtain needed commit b9348447b9ed82b738856b7c105857dde1026fca06:02
SpamapSwhile processing commit dbacf3e8dfe55beb31cae7ed1bb135e2fe17a1a8.06:02
clarkbSpamapS: does it work fetching from review.openstack.org?06:04
SpamapSroot     31005 31004 99 Nov18 ?        R    4087:07 git-remote-https origin https://git.openstack.org/cgit/openstack/dis06:04
SpamapSwtf?06:04
SpamapSthat has a CPU pegged06:04
SpamapSsince.. it ran06:04
SpamapScurl_multi_perform(0xa51160, 0x7fff4ec0019c, 0x7fff4ec0019c, 0x7fff4ec00138, 0xd2c87c) = 006:05
SpamapSstreaming that06:05
SpamapSprobably found a curl bug06:05
SpamapSbig surprise06:05
SpamapSclarkb: trying that now06:08
*** akuznetsov has quit IRC06:08
*** toci-bot has joined #tripleo06:09
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_L7nw7pw/06:09
*** toci-bot has quit IRC06:09
SpamapSclarkb: yes, pulling from review orks06:09
SpamapSworks even06:09
SpamapSclarkb: broken sync?06:09
clarkbcould be06:09
*** vkozhukalov has quit IRC06:11
*** akuznetsov has joined #tripleo06:11
SpamapSclarkb: honestly, I've had nothing but problems since I started using git.o.o06:11
SpamapSclarkb: I find it hard to recommend anymore.06:12
clarkbreally?06:12
SpamapSclarkb: missing commits all the time06:12
clarkbI ask seriously since the gate runs thousands of tests per day and they seem to be ok06:12
clarkbthough something may be masking problem (like cloning the zuul ref from zuul may be pulling commits we are missing)06:12
SpamapSthe gate has a known state and tracks it constantly06:13
SpamapSfor instance this does git pull --ff-only every hour or so06:13
clarkbah06:13
clarkbthe gate does a git remote update then git reset --hard $BRANCH06:13
SpamapSshould work. always.06:13
clarkband keeps full repos instead of shallow clones06:14
SpamapSyeah we just want master06:14
SpamapSwe do not have a shallow clone06:14
clarkboh sorry06:14
SpamapSwe used to, caused problems. ;)06:14
clarkbyou are just fast forwarding, don't know why I read that as shallow clone06:14
clarkbSpamapS: I cannot reproduce a problem cloning dib from the proxy or backends directly06:16
SpamapSclarkb: and I forgot (once again) to copy off the git repo that was having the problem before switching remotes06:16
*** rpodolyaka1 has joined #tripleo06:16
clarkbgit clone http://git04.openstack.org:8080/openstack/diskimage-builder change git04 to 01-04 to hit all backends06:16
clarkbyou can do https too, but that requires forcing git to not check the cert06:17
clarkbat port 444306:17
clarkbthough a clone is slightly different than a fetch06:17
SpamapSright06:17
* clarkb tries a remote update06:17
SpamapSsimpler operation I'm sure06:17
SpamapShm interesting06:18
SpamapSpulling/updating tripleo-image-elements06:18
SpamapSerror: Failed to connect to 2001:4800:7813:516:3bc3:d7f6:ff04:aacb: Network is unreachable (curl_result = 7, http_code = 0, sha1 = c816b83e45858d07dd5a5c2e1a24337b8ac653d0)06:18
SpamapSerror: Unable to find 897a47e32b007ec62a416a2571c6ed1c2c6a2f1b under https://git.openstack.org/cgit/openstack/tripleo-image-elements06:18
lifelesscheck ip -6 route ?06:18
clarkbfetches are fine and git fsck --full si fine for me06:19
SpamapSno default ipv6 route06:19
clarkbI am only ipv406:19
SpamapSand no real addresses06:19
clarkbhow are you updating the repos? does git use curl?06:19
SpamapSclarkb: yes git uses curl for https06:19
SpamapSwhy on earth is it doing an ipv6 address connect though?06:20
clarkbSpamapS: do you have an ipv6 interface?06:20
SpamapSlifeless: should we ask for ipv6 transit on our undercloud? :-P06:20
SpamapSclarkb: no real ones.06:20
clarkbfor some reason your box has decided that AAAA record is valid06:21
SpamapShttp://paste.openstack.org/show/5372206:21
SpamapSwell..06:21
SpamapStechnically it is06:21
SpamapSwe just don't have a route06:21
clarkbright you can't hit it so it should ignore it06:21
clarkbgit version 1.7.9.5 on my end06:22
SpamapScurl may be stupid and doing something low level so it does not interpret "no route" as "try ipv4" ...06:22
SpamapS1.7.10.4 here06:22
SpamapSubuntu 13.0406:22
SpamapSweird06:28
SpamapSsometimes it works fine, we do have to do the ipv6 ENETUNREACH dance first..06:29
SpamapShm06:32
SpamapSwhat happened to the local pypi mirror?06:32
*** rpodolyaka1 has quit IRC06:33
SpamapSNov 21 06:15:01 undercloud-notcompute-jws3awlsb2kh CRON[29756]: (CRON) info (No MTA installed, discarding output)06:33
SpamapSwe'll never know.. :-P06:33
SpamapSlifeless: so it looks like tripleo-cd was hung up on a hung-up git pull06:36
SpamapSyet another feature jenkins has that we don't... timeouts.06:37
SpamapSlifeless: starting to think we should have just used jenkins instead of our janky while loop06:37
*** cd-undercloud has joined #tripleo06:39
cd-undercloud************** overcloud complete status=1 ************06:39
*** cd-undercloud has quit IRC06:39
clarkbSpamapS gerrit git prep has a timeout git retry loop06:40
clarkbsuper ugly but it works06:40
SpamapSwe can shove alarms all over the place sure06:42
SpamapSor even just alarm our while loop06:42
SpamapSpoint being.. we have a thing that does this already.. we shouldn't spend more time enhancing a new thing that does this.06:42
*** asalkeld has joined #tripleo06:43
*** michchap has quit IRC06:43
SpamapS/opt/stack/tripleo-incubator/scripts/devtest_overcloud.sh: line 87: OVERCLOUD_SWIFT_PASSWORD: unbound variable06:43
*** asalkeld is now known as angus_presenting06:43
*** michchap has joined #tripleo06:44
*** rpodolyaka1 has joined #tripleo06:44
*** rushiagr has joined #tripleo06:47
*** nosnos has quit IRC06:48
SpamapSahh06:48
SpamapSwe added a new password06:48
SpamapSand we don't run that with --overwrite06:48
*** michchap has quit IRC06:48
*** nosnos has joined #tripleo06:48
*** angus_presenting has quit IRC06:54
openstackgerritA change was merged to openstack/tripleo-incubator: Use env variables as tripleo-cd cloud params  https://review.openstack.org/5695607:07
*** jcoufal has joined #tripleo07:14
*** akuznetsov has quit IRC07:15
*** rpodolyaka1 has quit IRC07:17
openstackgerritA change was merged to openstack/tripleo-image-elements: Run DB sync in nova-baremetal element post conf.  https://review.openstack.org/5745907:21
openstackgerritA change was merged to openstack/tripleo-image-elements: Run DB sync in glance element post confgure.  https://review.openstack.org/5746007:22
openstackgerritA change was merged to openstack/tripleo-image-elements: Run DB sync in heat-api element post configure.  https://review.openstack.org/5746107:23
*** cd-undercloud has joined #tripleo07:23
cd-undercloud************** overcloud complete status=1 ************07:23
*** cd-undercloud has quit IRC07:23
openstackgerritA change was merged to openstack/tripleo-image-elements: Run DB sync in cinder element post configure.  https://review.openstack.org/5746207:23
openstackgerritA change was merged to openstack/tripleo-image-elements: Run DB sync in ironic-api element post configure.  https://review.openstack.org/5746307:24
openstackgerritA change was merged to openstack/tripleo-image-elements: Run DB sync in tuskar-api element post configure.  https://review.openstack.org/5746407:24
openstackgerritA change was merged to openstack/tripleo-image-elements: Run DB sync in keystone element post configure.  https://review.openstack.org/5746507:25
*** akuznetsov has joined #tripleo07:32
*** akuznetsov has quit IRC07:34
SpamapSok so status=1 from the last run was <attribute 'message' of 'exceptions.BaseException' objects> (HTTP Unable to establish connection to http://138.35.77.407:46
SpamapS:35357/v2.0/users/7f8907cd3649427880f53580fbdc53f8)07:46
*** jprovazn has joined #tripleo07:52
GheRiveromorning TripleO07:53
*** vkozhukalov has joined #tripleo07:54
rpodolyakamorning guys07:56
openstackgerritA change was merged to openstack/tripleo-image-elements: Open up gearman server port.  https://review.openstack.org/5751207:57
openstackgerritA change was merged to openstack/tripleo-incubator: Install the cinder client on devtest host  https://review.openstack.org/5753507:58
SpamapSheh, so unfair that we can land stuff while the rest of OpenStack has to twiddle their thumbs ;)07:58
rpodolyaka:)07:59
ogelbukh:)07:59
openstackgerritA change was merged to openstack/tripleo-image-elements: Add cinder to the list of installed clients  https://review.openstack.org/5752907:59
rpodolyakaI think we help them a lot every day though, because going through devtest story allows to spot many bugs08:00
openstackgerritA change was merged to openstack/tripleo-incubator: Fix floating ips when devtest.sh is hand executed  https://review.openstack.org/5695708:01
openstackgerritA change was merged to openstack/tripleo-incubator: Use separate stack-create overcloud call in docs  https://review.openstack.org/5744508:01
rpodolyakaafter fix of openstack.common.local in nova at least Neutron gates must become more stable08:02
*** akuznetsov has joined #tripleo08:05
*** jtomasek has joined #tripleo08:05
*** cd-undercloud has joined #tripleo08:11
cd-undercloud************** overcloud complete status=1 ************08:11
*** cd-undercloud has quit IRC08:11
*** bauzas has joined #tripleo08:23
*** bauzas has quit IRC08:24
*** bauzas has joined #tripleo08:24
*** nosnos_ has joined #tripleo08:30
lifelessevening evening folk08:33
lifelessSpamapS: I think we're going to need to add host firewalls08:33
*** nosnos has quit IRC08:33
lifelessSpamapS: since swift replication will open internal-only ports on * unless we're really careful08:33
*** akrivoka has joined #tripleo08:40
*** boris-42 has quit IRC08:41
*** jtomasek has quit IRC08:45
*** jtomasek has joined #tripleo08:47
*** jistr has joined #tripleo08:48
*** derekh has joined #tripleo08:58
*** lucasagomes has joined #tripleo08:59
*** Ng has joined #tripleo09:06
*** Ng has quit IRC09:06
*** Ng has joined #tripleo09:07
*** ChanServ sets mode: +v Ng09:07
Ngmorning09:08
lifelessNg: morning!09:08
Ngdamn irssi died last night, apparently09:09
SpamapSlifeless: I'd rather have host firewall and be careful than _think_ we're careful09:16
lifelessSpamapS: exactly.09:17
SpamapSI do so wish cron jobs could just be made to log to syslog instead of MTA09:18
NgSpamapS: procmail for root@ that pipes mails to logger(1) :D09:23
* Ng feels the evil radiating from that suggestion09:24
*** pblaho has joined #tripleo09:26
lifelessNg: eww? :)09:27
*** rdopieralski has joined #tripleo09:29
*** pblaho has quit IRC09:38
*** pblaho has joined #tripleo09:38
SpamapSok time for the sleeping has arrived.09:39
*** cd-undercloud has joined #tripleo09:40
cd-undercloud************** overcloud complete status=1 ************09:40
*** cd-undercloud has quit IRC09:40
SpamapS(that fail, btw, was that the notcompute box never fully booted)09:40
* SpamapS will finish his instance<->baremetal node mapper tomorrow09:40
SpamapSstill think it is feasible that we just have bad hardware gumming up the works. :-P09:41
* SpamapS sleeps09:41
*** athomas has joined #tripleo09:43
*** boris-42 has joined #tripleo09:48
*** jtomasek has quit IRC09:51
*** boris-42 has quit IRC09:52
*** rdopieralski has quit IRC09:53
*** martyntaylor has joined #tripleo09:53
martyntaylorGood Morning all09:55
*** jtomasek has joined #tripleo09:55
jistrmorning :)09:58
*** akuznetsov has quit IRC09:59
*** jcoufal has quit IRC10:09
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Add the nova V3 api endpoint  https://review.openstack.org/5753310:13
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Timeout if seed fails to get IP address  https://review.openstack.org/5753110:20
*** arata has joined #tripleo10:31
*** arata has left #tripleo10:31
*** cd-undercloud has joined #tripleo10:32
cd-undercloud************** overcloud complete status=1 ************10:32
*** cd-undercloud has quit IRC10:32
openstackgerritA change was merged to openstack/tripleo-incubator: Add swift roles to keystone  https://review.openstack.org/5753210:34
GheRiveromorning10:38
*** akuznetsov has joined #tripleo10:38
*** martyntaylor has quit IRC10:39
*** martyntaylor has joined #tripleo10:49
*** cd-undercloud has joined #tripleo10:51
cd-undercloud************** overcloud complete status=1 ************10:51
*** cd-undercloud has quit IRC10:51
marios lifeless: still around?10:55
openstackgerritMarios Andreou proposed a change to openstack/tripleo-heat-templates: Make merge.py invokable from another script  https://review.openstack.org/5694711:00
*** arata has joined #tripleo11:09
*** arata has quit IRC11:09
*** nosnos_ has quit IRC11:20
*** nosnos has joined #tripleo11:21
*** nosnos has quit IRC11:25
*** cd-undercloud has joined #tripleo11:35
cd-undercloud************** overcloud complete status=1 ************11:35
*** cd-undercloud has quit IRC11:35
*** boris-42 has joined #tripleo11:45
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Wait for nova to initialise  https://review.openstack.org/5765512:02
derekhok, I thinkcd cloud is failing because https://review.openstack.org/#/c/57466/ has to be merged12:09
derekhbut merging causes a problem with seed/undercloud12:09
derekhso we need to merge https://review.openstack.org/#/c/57655/ first12:09
derekhany cores wanna review it ^^12:10
derekhany anybody else for that matter12:10
derekh:-)12:10
mariosderekh: looking12:10
derekhbrb12:10
*** morazi has joined #tripleo12:16
*** derekh changes topic to "FIREDRILL - unplanned work : cd-undercloud failing https://trello.com/b/0jIoMrdo/tripleo"12:19
*** jprovazn has quit IRC12:22
*** martyntaylor1 has joined #tripleo12:26
*** martyntaylor has quit IRC12:27
*** michchap has joined #tripleo12:28
openstackgerritA change was merged to openstack/tripleo-incubator: Timeout if seed fails to get IP address  https://review.openstack.org/5753112:40
openstackgerritA change was merged to openstack/tripleo-image-elements: Use upstream glance policy.json  https://review.openstack.org/5753012:41
*** jprovazn has joined #tripleo12:42
openstackgerritA change was merged to openstack/tripleo-heat-templates: enable Ironic password for keystone auth  https://review.openstack.org/5756412:43
*** lucasagomes is now known as lucas-hungry12:44
*** max_lobur_afk is now known as max_lobur12:46
*** pblaho has quit IRC12:54
*** rushiagr has quit IRC12:55
*** akrivoka has quit IRC12:55
*** rushiagr has joined #tripleo12:56
*** rdopieralski has joined #tripleo12:56
*** pblaho has joined #tripleo12:57
*** rdopieralski has quit IRC12:57
*** rdopieralski has joined #tripleo12:58
*** rdopieralski has quit IRC12:58
*** rdopieralski has joined #tripleo12:58
*** rdopiera1ski has joined #tripleo12:58
*** martyntaylor1 has quit IRC13:01
*** martyntaylor has joined #tripleo13:02
*** martyntaylor has left #tripleo13:03
*** akrivoka has joined #tripleo13:03
*** rdopiera1ski has quit IRC13:20
*** w_ has joined #tripleo13:23
*** olaph has quit IRC13:25
*** jdob has joined #tripleo13:34
*** CaptTofu has quit IRC13:39
*** CaptTofu has joined #tripleo13:39
marios  /exit13:41
*** marios has quit IRC13:41
*** marios has joined #tripleo13:41
*** jergerber has joined #tripleo13:45
*** noslzzp has joined #tripleo13:46
*** lucas-hungry is now known as lucasagomes13:48
*** jdob has quit IRC13:52
*** jdob has joined #tripleo13:52
*** jergerber has quit IRC13:52
*** dprince has joined #tripleo13:53
*** martyntaylor has joined #tripleo13:55
*** jergerber has joined #tripleo13:55
*** rushiagr has quit IRC13:56
*** tzumainn has joined #tripleo13:59
*** toci-bot has joined #tripleo14:02
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_WJI6eJZ/14:02
*** toci-bot has quit IRC14:02
*** w_ is now known as olaph14:05
*** jdob has quit IRC14:11
*** jdob has joined #tripleo14:11
*** rongze has joined #tripleo14:12
*** martyntaylor has quit IRC14:13
*** martyntaylor has joined #tripleo14:14
openstackgerritMarios Andreou proposed a change to openstack/tuskar: WIP : sketch of what using merge.py looks like for tuskar  https://review.openstack.org/5204514:18
*** martyntaylor has quit IRC14:18
*** markmc has joined #tripleo14:22
*** julim has joined #tripleo14:22
*** martyntaylor has joined #tripleo14:22
*** julim has quit IRC14:23
*** rongze_ has joined #tripleo14:24
*** rongze has quit IRC14:27
*** rongze has joined #tripleo14:30
*** rongze_ has quit IRC14:33
*** pblaho has quit IRC14:36
matelHi all - Is there any hypervisor that could be deployed with tripleo other than KVM?14:38
derekhmatel: not currently, but we always welcome patches :-)14:40
*** julim has joined #tripleo14:40
matelI am thinking about how to do that with XenServer.14:41
matelAnd I have a thought in my mind: "A bare metal driver for XenServer"14:42
matelSo that you deploy the hypervisor with TripleO, then you treat that as a bare metal provider, to provision your domU.14:42
*** marun has joined #tripleo14:43
matelbecause in case of XenServer, you install nova to a separate VM.14:43
matelWhat is the best forum to have these discussions?14:43
derekhright here, or a mail to openstack-dev@lists.openstack.org  afraid I don't know much about XenServer but what do describe at a high level looks ok14:45
matelIs there a mailing list for tripleo as well?14:45
matelThere seem to be a high traffic on dev.14:45
matelBut anyhow, I will drop a mail to the list, thanks derekh.14:46
derekhno seperate list, you should just tag the subject with [TripleO]14:46
derekhyup it is high traffic14:46
GheRiverothere is an ongoing thread about splitting the -dev list, but nothing yet confirmed14:46
*** CaptTofu has quit IRC14:50
Ngimo the list shouldn't be split14:53
Ngnot that my opinion is particularly important ;)14:53
Ngemail is cheap and filtering is easy :)14:53
NgI have gmail shove all of -dev into a tag that skips the inbox, except things with [blah] I'm interested in, so I see 100% relevant threads quickly, and I can scan over the rest at longer intervals14:54
GheRiveroNg: +1, filtering is cheap and space is no longer an issue14:54
Nghaving said that, if they do split it, I'll just subscribe to all the lists and have everything still go into a single gmail tag ;)14:55
NobodyCamGood morning TripleO15:03
*** rushiagr has joined #tripleo15:11
*** pblaho has joined #tripleo15:14
*** jtomasek has quit IRC15:19
*** vkozhukalov has quit IRC15:21
*** blamar has quit IRC15:22
*** jtomasek has joined #tripleo15:33
*** CaptTofu has joined #tripleo15:35
*** rongze_ has joined #tripleo15:37
*** rongze has quit IRC15:37
*** john-n-seattle1 has quit IRC15:44
*** boris-42 has quit IRC15:48
derekhanybody around to approve https://review.openstack.org/#/c/57655/ so we can cancel the firedrill (hopefully)15:51
*** dmojoryder has joined #tripleo15:54
dmojoryderRe: the baremetal fault tolerance challenges, has anyone tried something like pacemaker/corosync in an active/passive setup to address that in the interim?15:57
*** edmund has joined #tripleo16:00
*** ifarkas has quit IRC16:04
SpamapSmatel: re high traffic, you can just ask the mailing list server to send you certain tags, such as TripleO.16:06
*** rushiagr has quit IRC16:08
*** john-n-seattle1 has joined #tripleo16:09
*** john-n-seattle1 has quit IRC16:09
*** blamar has joined #tripleo16:10
*** akuznetsov has quit IRC16:13
*** CaptTofu has quit IRC16:17
*** CaptTofu has joined #tripleo16:17
*** rongze_ has quit IRC16:17
*** CaptTofu has quit IRC16:18
*** CaptTofu has joined #tripleo16:19
openstackgerritA change was merged to openstack/tuskar-ui: Use baremetal node id for node detail page  https://review.openstack.org/5754916:20
*** akrivoka has quit IRC16:22
*** rongze has joined #tripleo16:23
*** CaptTofu has quit IRC16:23
*** akrivoka has joined #tripleo16:27
*** CaptTofu has joined #tripleo16:28
*** pblaho has quit IRC16:41
Ngderekh: approved</lag>16:43
openstackgerritA change was merged to openstack/tripleo-incubator: Wait for nova to initialise  https://review.openstack.org/5765516:45
openstackgerritA change was merged to openstack/tripleo-image-elements: Refactor reset-db so we don't sync by default.  https://review.openstack.org/5746616:46
derekhNg: thanks, have approve the other patch, I think both together should sort out the cd-overcloud problems16:46
Ngwoot16:47
* derekh crosses his fingers16:47
*** derekh changes topic to "Using OpenStack to deploy OpenStack; meetings Tuesday 1900 UTC in #openstack-meeting-alt"16:47
SpamapStripleo-cd is broken again16:49
SpamapSroot       463 99.7  0.0 105140 18196 ?        R    11:35 312:14 git-remote-https origin https://git.openstack.org/cgit/openstack/swift16:49
SpamapScurl_multi_perform(0xea23b0, 0x7fff475165cc, 0x7fff475165cc, 0x7fff47516568, 0x7fff47516540) = 016:49
SpamapSCloning into '/root/.cache/image-create/repository-sources/d54dde99e66d3f592e7ac2dd2f4651fc5e21e693.tmp'...16:50
SpamapSswift clone16:50
SpamapSclarkb: ^^ broken repo of a different kind16:51
*** rushiagr has joined #tripleo17:01
*** akrivoka has quit IRC17:02
*** athomas has quit IRC17:05
*** boris-42 has joined #tripleo17:16
*** vkozhukalov has joined #tripleo17:17
Ngderekh: do you have that offline patch to hand again? the fedora pastebin seems to have expired it :/17:30
derekhNg: http://paste.fedoraproject.org/55794/05512313/17:32
derekhNg: i know its on fedora pastebin again17:33
derekhNg: its just to easy to pip into fpaste17:33
derekh*pipe17:33
Ngderekh: np, I'll grab it locally this time :)17:33
NgI think I'm close to done, just clearing up some weirdness with the seed image being rebuilt each time17:33
derekhNg: cool, -c to the seed vm script sorts that out17:34
Ngderekh: yeah I spaced on that and was modifying boot-seed-vm itself, but I'm running a test with -c now to see if it's happier17:35
derekhhmm, I wonder would anybody use a paste utility so they can pipe things to http://paste.openstack.org/17:35
* SpamapS points at pastebinit17:37
SpamapShttp://paste.openstack.org/show/5375817:37
SpamapSuse that for ~/.pastebinit.xml17:38
*** athomas has joined #tripleo17:38
derekhSpamapS: thanks17:39
* derekh puts away curl command17:39
*** martyntaylor has quit IRC17:39
*** jistr has quit IRC17:41
openstackgerritClint Byrum proposed a change to openstack/tripleo-image-elements: Fix PATH in pypi-mirror cron job  https://review.openstack.org/5772417:49
openstackgerritClint Byrum proposed a change to openstack/tripleo-image-elements: Log pypi-mirror cron job output to syslog  https://review.openstack.org/5772517:49
*** csd has joined #tripleo17:51
*** CaptTofu has quit IRC17:52
*** ccrouch has quit IRC17:53
SpamapShm17:53
SpamapSwhy are nova commands so damn slow on the undercloud?17:53
*** akuznetsov has joined #tripleo17:53
*** julim has quit IRC17:53
*** rpodolyaka1 has joined #tripleo17:55
derekhSpamapS: good question, I noticed this yesterday and first blamed dns but ruled that out with a curl to nova, that hit it fast, then I got distracted by something else17:55
SpamapS| POST http://localhost:5000/v2.0/tokens                                                                | 3.50206494331 |17:56
SpamapS| GET https://cd-undercloud.tripleo.org:13774//v2/4956c533154c476799c688eda7ed65ab/os-baremetal-nodes/4 | 9.6344318389917:56
SpamapS9.6 seconds to do what is basically a single row query17:56
SpamapSoh.. I wonder if stunnel is making things slower than they need to be17:56
SpamapSno.. stunnel is going pretty fast17:57
SpamapSwtf..17:58
SpamapSselect(0, NULL, NULL, NULL, {0, 9977})  = 0 (Timeout)17:59
SpamapSwait4(0, 0x7fffb4cb0300, WNOHANG, NULL) = 017:59
SpamapSselect(0, NULL, NULL, NULL, {0, 9977})  = 0 (Timeout)17:59
SpamapSnova-api just spins on that17:59
*** cd-undercloud has joined #tripleo18:00
cd-undercloud************** overcloud complete status=1 ************18:00
*** cd-undercloud has quit IRC18:00
*** derekh has quit IRC18:02
*** zaro has joined #tripleo18:02
*** boris-42 has quit IRC18:05
SpamapS6 seconds waiting for keystone to respond to /v2.0/tokens/revoked18:06
rpodolyaka1wow18:06
rpodolyaka1doesn't sound really scalable18:07
SpamapSits entirely possible we have a realy broken keystone version18:07
*** epim has joined #tripleo18:07
*** ccrouch has joined #tripleo18:11
*** panda_ has quit IRC18:13
lifelessSpamapS: - and this is why images: https://review.openstack.org/#/c/57527/18:15
openstackgerritBen Nemec proposed a change to openstack/tripleo-incubator: Set LIBVIRT_DEFAULT_URI in .bashrc  https://review.openstack.org/5773518:15
lifelesswut18:16
SpamapShttp://paste.openstack.org/show/5376118:16
SpamapSthat is the "give me the list of revoked tokens" query18:17
SpamapSit runs basically all the time18:17
* lifeless exercises -2.18:17
* Ng dinners18:17
SpamapSlifeless: hah yeah exactly on the "why images"18:17
SpamapS1960 rows in set (2.72 sec)18:18
*** boris-42 has joined #tripleo18:18
SpamapSThere isn't really a way to make this query much faster18:18
lifelesssure there is18:19
lifelesspass the revoked query being tested into the query18:19
*** boris-42 has quit IRC18:19
lifelessrevoked token18:20
SpamapSthe problem is that we are streaming 1960 11k json documents back to the client18:20
lifelessthat is indeed a problem18:20
lifelesswhy is the client asking for 1960 documents?18:20
SpamapSit will periodically ask for a list of revoked tokens18:20
SpamapSthe query I think just needs to be changed to grab the token ids.18:21
SpamapSlooking now but I don't think keystone actually uses the 'extra' field (the json thing) in this particular context18:21
SpamapSstill slow without the json docs, but half as slow18:22
SpamapS1.32s or so on average vs. 2.7218:22
lifelesswhy is the client asking for the tokens though18:22
lifelesswhy does a /client/ want all revoked tokens?18:22
SpamapSlifeless: nova is asking for the revoked token list18:22
SpamapSlifeless: I believe they've already landed patches for token validators like nova to keep a local cache of revoked tokens for a few minutes18:23
lifelessmmm18:23
lifelesseither consult the oracle or cache18:24
lifelessyeah18:24
SpamapSof course, memcached would make this faster...18:24
lifelessI would consuld the oracle directly18:24
lifeless'hey keystone is X revoked'18:24
lifelessand make that API call ms fast.18:24
lifelessbut hey, I'm strange like this.18:24
SpamapSlifeless: that would pound keystone to death which is what the non-pki token method does18:24
SpamapSlifeless: this only pounds keystone to unconsciousness18:25
lifelessSpamapS: why would it pound keystone to death ?18:25
SpamapSif we take out the expired check we return 35000 rows.. but we do so in 0.03s18:25
lifelessSpamapS: unindexed?18:25
SpamapSlifeless: every API request would have to ask keystone if tokens are valid.18:26
lifelesswhats the plan with the expiry check?18:26
lifelessSpamapS: sure, which means a single http persistent connection held open, doing 200 byte or so queries.18:26
SpamapSlifeless: see pastebin, it is indexed but it has to do a range query on expires18:26
lifelessSpamapS: to a scalable service.18:26
SpamapSlifeless: right, or perhaps we could just use a key/value store that scales like crazy directly. :)18:27
lifelessoh wow, I didn't even click that was an explain18:27
*** panda_ has joined #tripleo18:27
lifelessI'm so used to the detailed data that pg gives18:27
lifelessSpamapS: are range queries slow in mysql?18:27
lifelesspostgresql would be doing a btree index lookup and then a linear walk.18:27
SpamapSlifeless: no, but the range yields 34296 rows which must then be filtered for "valid"18:28
SpamapSthe 0.03s one uses a covering index18:28
lifelessSpamapS: composite index time ?18:28
SpamapSbasically just reads the b-tree and spits out the ids18:28
SpamapSlifeless: yep18:29
SpamapSlifeless: compound index to the rescue18:29
SpamapSlifeless: yephttp://paste.openstack.org/show/5376218:29
SpamapSyields 0.02s for the query18:29
SpamapSnow is where I look at current keystone and find that this index already exists..18:29
*** julim has joined #tripleo18:30
lifelessor not :)18:31
SpamapSyeah looks like they missed that one18:31
rpodolyaka1:)18:31
SpamapSdoh18:32
SpamapSok well the API call still fetches the entire row18:32
SpamapSand the uses only the id18:32
lifelessdon't run a cloud in production -> don't have a production product.18:32
SpamapSlifeless: I actually think that it is just that people are only using memcached in production.18:35
SpamapSjust dealing with the "oops I lost your tokens" problems.18:35
lifelessOops, I did it again :P18:36
SpamapSI played with your keys18:36
SpamapSgot lost in the RAM18:36
*** funzo has joined #tripleo18:36
*** funzo has quit IRC18:36
*** funzo has joined #tripleo18:37
lifelessOh no...18:37
*** cwolferh has joined #tripleo18:40
*** rongze has quit IRC18:40
SpamapS        query = session.query(TokenModel)18:41
SpamapS        query = query.filter(TokenModel.expires > now)18:41
SpamapSgod I hate ORMS18:41
SpamapSbasically "always give me all columns because hey who knows, I might want them. sometimes."18:41
lifelesssqlaclhemy can do much better18:42
rpodolyaka1it's fine, it's up to developers to use it correctly ;)18:42
SpamapSlifeless: so they're just being lazy?18:42
rpodolyaka1yep18:42
lifelessyes18:42
SpamapSbecause even without the covering index query, just pulling id and expires (all that we need in this case) is 0.02s but pulling token.extra is 2.5-3s18:42
rpodolyaka1should really be session.query(TokenModel.id)18:43
* SpamapS prepares two patches18:43
rpodolyaka1then18:43
lifelessquery = session.query(TokenModel.uuid).filter(TokenModel.expires > now).filter(TokenModel.valid==0)18:43
lifelessor whatever.18:43
SpamapSwhich will probably be rejected because they are going to explode the whole revoked thing anyway18:43
SpamapSlifeless: I need two fields, id and expires18:43
SpamapSso TokenModel.id, TokenModel.expires ?18:43
SpamapSinside session.query ?18:44
lifelesssession.query lists what you want to SELECT18:45
lifelessquery -> SELECT, filter -> WHERE18:45
*** cd-undercloud has joined #tripleo18:45
cd-undercloud************** overcloud complete status=2 ************18:45
*** cd-undercloud has quit IRC18:45
lifelessand filter -> HAVING (no need to select something just for a having clause)18:46
*** cd-undercloud has joined #tripleo18:47
cd-undercloud************** overcloud complete status=1 ************18:47
*** cd-undercloud has quit IRC18:47
lifelessrpodolyaka1: you're working on rebuild right?18:48
*** cd-undercloud has joined #tripleo18:49
cd-undercloud************** overcloud complete status=1 ************18:49
*** cd-undercloud has quit IRC18:49
rpodolyaka1lifeless: I was trying to figure why neutron failed yesterday (TLS bug) and did releases of projects today. So I'm going to be focused on rebuild starting from tomorrow18:50
rpodolyaka1two releases left, but it shouldn't take much time18:50
*** cd-undercloud has joined #tripleo18:51
cd-undercloud************** overcloud complete status=1 ************18:51
*** cd-undercloud has quit IRC18:51
SpamapSgah18:52
SpamapSwhy won't keystone log anything?18:52
rpodolyaka1lifeless: so the plan was to have a 'native' rebuild, not 'destroy + spawn', right?18:53
rpodolyaka1lifeless: I mean, nova doesn't seem to address this right now, because for VMs it was ok to use 'destroy + spawn' approach18:54
lifelessrpodolyaka1: thats right - https://review.openstack.org/#/c/54308/18:56
*** epim has quit IRC18:57
*** panda_ has quit IRC18:59
SpamapSnice..19:03
*** epim has joined #tripleo19:04
SpamapSok I monkey patched the keystone on the undercloud and now nova list "only" takes about 1.5s19:04
*** panda_ has joined #tripleo19:05
SpamapShttps://bugs.launchpad.net/keystone/+bug/1253755 filed19:09
uvirtbotLaunchpad bug 1253755 in keystone "keystone.token.backends.sql list_revoked_tokens performs very poorly" [Undecided,New]19:09
*** rongze has joined #tripleo19:11
*** akuznetsov has quit IRC19:11
*** lucasagomes has quit IRC19:13
*** akuznetsov has joined #tripleo19:17
SpamapSoh man19:18
SpamapSlike night and day19:18
*** rongze has quit IRC19:18
SpamapSfor i in $(nova baremetal-node-list | awk '{print $2}') ; do instance_id=$(nova baremetal-node-show $i | awk '/instance_uuid/ {print $4}') ; echo $i,$instance_id; done19:18
*** rushiagr has quit IRC19:19
SpamapSthat ties hardware to instance ids..19:19
SpamapStook around 5 minutes before the keystone fix..19:19
SpamapSnow about 45s19:19
lifelessgreat19:23
SpamapSwow.. keystone's unit tests are .. brutal19:23
SpamapS20964 clint     20   0  785m 663m 7608 D  14.3  4.2   3:51.80 python19:24
*** max_lobur is now known as max_lobur_afk19:27
*** jtomasek has quit IRC19:28
openstackgerritClint Byrum proposed a change to openstack/tripleo-image-elements: In tripleo-cd log hardware to instance map  https://review.openstack.org/5774919:34
*** cd-undercloud has joined #tripleo19:35
cd-undercloud************** overcloud complete status=2 ************19:35
*** cd-undercloud has quit IRC19:35
SpamapSService identity created with password e74aad52188b894c16f690af0d87889ecc1ba02119:37
SpamapSAuthorization Failed: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP Unable to establish connection t19:37
SpamapSo http://138.35.77.4:5000/v2.0/tokens)19:37
SpamapSoh whoops19:37
SpamapShaha19:37
* SpamapS quietly changes that password19:37
*** ccrouch has quit IRC19:39
SpamapS<attribute 'message' of 'exceptions.BaseException' objects> (HTTP Unable to establish connection to http://138.35.77.4:319:39
SpamapS5357/v2.0/OS-KSADM/services)19:39
SpamapSseems that we may just be having problems talking to the overcloud's keystone19:39
*** rongze has joined #tripleo19:45
*** akuznetsov has quit IRC19:46
*** rongze has quit IRC19:50
*** rpodolyaka1 has quit IRC19:51
*** epim has quit IRC19:51
*** panda_ has quit IRC19:59
Ngnice, 18m42.125s for an offline run of devtest.sh20:08
openstackgerritChris Jones proposed a change to openstack/tripleo-incubator: Add offline support to devtest.sh.  https://review.openstack.org/5775520:08
lifelessSpamapS: hah20:10
lifelessSpamapS: we'll want to sanitise that when we get logstash :)20:10
*** vipul is now known as vipul-away20:15
*** vipul-away is now known as vipul20:15
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Update nova-kvm to destroy the default libvirt net  https://review.openstack.org/5776320:22
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Update nova-kvm to destroy the default libvirt net  https://review.openstack.org/5776320:23
*** cd-undercloud has joined #tripleo20:25
cd-undercloud************** overcloud complete status=1 ************20:25
*** cd-undercloud has quit IRC20:25
*** CaptTofu has joined #tripleo20:31
*** ccrouch has joined #tripleo20:34
*** vipul is now known as vipul-away20:38
*** vipul-away is now known as vipul20:38
*** rongze has joined #tripleo20:47
*** markmc has quit IRC20:50
jamezpolleyLooking at Chris Jones' offline support review, I notice there are some lines that have been left with their original indentation; I first thought that was to make for a nice clean git blame, but then I realised that other lines right beside them had been reindented20:50
jamezpolleyIs there a style guide that covers this kind of thing?20:50
*** rongze has quit IRC20:51
*** vipul is now known as vipul-away20:53
*** vipul-away is now known as vipul20:54
lifelessjamezpolley: interesting, url me up?20:54
*** csd has quit IRC20:56
*** csd has joined #tripleo20:56
jamezpolleyhttps://review.openstack.org/#/c/57755/1/scripts/devtest.sh - line 16420:58
lifelessjamezpolley: I'm now in two meetings at once, so will be low bw to look at this for ~ 60m21:02
*** dprince has quit IRC21:03
*** cd-undercloud has joined #tripleo21:10
cd-undercloud************** overcloud complete status=2 ************21:10
*** cd-undercloud has quit IRC21:10
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface  https://review.openstack.org/5777621:14
*** asalkeld has joined #tripleo21:19
*** marun has quit IRC21:23
*** ccrouch has quit IRC21:37
*** ccrouch has joined #tripleo21:38
*** rongze has joined #tripleo21:47
*** rongze has quit IRC21:52
*** CaptTofu has quit IRC21:52
*** CaptTofu has joined #tripleo21:53
*** cd-undercloud has joined #tripleo21:59
cd-undercloud************** overcloud complete status=1 ************21:59
*** cd-undercloud has quit IRC21:59
*** julim has quit IRC22:03
*** athomas has quit IRC22:03
*** jdob has quit IRC22:04
lifelessjog0: wooo, so we have jog slices again ?22:09
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack/tripleo-incubator: Add create-nodes option for baremetal interface  https://review.openstack.org/5777622:09
jog0lifeless: soon yeah22:09
lifelesswoo woo woo22:09
jprovaznHi, what is the best source to get list of tasks/features we want/need for Icehouse? I'm looking at etherpads linked from https://wiki.openstack.org/wiki/Summit/Icehouse/Etherpads#TripleO_.28Deployment.29 , anything else I should read (what would help me to get on the same page with upstream plans)?22:13
lifelessjprovazn: hi, so the icehouse summit schedule is what I'd pull from22:14
lifelessjprovazn: + trello22:14
lifelessjprovazn: https://wiki.openstack.org/wiki/Summit/Icehouse/Etherpads#TripleO_.28Deployment.2922:14
jog0lifeless: another few days and things should be calmer (have to finish up a summery email) and make sure the next steps arein place22:17
jprovaznlifeless, thanks, ^ the icehosue summit schedule is the same I mentioned above, right?22:17
jprovaznlifeless, trello is for CD only or for for general tripleo?22:18
jprovaznha, it's not, cool22:20
lifelessgeneral TripleO22:24
*** tzumainn has quit IRC22:25
lifelessjamezpolley: oh hai22:27
lifelessjamezpolley: so, his patch needs review, were you asking about what is appropriate to say in such a review?22:27
jamezpolleyRoughly, although I'd say it's more about how you review rather than what to review. Do you have a bash style guide (I've found out the answer is no); do you have guides/traditions about preserving a clean git blame vs clean indentation (although in the absence of a style guide maybe the idea of "clean" indentation is meaningless)22:30
*** vkozhukalov has quit IRC22:31
lifelessoh, no we don't particularly care about clean git blame22:34
*** michchap has quit IRC22:39
*** michchap has joined #tripleo22:42
*** asalkeld has quit IRC22:44
*** rongze has joined #tripleo22:48
*** ccrouch has quit IRC22:54
*** rongze has quit IRC22:54
*** cd-undercloud has joined #tripleo22:55
cd-undercloud************** overcloud complete status=1 ************22:55
*** cd-undercloud has quit IRC22:55
lifelessjamezpolley: there are docs (search for review) on the openstack wiki about review process/goals/etc22:56
*** jprovazn has quit IRC22:57
*** asalkeld has joined #tripleo23:04
SpamapSseems like we have weird internet connecting problems on cd-undercloud23:15
lifelessSpamapS: does host www.<somthing>.com in a while loop error?23:22
*** ccrouch has joined #tripleo23:24
SpamapSlifeless: not when done for a few seconds, but the problems I'm seeing are more like every 10th try a curl or a git pull fails23:24
*** CaptTofu has quit IRC23:25
*** CaptTofu has joined #tripleo23:26
clarkbSpamapS: so it isn't just git.o.o?23:28
Ngwoot, my tripleo talk for FLOSS UK 2014 has been accepted \o/23:29
Ngcody-somerville: (I'll forward you the details of that tomorrow morning when I'm at the machine with HP mail on it :)23:30
pleia2congrats :)23:30
*** CaptTofu_ has joined #tripleo23:30
SpamapSclarkb: right, I think there's something else mussed up23:31
*** CaptTofu has quit IRC23:31
*** asalkeld has quit IRC23:32
*** victor_lowther has joined #tripleo23:46
SpamapSlifeless: found a node that doesn't seem to want to PXE boot23:46
SpamapSfreecloud002723:46
*** dmojoryder has left #tripleo23:49
*** rongze has joined #tripleo23:50
*** cd-undercloud has joined #tripleo23:51
cd-undercloud************** overcloud complete status=1 ************23:51
*** cd-undercloud has quit IRC23:51
*** rongze has quit IRC23:56
*** michchap has quit IRC23:58
*** michchap has joined #tripleo23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!