Sunday, 2014-01-19

*** sdake has quit IRC00:14
*** ccrouch has joined #tripleo00:56
*** ccrouch has joined #tripleo00:57
*** akuznetsov has quit IRC01:08
*** akuznetsov has joined #tripleo01:13
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Add more to the json file  https://review.openstack.org/6768601:15
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Use BASH_SOURCE[0] to find scripts directory  https://review.openstack.org/6768701:16
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Upgrade setuptools to 1.10 or more in a newly created virtualenv  https://review.openstack.org/6768801:16
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Change communication with remote host  https://review.openstack.org/6768901:16
derekhlifeless: got to the point where the seed vm is booting on the testenv, doesn't look like it got a IP but I havn't looked into it yet,01:19
derekhlifeless: pretty much all my patches are needed to do this along with the instructions on the etherpad.01:20
derekhlifeless: One problem I hit and just kinda ignored is that DIB was failling towards the end of the image build, didn't even look at how to fix and just tried a cirros image instead01:21
*** derekh has quit IRC01:24
lifelessdevananda: cool01:35
lifelessbah01:35
lifelessdevananda: sorry, tab fail :)01:35
*** akuznetsov has quit IRC01:52
lifelesshuh02:04
lifelessthere's no dkms datapath module in trusty?02:04
lifelessoh, missing universe I bet02:05
openstackgerritA change was merged to openstack-infra/tripleo-ci: Get IP for Test Host from neutron  https://review.openstack.org/6694202:30
*** ccrouch1 has joined #tripleo02:49
*** ccrouch has quit IRC02:52
lifelessci-overcloud controller down while i try to upgrade ovs on it03:06
openstackgerritA change was merged to openstack-infra/tripleo-ci: Get seed IPs from configured network  https://review.openstack.org/6694303:07
openstackgerritA change was merged to openstack-infra/tripleo-ci: Change to tripleo-ci directory  https://review.openstack.org/6722103:08
openstackgerritA change was merged to openstack-infra/tripleo-ci: Change path to tripleo-incubator  https://review.openstack.org/6722203:08
*** tzumainn has joined #tripleo03:28
*** noslzzp has quit IRC03:39
*** noslzzp has joined #tripleo04:04
*** noslzzp has quit IRC04:26
*** noslzzp has joined #tripleo04:26
*** RomAbptTHN has joined #tripleo04:33
*** RomAbptTHN has left #tripleo04:33
*** noslzzp has joined #tripleo05:05
*** vkozhukalov has joined #tripleo05:38
*** noslzzp has quit IRC05:56
*** zIWYIiWPIS has joined #tripleo06:05
*** zIWYIiWPIS has left #tripleo06:05
*** tzumainn has quit IRC06:39
lifelessoh joy06:58
lifeless    @six.add_metaclass(abc.ABCMeta)06:58
lifelessAttributeError: 'module' object has no attribute 'add_metaclass'06:58
*** rlandy has joined #tripleo07:11
*** bnemec_ has joined #tripleo07:24
*** bnemec has quit IRC07:25
*** csd has quit IRC07:25
*** lifeless has quit IRC07:25
*** rbrady has quit IRC07:26
*** rbrady has joined #tripleo07:26
*** lifeless has joined #tripleo07:27
*** ChanServ sets mode: +v lifeless07:27
*** csd has joined #tripleo07:27
*** akuznetsov has joined #tripleo08:07
*** julim has joined #tripleo08:21
*** julim has quit IRC08:25
*** derekh has joined #tripleo08:27
derekhlifeless: for your problems with python.six https://review.openstack.org/#/c/67178/08:30
lifelessderekh: hey!08:31
lifelessderekh: I'm currently debugging a weird gre tunnel issue08:31
lifelessderekh: gre packets from the compute node to the controller node are not hitting the openflow flow on br-tun on the control node08:32
derekhlifeless: cool, the net slowness is a PITA, I've been working around it with a proxy on the undercloud08:32
lifelessderekh: they do hit br-untagged08:32
lifelessderekh: I upgraded the network node to ovs 2.0.108:33
lifelessI'm pondering whether upgrading the hypervisor will make it better... or worse ;)08:33
derekhlifeless: it can't get much worse08:34
lifelesshah08:34
lifelessI see no flows on the gre incoming port (13) in dpctl dump-flows on the controller08:40
lifelessbut there is one on the hypervisor08:41
lifelesstunnel(tun_id=0x1,src=10.10.16.175,dst=10.10.16.176,tos=0x0,ttl=64,flags(key)),in_port(3),eth(src=fa:16:3e:c7:fd:70,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=10.0.0.1,tip=10.0.0.6,op=1,sha=fa:16:3e:c7:fd:70,tha=00:00:00:00:00:00), packets:3664, bytes:153888, used:0.424s, actions:push_vlan(vid=1,pcp=0),1,pop_vlan,8,16,14,push_vlan(vid=1,pcp=0),6,pop_vlan,12,1008:41
derekhlifeless: a lot of the openslow stuff goes over my head, the only I've seen missing flows it was because for firewall on fedora blocking gre but I guess that wouldn't be an issue here08:42
lifelessI checked that already :)08:42
derekh*openflow08:42
* derekh wonders if openslow was a freudian slip08:43
lifelessso definitely a missing flow08:46
lifelesswonder wtf that is08:46
lifelessovs-appctl dpif/dump-flows br-tun08:46
lifelesstunnel(tun_id=0x1,src=10.10.16.175,dst=10.10.16.176,tos=0x0,ttl=64,flags(key)),in_port(3),eth(src=fa:16:3e:c7:fd:70,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=10.0.0.1,tip=10.0.0.6,op=1,sha=fa:16:3e:c7:fd:70,tha=00:00:00:00:00:00), packets:3963, bytes:166446, used:0.756s, actions:push_vlan(vid=1,pcp=0),1,pop_vlan,8,16,14,push_vlan(vid=1,pcp=0),6,pop_vlan,12,1008:46
lifelessin_port(2),eth(src=56:96:98:5e:94:4a,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(src=0.0.0.0,dst=255.255.255.255,proto=17,tos=0x10,ttl=128,frag=no),udp(src=68,dst=67), packets:0, bytes:0, used:4.610s, actions:drop08:46
lifelessthats from the hypervisor08:46
lifelessovs-appctl dpif/dump-flows br-tun08:46
lifeless#08:46
lifelessfrom the controller08:46
lifelessderekh: it's worth getting up to speed on ovs08:49
lifelesshttp://openvswitch.org/slides/OVS-Debugging-110414.pdf for instance08:49
derekhlifeless: thanks been meaning to for a while now, will move it up the list of priorities08:50
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Change metadata names to match  https://review.openstack.org/6760009:00
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Add more to the json file  https://review.openstack.org/6768609:00
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Switch test environment users  https://review.openstack.org/6761409:00
derekhlifeless: gotta run again, just wanted to unstick that branch (I've removed the heat resource name patch will sort that out later)09:01
*** derekh has quit IRC09:02
*** kpavel has joined #tripleo09:13
lifelessok, same behaviour with the hypervisor09:25
lifelessfark09:25
*** e0ne has joined #tripleo10:37
*** e0ne has quit IRC10:44
*** e0ne has joined #tripleo10:46
*** e0ne has quit IRC10:59
*** e0ne has joined #tripleo10:59
*** d0ugal has joined #tripleo11:42
*** d0ugal has quit IRC12:05
*** d0ugal has joined #tripleo12:06
*** d0ugal has quit IRC12:14
*** rushiagr has joined #tripleo12:33
*** rushiagr has quit IRC12:47
*** rushiagr_ has joined #tripleo12:48
*** rushiagr has joined #tripleo12:49
*** rushiagr_ has quit IRC12:56
*** e0ne has quit IRC13:44
*** e0ne has joined #tripleo13:45
*** e0ne has quit IRC14:01
*** e0ne has joined #tripleo14:02
*** e0ne has quit IRC14:30
*** rlandy has quit IRC14:51
*** akrivoka has joined #tripleo14:57
*** e0ne has joined #tripleo15:01
*** e0ne has quit IRC15:02
*** e0ne has joined #tripleo15:03
*** e0ne has quit IRC15:05
*** bauzas has joined #tripleo15:08
*** d0ugal has joined #tripleo15:21
*** coolsvap has joined #tripleo15:25
*** e0ne has joined #tripleo15:29
*** akrivoka has quit IRC15:29
*** kpavel has quit IRC15:30
*** noslzzp has joined #tripleo15:55
*** noslzzp has quit IRC15:56
*** tzumainn has joined #tripleo16:06
*** ccrouch1 has quit IRC16:38
*** ccrouch has joined #tripleo16:38
*** ccrouch has quit IRC16:38
*** ccrouch has joined #tripleo16:38
*** sdake has joined #tripleo16:47
*** shadower has joined #tripleo17:10
*** ccrouch has quit IRC17:15
*** ccrouch has joined #tripleo17:23
*** sdake has quit IRC17:28
*** sdague has quit IRC17:30
*** sdague has joined #tripleo17:30
*** coolsvap has quit IRC17:50
*** ohadlevy_ is now known as ohadlevy18:03
*** ohadlevy has joined #tripleo18:03
*** csd has quit IRC18:19
*** akuznetsov has quit IRC18:20
*** bauzas has quit IRC18:29
*** rlandy has joined #tripleo19:06
*** bauzas has joined #tripleo19:20
lifelessSpamapS: which reminds me; why does oac know the occ cache path?19:46
*** sdake has joined #tripleo19:48
rlandylifeless: hi - are there unit (or other specific) tests for tripleo?19:50
lifelessrlandy: no - tripleo is a program not a project; for specific projects like e.g. os-collect-config, yes we have tests.19:51
lifelessNot all projects do, and we're planning on tempest being the bulk of tests once we're at the 'it works in a CI loop' stage19:52
*** vkozhukalov has quit IRC19:58
*** rlandy has quit IRC20:17
*** noslzzp has joined #tripleo20:27
openstackgerritA change was merged to openstack-infra/tripleo-ci: Change metadata names to match  https://review.openstack.org/6760020:44
*** panda___ has joined #tripleo21:23
*** e0ne has quit IRC21:24
*** panda has quit IRC21:27
SpamapSlifeless: it doesn't have to, but it has been able to live without a config file and we'd still need a default.21:30
lifelessSpamapS: I'm just wondering whether the layering is very wrong21:30
lifelessSpamapS: like21:30
lifelessSpamapS: maybe we shouldn't be offering oac direct use.21:30
lifelessSpamapS: maybe it should be o-c-c --cached --run -- o-a-c --blah21:31
SpamapSI'm entirely o-k with drawing that line.21:31
lifelesscause right now oac works by knowing about occ plumbing21:31
SpamapSwell I don't think we ever want to run os-apply-config without the full o-r-c run really21:31
lifelesswhich makes oac usage with e.g. chef or puppet harder21:31
SpamapSit works that way out of convenience, and -m exists for such programs to override said plumbing21:32
SpamapSbut I'm fine with eliminating that.21:32
SpamapS(also OS_CONFIG_FILES exists and overrides that happenstance knowledge too)21:33
lifelessfuckity fuck fuck fuck21:33
lifelessup10:32 <mordred> it is. when I say "have him do" I mean "I'm going to point the problem out to him"21:33
lifeless10:33 <lifeless> ok :)21:33
lifeless10:33 <mordred> otoh, he's the only hardcore js hacker, so there is a certain amount of self-selecting that21:33
lifeless                goes on21:33
lifelessupstream kernel ovs datapath also naffed21:33
SpamapSlifeless: seriously, I think it is fine to just say "this always has to be informed where the input is"21:33
SpamapSlifeless: cat on the middle click? ;)21:34
lifelessthat too21:34
lifelessroot@ci-overcloud-notcompute-7sdhmmrtddid:/etc/apt# 2014-01-19T21:35:06.839Z|00014|ofproto_dpif_upcall(miss_handler)|INFO|received packet on unassociated datapath port 021:35
lifelessbut mainly openvswitch woes21:35
SpamapSlifeless: so you're trying 2.0 and it is also not improving the throughput?21:36
lifelessSpamapS: http://lists.openstack.org/pipermail/openstack-operators/2014-January/003893.html21:37
lifelessSpamapS: (are you not on operators?)21:37
SpamapSI am not21:38
lifelessah :)21:38
SpamapSbecause I am not an operator. :)21:38
lifelessSpamapS: you are21:38
SpamapSworth adding more email reading time to me?21:38
lifelessSpamapS: its fairly low traffic21:38
SpamapSlifeless: ok21:40
SpamapSI did recently unsub from all of the juju mailing lists.. :)21:40
SpamapSlifeless: oh and doh you're right I am an operator now. :-P21:40
lifeless:)21:44
SpamapSlifeless: btw, Heat should be fine behind stunnel now21:49
lifelessSpamapS: cool; that bug is closed ?21:51
SpamapSlifeless: Fix Committed in Heat. Did we open a bug in TripleO?21:52
lifelessSpamapS: don't think so21:52
lifelessgnnnnar21:56
lifeless2.0.0 release userspace with ubuntu 3.11 kernel module fails in the same way21:57
lifelesstrying 1.1121:58
SpamapSlifeless: what about 3.13 from trusty?21:58
lifelessSpamapS: haven't tried that yet21:58
lifelessalso haven't tried removing br-untagged21:58
lifelessand running against the raw eth221:58
lifelessoh thats interesting22:02
lifelessnow I'm not getting the arp responses from the hypervisor host etf22:02
lifelesswtf22:02
lifelessnow stab22:08
lifeless2014-01-19T22:07:07Z|00051|dpif|WARN|system@ovs-system: failed to add gre-10.10.16.175 as port: Address family not supported by protocol22:08
lifelessok trying 1.11 with kernel datapath on both machines22:14
lifelesswow, so much better. How I get 2.7KB/s.22:14
lifelessW.22:14
lifelessT.22:14
lifelessF.22:14
lifelessyanked the mtu down to 1400, and booyah22:15
lifelessso its a pmtud problem22:15
lifelessgoing to pull on the ovs version thing a little more22:16
lifelesshttps://bugs.launchpad.net/tripleo/+bug/127064622:20
lifelessI don't believe it...22:33
lifeless     Resume: pmtu 65535 hops 19 back 5022:33
lifelessnothing has a 64K pmtu across the internets22:34
lifelesshttps://bugs.launchpad.net/tripleo/+bug/127064622:36
*** bauzas has quit IRC22:44
openstackgerritSteve Kowalik proposed a change to openstack/diskimage-builder: Allow use of mirrors when building Ubuntu images  https://review.openstack.org/6698423:40
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Add a new remote ops JSON value to boot-seed-vm  https://review.openstack.org/6736623:48

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!