| greghaynes | lifeless: that one got hit by the invalid checksum for trusty to | 00:00 |
|---|---|---|
| *** yfaykya has joined #tripleo | 00:00 | |
| lifeless | greghaynes: ack | 00:00 |
| *** yfaykya has quit IRC | 00:05 | |
| *** yfaykya has joined #tripleo | 00:06 | |
| SpamapS | lifeless: Oh the revert for heat is landed. | 00:07 |
| SpamapS | lifeless: 2 hours ago | 00:07 |
| *** yfaykya has quit IRC | 00:11 | |
| *** yfaykya has joined #tripleo | 00:12 | |
| *** yamahata has joined #tripleo | 00:14 | |
| *** hrushig_ has quit IRC | 00:15 | |
| *** yfaykya has quit IRC | 00:17 | |
| xuhaiwei | Hi, i rebuilded devtest, but it failed in building undercloud, could anyone see this? https://bugs.launchpad.net/tripleo/+bug/1331867 | 00:18 |
| uvirtbot | Launchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New] | 00:18 |
| *** yfaykya has joined #tripleo | 00:19 | |
| *** matsuhashi has joined #tripleo | 00:20 | |
| vinsh | what does the charset part at then end of this mean? | 00:21 |
| vinsh | ovs_db: mysql://neutron:unset@localhost/ovs_neutron?charset=utf8 | 00:21 |
| vinsh | and.. shouldn't this point to a virtual ip in a multi controller setup? | 00:21 |
| greghaynes | yes | 00:21 |
| greghaynes | although we havent dont that part (mysql behind haproxy) yet | 00:21 |
| vinsh | it seems that the controller nodes can't on setup.. as if its not setup yet | 00:21 |
| greghaynes | maybe I should implement this | 00:22 |
| * greghaynes hacks | 00:22 | |
| *** yfaykya has quit IRC | 00:23 | |
| *** yfaykya has joined #tripleo | 00:25 | |
| *** nati_ueno has quit IRC | 00:26 | |
| vinsh | man.. NeutronDSN.. | 00:27 |
| * vinsh 'splodes | 00:27 | |
| lifeless | greghaynes: stil sploding | 00:28 |
| *** ramishra has joined #tripleo | 00:28 | |
| lifeless | 2014-06-19 00:07:46.977 | + grep trusty-server-cloudimg-amd64-root.tar.gz SHA256SUMS.ubuntu.trusty.amd64 | 00:28 |
| lifeless | 2014-06-19 00:07:48.434 | trusty-server-cloudimg-amd64-root.tar.gz: FAILED | 00:28 |
| lifeless | 2014-06-19 00:07:48.437 | sha256sum: WARNING: 1 computed checksum did NOT match | 00:28 |
| * greghaynes does sha check on personal box | 00:29 | |
| lifeless | I bet we have a squid cache in that region | 00:29 |
| *** yfaykya has quit IRC | 00:29 | |
| greghaynes | yep, must be a cache issue | 00:30 |
| greghaynes | the sha checked out for me | 00:31 |
| *** yfaykya has joined #tripleo | 00:31 | |
| lifeless | what we need is cache-bust-on-sha-fail | 00:31 |
| StevenK | lifeless: Ng has a patch up for that | 00:31 |
| lifeless | StevenK: Idon't see it in dib | 00:32 |
| SpamapS | lifeless: I thought there was a patch in flight that did something like that | 00:32 |
| *** ramishra has quit IRC | 00:32 | |
| lifeless | ah its there but failed CI | 00:33 |
| lifeless | 88084 | 00:33 |
| StevenK | I just found it myself | 00:33 |
| SpamapS | haha | 00:33 |
| SpamapS | we all just wasted time in parallel | 00:33 |
| lifeless | it passed our functional tests | 00:34 |
| *** yfaykya has quit IRC | 00:35 | |
| *** CaptTofu_ has quit IRC | 00:36 | |
| *** yfaykya has joined #tripleo | 00:37 | |
| *** CaptTofu_ has joined #tripleo | 00:37 | |
| *** yfaykya has quit IRC | 00:41 | |
| *** CaptTofu_ has quit IRC | 00:41 | |
| *** chuckC has quit IRC | 00:42 | |
| *** yfaykya has joined #tripleo | 00:43 | |
| *** nosnos has joined #tripleo | 00:43 | |
| vinsh | very confused here.. compute nodes try to talk to ovs_db ending in a url of "neutron" where as the controller nodes are attempting at "ovs_neutron" | 00:43 |
| vinsh | yet.. control nodes can not talk to "ovs_neutron" sql | 00:44 |
| vinsh | wish I knew what each was. :) | 00:44 |
| SpamapS | AFAIK there is no "neutron" database | 00:45 |
| SpamapS | ovs_neutron is the only one | 00:45 |
| lifeless | vinsh: pastebin ? | 00:46 |
| vinsh | checkout overcloud-source.yaml | 00:46 |
| vinsh | for ovs_db on compute nodes they reference: | 00:47 |
| vinsh | NeutronDSN: {"Fn::Join": ['', ['mysql://neutron:unset@', {'Fn::Select': [ip_address, 'Fn::Select': [0, 'Fn::GetAtt': [ControlVirtualIP, fixed_ips]]]}, '/neutron']]} | 00:47 |
| SpamapS | vinsh: that may be erroneous, because IIRC, the agents only speak over RPC | 00:47 |
| SpamapS | so compute nodes aren't accessing neutron's DB | 00:47 |
| vinsh | yet controller try for " ovs_db: mysql://neutron:unset@localhost/ovs_neutron?charset=utf8" | 00:47 |
| vinsh | which fails | 00:47 |
| vinsh | hrm. | 00:48 |
| *** yfaykya has quit IRC | 00:48 | |
| vinsh | I wonder which element initializes the ovs_neutron entries? | 00:48 |
| lifeless | huh I thought the agents spoke to DB too | 00:48 |
| vinsh | I have all the controllers talking to mysql on one node for now.. | 00:48 |
| lifeless | because we had races with neutrons special approach to schema applications | 00:49 |
| vinsh | and everything works.. but this ovs_db business. | 00:49 |
| vinsh | causes a stack trace on " | 00:49 |
| vinsh | + neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" | 00:49 |
| *** yfaykya has joined #tripleo | 00:49 | |
| lifeless | there's something wrong for sure | 00:49 |
| vinsh | orly? ;) | 00:49 |
| lifeless | :) | 00:50 |
| vinsh | ah. ./elements/boot-stack/bin/reset-db:os-db-create ovs_neutron neutron $db_pass | 00:51 |
| greghaynes | its possible were just creating the db and not actually using it? | 00:51 |
| lifeless | ok -> gym with C, bbiac hours | 00:52 |
| vinsh | compute nodes are using "neutron" control nodes are trying for "ovs_neutron" | 00:52 |
| *** chuckC has joined #tripleo | 00:52 | |
| *** yfaykya has quit IRC | 00:53 | |
| lifeless | well this might explain the epic vif plugging failures we have atm | 00:54 |
| vinsh | what I am rambling about? or something else? | 00:54 |
| *** yfaykya has joined #tripleo | 00:55 | |
| *** yfaykya has quit IRC | 01:00 | |
| *** TravT has quit IRC | 01:01 | |
| *** yfaykya has joined #tripleo | 01:01 | |
| *** dprince has joined #tripleo | 01:03 | |
| dprince | CI update: the undercloud jobs all seem to fail due to https://bugs.launchpad.net/tripleo/+bug/1331874 | 01:04 |
| uvirtbot | Launchpad bug 1331874 in tripleo "trusty-server-cloudimg-amd64 checksum mismatch" [High,Triaged] | 01:04 |
| dprince | Is there anything we can do to work around that? ^^^ | 01:04 |
| *** yfaykya has quit IRC | 01:06 | |
| *** yfaykya has joined #tripleo | 01:08 | |
| *** CaptTofu_ has joined #tripleo | 01:08 | |
| michchap | Hi folks, is precise supported for running devtest? | 01:09 |
| vinsh | 13.04 -14.10 | 01:09 |
| vinsh | Can't say for precise :) | 01:09 |
| StevenK | michchap: It *may* work with the Ubuntu Cloud Archive, but as vinsh says | 01:09 |
| *** spzala has quit IRC | 01:10 | |
| michchap | Thanks. Just trying to evaluate tripleo rather than deploy anything serious at this point. Is devtest what I should be using for that? | 01:10 |
| *** dprince has quit IRC | 01:11 | |
| *** yfaykya has quit IRC | 01:12 | |
| *** yfaykya has joined #tripleo | 01:14 | |
| *** yfaykya has quit IRC | 01:18 | |
| *** yfaykya has joined #tripleo | 01:20 | |
| *** yfaykya has quit IRC | 01:24 | |
| *** eguz has joined #tripleo | 01:25 | |
| *** yfaykya has joined #tripleo | 01:26 | |
| *** ramishra has joined #tripleo | 01:29 | |
| *** eghobo has quit IRC | 01:29 | |
| *** mestery has joined #tripleo | 01:29 | |
| *** yfaykya has quit IRC | 01:30 | |
| *** yfaykya has joined #tripleo | 01:32 | |
| *** ramishra has quit IRC | 01:34 | |
| *** mestery has quit IRC | 01:34 | |
| *** yfaykya has quit IRC | 01:37 | |
| *** yfaykya has joined #tripleo | 01:38 | |
| *** eguz has quit IRC | 01:42 | |
| *** yfaykya has quit IRC | 01:43 | |
| *** yfaykya has joined #tripleo | 01:44 | |
| *** yfaykya has quit IRC | 01:49 | |
| *** CaptTofu_ has quit IRC | 01:50 | |
| *** CaptTofu_ has joined #tripleo | 01:50 | |
| *** yfaykya has joined #tripleo | 01:50 | |
| *** mestery has joined #tripleo | 01:54 | |
| *** CaptTofu_ has quit IRC | 01:55 | |
| *** yfaykya has quit IRC | 01:55 | |
| *** yfaykya has joined #tripleo | 01:57 | |
| *** yfaykya has quit IRC | 02:01 | |
| *** yfaykya has joined #tripleo | 02:03 | |
| *** yfaykya has quit IRC | 02:07 | |
| *** matsuhashi has quit IRC | 02:07 | |
| *** matsuhashi has joined #tripleo | 02:08 | |
| *** matsuhashi has quit IRC | 02:08 | |
| *** matsuhashi has joined #tripleo | 02:08 | |
| *** yfaykya has joined #tripleo | 02:09 | |
| *** yfaykya has quit IRC | 02:13 | |
| *** yfaykya has joined #tripleo | 02:15 | |
| *** yfaykya has quit IRC | 02:19 | |
| lifeless | back | 02:20 |
| *** mestery_ has joined #tripleo | 02:20 | |
| *** yfaykya has joined #tripleo | 02:21 | |
| *** mestery_ has quit IRC | 02:21 | |
| *** mestery_ has joined #tripleo | 02:22 | |
| lifeless | ok 60 seconds wasn't enoug | 02:22 |
| lifeless | enough | 02:22 |
| *** mestery has quit IRC | 02:23 | |
| *** yfaykya has quit IRC | 02:26 | |
| *** yfaykya has joined #tripleo | 02:27 | |
| *** CaptTofu_ has joined #tripleo | 02:29 | |
| *** ramishra has joined #tripleo | 02:29 | |
| *** bnemec has quit IRC | 02:30 | |
| lifeless | greghaynes: so ping | 02:31 |
| greghaynes | such ping? | 02:31 |
| lifeless | greghaynes: AIUI mysql isn't accessible through haproxy yet, right ? | 02:31 |
| greghaynes | yes | 02:31 |
| lifeless | so all these NeutronDSN: {"Fn::Join": ['', ['mysql://neutron:unset@', {'Fn::Select': [ip_address, 'Fn::Select': [0, 'Fn::GetAtt': [ControlVirtualIP, fixed_ips]]]}, '/neutron']]} | 02:31 |
| lifeless | are entirely bogus / untested | 02:31 |
| * greghaynes verifies | 02:32 | |
| *** yfaykya has quit IRC | 02:32 | |
| lifeless | or is it working because mysql is bound to * ? | 02:32 |
| lifeless | indeed neutron_dsn: {Ref: NeutronDSN} | 02:33 |
| lifeless | is not used in tie at all | 02:33 |
| lifeless | much bong | 02:33 |
| greghaynes | mysql is infact bound to * | 02:33 |
| *** yfaykya has joined #tripleo | 02:33 | |
| lifeless | vinsh: your local elements must have a substantial delta vs tie | 02:33 |
| lifeless | vinsh: since we used: | 02:33 |
| lifeless | elements/neutron/os-apply-config/etc/neutron/plugins/ml2/ml2_conf.ini:connection = {{neutron.ovs_db}} | 02:34 |
| lifeless | elements/neutron/os-apply-config/etc/neutron/plugins/ml2/ml2_conf.ini~:connection = {{neutron.ovs_db}} | 02:34 |
| lifeless | elements/neutron/os-apply-config/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini:sql_connection = {{neutron.ovs_db}} | 02:34 |
| *** ramishra has quit IRC | 02:35 | |
| lifeless | ahh | 02:35 |
| lifeless | we go NeutronDSN -> neutron_dsn -> ovs_db | 02:35 |
| vinsh | hrmmmmm | 02:35 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-heat-templates: Use the right DSN for neutron agents. https://review.openstack.org/101080 | 02:36 |
| vinsh | so ya.. mysql is bound to * | 02:37 |
| vinsh | only thing having issue using it is ovs_db on controller nodes | 02:37 |
| vinsh | all other services going that route (to vip mysql) work. | 02:37 |
| vinsh | ya.. awful chain of dsn links there :) | 02:38 |
| *** ramishra has joined #tripleo | 02:38 | |
| vinsh | lol @101080 | 02:39 |
| greghaynes | cant argue with that | 02:39 |
| lifeless | fingers cossed it will be the issue | 02:40 |
| vinsh | where do you see this issue at? | 02:40 |
| lifeless | vinsh: all our CI job are failing for the last ~36 hours | 02:40 |
| vinsh | 0Oooo | 02:40 |
| lifeless | vinsh: image deployed in the OC can't be pinged. | 02:40 |
| lifeless | vinsh: and folk are having trouble reproducing locally | 02:41 |
| vinsh | well. whatever the case is.. that ovs thing is giving me hell. | 02:41 |
| vinsh | node can connect to it on local host just fine. | 02:41 |
| vinsh | try to use node ip/vip | 02:41 |
| vinsh | bzzzt http://paste.openstack.org/show/84439/ | 02:42 |
| greghaynes | vinsh: is 192.0.2.33 your controller VIP? | 02:44 |
| vinsh | ya | 02:44 |
| vinsh | So in https://review.openstack.org/#/c/101080/1/overcloud-source.yaml .. you have compute nodes using the vip to get ovs_db | 02:45 |
| lifeless | is your mysql bindigto 0000? | 02:45 |
| greghaynes | is controller0 up and done bootstrapping? | 02:45 |
| vinsh | but on line 466 control nodes use localhost for that? | 02:45 |
| vinsh | greghaynes, its not..it fails early on looping on that failure | 02:45 |
| lifeless | vinsh: oh I'm sure there are other issues, I just trying something to see if it unbreaks CI | 02:45 |
| lifeless | vinsh: this is why we need CI to have 3 nodes ctl plane :) | 02:46 |
| greghaynes | vinsh: I bet keepalived hasnt brought up the VIP yet | 02:46 |
| vinsh | oooo | 02:46 |
| vinsh | snap | 02:46 |
| vinsh | snap-sticks. | 02:46 |
| lifeless | vinsh: also note that the hypervisors may try to connect before the ctl plane is even deployed | 02:46 |
| lifeless | vinsh: we sequence the configs, but not the controller machine nodes | 02:46 |
| lifeless | vinsh: so errors like that are entirely normal, in that situation | 02:46 |
| vinsh | hm. hard to work around in this case. | 02:47 |
| vinsh | until I cluster mysql. | 02:47 |
| vinsh | got a set of 6 patches that do that from Therese that she gathered from upstream. | 02:48 |
| *** mestery_ is now known as mestery | 02:48 | |
| greghaynes | vinsh: try restarting keepalived service on controller0 | 02:48 |
| vinsh | at what point? | 02:48 |
| greghaynes | just by hand to see if that is the issue | 02:48 |
| vinsh | or .. ip addr show br-ex ? | 02:49 |
| vinsh | see if vip exits | 02:49 |
| greghaynes | looks like we restart keepalived in post-config | 02:49 |
| vinsh | *exists | 02:49 |
| greghaynes | sure | 02:49 |
| *** noslzzp has joined #tripleo | 02:49 | |
| greghaynes | so that should probably get moved up | 02:49 |
| greghaynes | into config.d/50- or somesuch | 02:49 |
| lifeless | we do most of our restarts post-config | 02:50 |
| lifeless | perhaps post-config 10 or something | 02:50 |
| vinsh | is it right to want to access mysql on vip though? | 02:50 |
| greghaynes | We need to hit mysql in config.d/51- | 02:50 |
| lifeless | vinsh: does the whole thing halt for you - it should try to reconfigure 30 seconds later | 02:50 |
| vinsh | or should it be done on localhost.. with mysql clustered? | 02:50 |
| greghaynes | vinsh: needs to be VIP | 02:51 |
| vinsh | lifeless, ya. it tries every 30 seconds and fails. | 02:51 |
| lifeless | greghaynes: arguably then the reconfigure should pause until the VIP is pingable | 02:51 |
| lifeless | greghaynes: or some timeout occurs | 02:51 |
| lifeless | e.g. wait_for 30 1 ping -c 1 $vip | 02:51 |
| lifeless | if it doesn't come up on *some* control node, we can't proceed | 02:51 |
| *** ramishra has quit IRC | 02:51 | |
| greghaynes | the problem is we need to restart keepalived after weve templated the config when metadata is up, and before we try to hit mysql | 02:51 |
| greghaynes | otherwise we have right config on disk but its not applied | 02:52 |
| greghaynes | so VIP isnt up | 02:52 |
| lifeless | greghaynes: wait why | 02:52 |
| greghaynes | restarting keepalived is what actually adds the VIP to the interface | 02:53 |
| lifeless | greghaynes: popping out for a second, but local db creation etc has to be local because of the cluster init races, no ? | 02:53 |
| lifeless | vinsh: is this error on the controller or the hypervisor? | 02:53 |
| lifeless | bbs | 02:53 |
| vinsh | controller, compute nodes do not hit this (probably cause they wait till vip is up on controller) | 02:54 |
| vinsh | got a stack re-launching to check on vip status during this failure loop. | 02:54 |
| *** ramishra_ has joined #tripleo | 02:56 | |
| *** untriaged-bot has joined #tripleo | 03:00 | |
| untriaged-bot | Untriaged bugs so far: | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1329238 | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331337 | 03:00 |
| uvirtbot | Launchpad bug 1329238 in tripleo "OVS isn't persisting mac addresses on OVS bridges" [Undecided,In progress] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331867 | 03:00 |
| uvirtbot | Launchpad bug 1331337 in tripleo "install_dependencies fails if python-software-properties is not installed." [Undecided,In progress] | 03:00 |
| *** untriaged-bot has quit IRC | 03:00 | |
| uvirtbot | Launchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New] | 03:00 |
| *** ramishra has joined #tripleo | 03:00 | |
| *** ramishra_ has quit IRC | 03:01 | |
| vinsh | greghaynes, so here's the story. keepalived can restart fine. | 03:12 |
| vinsh | BUT, br-ex isn't up yet on the controller node | 03:12 |
| vinsh | so, there really isn't a vip setup yet. | 03:13 |
| vinsh | Good find :) | 03:13 |
| vinsh | root@overcloud-controller0-gw4pdihxsskk:~# ip addr show br-ex | 03:14 |
| vinsh | Device "br-ex" does not exist, error in cloud, stop, drop and roll. | 03:14 |
| greghaynes | Error in cloud! | 03:14 |
| vinsh | Naturally. | 03:14 |
| *** rwsu has quit IRC | 03:27 | |
| *** CaptTofu_ has quit IRC | 03:28 | |
| *** CaptTofu_ has joined #tripleo | 03:29 | |
| *** CaptTofu_ has quit IRC | 03:33 | |
| *** ramishra has quit IRC | 03:33 | |
| *** blamar has joined #tripleo | 03:38 | |
| *** blamar has quit IRC | 03:38 | |
| *** matsuhashi has quit IRC | 03:38 | |
| *** matsuhashi has joined #tripleo | 03:39 | |
| *** akuznetsov has joined #tripleo | 03:43 | |
| *** matsuhashi has quit IRC | 03:45 | |
| *** matsuhashi has joined #tripleo | 03:45 | |
| *** matsuhashi has quit IRC | 03:46 | |
| *** victor_lowther has quit IRC | 04:02 | |
| *** nosnos has quit IRC | 04:02 | |
| *** victor_lowther has joined #tripleo | 04:05 | |
| *** mestery has quit IRC | 04:06 | |
| *** mestery has joined #tripleo | 04:07 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-image-elements: Reserve the keystone high port out of ephemeral https://review.openstack.org/100407 | 04:07 |
| morganfainberg | StevenK, hey, for ^ does sysctl-set-value override the current value? | 04:09 |
| vinsh | greghaynes, how would I turn this into syntax to get the ip of controller0 only instead of the vip? :http://paste.openstack.org/show/84446/ | 04:09 |
| vinsh | just can't get the heat syntax right here | 04:09 |
| morganfainberg | StevenK, or does it do magic to append etc, because if it just does a plain override, you'd need to do port1,port2 instead of setting twice | 04:09 |
| morganfainberg | StevenK, haven't had a chance to look at that underlying code though yet. | 04:10 |
| *** mestery has quit IRC | 04:11 | |
| StevenK | morganfainberg: Hmm | 04:12 |
| StevenK | morganfainberg: Let me pull up a seed with the change | 04:12 |
| morganfainberg | StevenK, ++ | 04:12 |
| morganfainberg | StevenK, gonna clock out and grab some sleep, let me know if you need anything. | 04:13 |
| StevenK | morganfainberg: Aye, thanks | 04:14 |
| greghaynes | vinsh: Why would you want to do that? | 04:15 |
| greghaynes | vinsh: also, I dont know off the top of my head | 04:16 |
| vinsh | to get around this blocker... I can't use localhost.. because that isolates each node until I cluster mysql | 04:16 |
| vinsh | I cant use the vip | 04:16 |
| vinsh | but all other nodes are using controller0 for mysql.. (it usually gets the vip).. | 04:16 |
| vinsh | so I figure.. "hey lets hardcode that.. and move along for a while" | 04:16 |
| greghaynes | So I think the only two sane options are using the vip or localhost | 04:17 |
| vinsh | :D | 04:17 |
| greghaynes | preferrably, we use the vip | 04:17 |
| greghaynes | I dont know what bring up that interface though... | 04:17 |
| greghaynes | I also dont know what lifeless was hinting at re: startup race issues... I hope we dont bring up a mysql master people can connect to until its actually clustered | 04:18 |
| vinsh | roger that | 04:19 |
| greghaynes | wait, have you tried using mysql? | 04:20 |
| greghaynes | er sorry, have you tried using localhost? | 04:20 |
| vinsh | yes.. and that works.. no error connecting. But then when the demo vm is launched.. | 04:20 |
| vinsh | neutron never seems right | 04:20 |
| vinsh | devtest_overcloud fails on the neutron port-list part after nova boot | 04:21 |
| vinsh | Probably because each node is talking to ovs_db on its own localhost? | 04:21 |
| vinsh | so thats no good. | 04:21 |
| greghaynes | huh. I really dont think we should be proceeding beyond config.d/51- unless mysql is started and joined to the cluster | 04:22 |
| vinsh | I guess what I am saying.. is localhost doesn't seem any good until that mysql is clustered. | 04:22 |
| greghaynes | yes, but we shouldnt be starting mysql any other way | 04:22 |
| vinsh | ok sure, but I don't think there is any clustering of mysql in this yet. | 04:22 |
| vinsh | thats wip right? | 04:22 |
| greghaynes | Are you not using https://review.openstack.org/#/c/83883/ ? | 04:23 |
| vinsh | not yet, in what I have | 04:23 |
| vinsh | time to pull that in I think. | 04:23 |
| greghaynes | ah, use that, and there are some recent merged things in t-i-e you need too | 04:24 |
| vinsh | Ok, I follow. Need clustered mysql before anything else here. I'll pull that in to my mix tomorrow. Therese has been working on that clustering part.. and we have all the bits needed for it to work.. just need to merge it into this controller scaling work. Will report back after that. :) | 04:26 |
| greghaynes | Yep, localhost should always work IMO but VIP is preferred so we prevent excess rollbacks | 04:27 |
| vinsh | Off for now, have a good night/day/etc! | 04:29 |
| greghaynes | gn | 04:29 |
| *** vinsh is now known as vinsh_zzz | 04:29 | |
| lifeless | greghaynes: so a node thats joined the cluster can be connected to before the IST completes, AIUI | 04:31 |
| lifeless | greghaynes: that was the race I was referring to | 04:32 |
| greghaynes | and it will accept connections? | 04:33 |
| lifeless | thats my understanding | 04:33 |
| tchaypo | I've een having a horribly day; | 04:33 |
| greghaynes | :/ | 04:33 |
| tchaypo | ba | 04:34 |
| tchaypo | I've been having one of those horrible days where I just can't manage to engage my brain or type properly | 04:34 |
| tchaypo | I'd given up on getting anything useful done today | 04:34 |
| lifeless | I've got diaahorea, fever, dehydration | 04:34 |
| lifeless | -> useless | 04:34 |
| clarkb | everyone has been sick | 04:34 |
| greghaynes | wow, srsly | 04:35 |
| tchaypo | then a friend asks about gettins started with tripleo, i look at the getting started docs, notice some ways they can be improved, suddenly I'm all cylenders go | 04:35 |
| tchaypo | except the spelling cylinder | 04:35 |
| tchaypo | Do we have any idea what's breaking CI yet? | 04:36 |
| tchaypo | maybe it's just caught the same virus as everyone else? | 04:37 |
| *** matsuhashi has joined #tripleo | 04:37 | |
| lifeless | NFI | 04:38 |
| lifeless | bnemec managed to reproduce, once. | 04:38 |
| tchaypo | s/reproduce/arrive at same symptoms, possibly (but unprovenly) via the same cause/ ? | 04:39 |
| *** nosnos has joined #tripleo | 04:43 | |
| greghaynes | lifeless: So I think the race condition depends on your SST method: http://galeracluster.com/documentation-webpages/nodeprovisioning.html#id4 | 04:43 |
| greghaynes | if we rsync (I think thats what we do?) its not an issue | 04:44 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-image-elements: Reserve the keystone high port out of ephemeral https://review.openstack.org/100407 | 04:46 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Fix --heat-env option handling for undercloud. https://review.openstack.org/96911 | 04:48 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Add -c support to devtest_seed.sh. https://review.openstack.org/96014 | 04:48 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Fix --build-only. https://review.openstack.org/96303 | 04:48 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Cleanup post controller parameter rename. https://review.openstack.org/96639 | 04:48 |
| StevenK | morganfainberg: Two calls leads to the element erroring because they conflict, so CI (if it was working) would have picked it up. I've pushed a fix | 04:50 |
| morganfainberg | StevenK, ++ [i swear i'm going to bed :P] | 04:51 |
| lifeless | StevenK: beep | 04:51 |
| StevenK | lifeless: Yeees? | 04:52 |
| lifeless | StevenK: check your review :) | 04:52 |
| StevenK | morganfainberg: Sure sure | 04:52 |
| *** akuznetsov has quit IRC | 04:53 | |
| StevenK | lifeless: Using the sysctl, elements can only say "Make it be this", and not "This should be in the values we set" | 04:54 |
| lifeless | I know | 04:54 |
| lifeless | note that I said 'reasonable place to fix...' | 04:55 |
| StevenK | Sure. | 04:55 |
| StevenK | I see a rabbit hole beneath me | 04:55 |
| *** matsuhashi has quit IRC | 04:56 | |
| lifeless | ferret maybe | 04:56 |
| *** matsuhashi has joined #tripleo | 04:57 | |
| clarkb | squirrel! | 04:57 |
| StevenK | Hmmm | 04:58 |
| *** akuznetsov has joined #tripleo | 04:58 | |
| StevenK | Can't work out why node["pm_password"] is null on CI | 04:59 |
| lifeless | StevenK: age? | 05:00 |
| lifeless | StevenK: we haven't redeployed in a bit | 05:00 |
| StevenK | Possibly | 05:01 |
| lifeless | the json dict is shown at the top of the run | 05:01 |
| tchaypo | https://wiki.openstack.org/w/index.php?title=TripleO&oldid=56304 <<- the most useful thing I've done all day | 05:01 |
| *** dshulyak_ has joined #tripleo | 05:13 | |
| StevenK | The sysctl element has two useless uses of cat | 05:14 |
| StevenK | :-( | 05:14 |
| *** CaptTofu_ has joined #tripleo | 05:17 | |
| tchaypo | hah | 05:18 |
| *** jtomasek has joined #tripleo | 05:21 | |
| *** CaptTofu_ has quit IRC | 05:21 | |
| *** tzumainn has quit IRC | 05:22 | |
| *** nati_ueno has joined #tripleo | 05:23 | |
| rushiagr | following os-collect-config readme gave me this error while installing it via pip | 05:30 |
| rushiagr | http://paste.openstack.org/show/84448/ | 05:30 |
| rushiagr | constitutes a (doc) bug? | 05:31 |
| *** rakesh_hs2 has joined #tripleo | 05:35 | |
| *** bmahalakshmi has joined #tripleo | 05:37 | |
| *** akuznetsov has quit IRC | 05:43 | |
| *** pcrews has quit IRC | 05:45 | |
| *** rbrady has quit IRC | 05:51 | |
| openstackgerrit | lokesh s proposed a change to openstack/tripleo-incubator: Make install-dependencies use sudo -E to enable use of env proxy. https://review.openstack.org/96383 | 05:55 |
| *** morganfainberg has quit IRC | 05:57 | |
| lifeless | yay + install-packages build-essential libz-dev libxslt-dev libxml2-dev python-dev | 05:58 |
| lifeless | failed | 05:58 |
| *** morganfainberg has joined #tripleo | 05:58 | |
| openstackgerrit | lokesh s proposed a change to openstack/tripleo-incubator: Make install-dependencies use sudo -E to enable use of env proxy. https://review.openstack.org/96383 | 05:58 |
| lifeless | rushiagr: it does yes, though if you had wheels of the xml bindings it probably wouldn't error | 05:59 |
| *** lazy_prince has joined #tripleo | 06:00 | |
| lifeless | python-dev : Depends: python (= 2.7.5-5ubuntu1) but 2.7.5-5ubuntu3 is to be installed | 06:00 |
| rushiagr | lifeless: okay. thanks | 06:00 |
| *** ProfFalk1n has joined #tripleo | 06:00 | |
| lifeless | sometimes metapackages give me the s**** | 06:01 |
| *** ProfFalk1n has quit IRC | 06:03 | |
| *** vigneshvar has joined #tripleo | 06:06 | |
| vigneshvar | HI, I am running an overcloud , i have seen that my stack still in progress and finally fails. | 06:07 |
| vigneshvar | I logged into the VM and checked the os-collect-config | 06:07 |
| vigneshvar | It says key db-password does not exist in ['/var/lib/os-collect-config/heat_local.json', '/var/lib/os-collect-config/ec2.json', '/var/lib/os-collect-config/cfn.json'] | 06:08 |
| vigneshvar | and iterates all the time. Can anyone help me | 06:08 |
| *** akuznetsov has joined #tripleo | 06:09 | |
| lifeless | can you pastebin heat resource-list overcloud ? | 06:09 |
| vigneshvar | sure. Just a min | 06:10 |
| *** mrunge has joined #tripleo | 06:11 | |
| vigneshvar | lifeless, http://pastebin.com/wtSN0SHB | 06:11 |
| lifeless | vigneshvar: so this | 06:12 |
| lifeless | | controller0Deployment | OS::Heat::StructuredDeployment | CREATE_IN_PROGRESS | 2014-04-18T19:21:32Z | | 06:12 |
| lifeless | I would have expected that to include db-password | 06:13 |
| vigneshvar | lifeless, yes it never completes | 06:13 |
| lifeless | vigneshvar: what does os-collect-config --print output (be sure to edit out passwords) | 06:13 |
| lifeless | vigneshvar: it will toggle to COMPLETE only once the controller pings-back | 06:14 |
| *** rakesh_hs2 has quit IRC | 06:15 | |
| *** rakesh_hs2 has joined #tripleo | 06:16 | |
| vigneshvar | lifeless, Give me a min, i will just pastebin the output | 06:19 |
| *** pcrews has joined #tripleo | 06:20 | |
| *** jtomasek has quit IRC | 06:21 | |
| *** andreaf has joined #tripleo | 06:22 | |
| openstackgerrit | xu-haiwei proposed a change to openstack/tripleo-incubator: Do not set NeutronControlPlaneID when building image only https://review.openstack.org/100828 | 06:23 |
| marios | yah so 'recheck on bug' doesn't work, fyi :) | 06:24 |
| marios | lifeless: are you going to tripleo mid-cycle - and which one (july/august?) | 06:25 |
| lifeless | marios: I hope so :) | 06:26 |
| marios | ah, just checked https://etherpad.openstack.org/p/juno-midcycle-meetup again but couldn't see you | 06:26 |
| marios | are you going to heat one? | 06:26 |
| lifeless | we're just sorting out paper work internally :( | 06:27 |
| lifeless | no | 06:27 |
| *** dshulyak_ has quit IRC | 06:29 | |
| *** dshulyak_ has joined #tripleo | 06:29 | |
| vigneshvar | lifeless, http://pastebin.com/h4Pe1nqp | 06:29 |
| vigneshvar | DEBUG heat.common.serializers [req-ae9c6a16-4e54-4d14-858f-c42c40e9c0de None demo] JSON response : {"software_deployment": {"status": "IN_PROGRESS", "server_id": "aa8ff79d-179a-41a2-8340-e6b9992e7f2d", "config_id": "2af2db82-9f77-4f1e-ac5e-3c97624ca45c", "output_values": null, "input_values": null, "action": "CREATE", "status_reason": "Deploy data available", "id": "7828c0fc-9391-4fdd-97da-5bf82730e07f"}} from (pid=19864) to_json /opt/stack/heat/heat/ | 06:30 |
| vigneshvar | common/serializers.py:43 | 06:30 |
| StevenK | This whole backup image thing is terrible | 06:30 |
| StevenK | -rw-r--r-- 1 steven steven 2.4G Jun 12 16:58 seed-2014.06.13-14.12.38.qcow2 | 06:30 |
| StevenK | -rw-r--r-- 1 steven steven 2.4G Jun 13 14:12 seed-2014.06.13-15.00.41.qcow2 | 06:30 |
| StevenK | -rw-r--r-- 1 steven steven 2.4G Jun 13 15:00 seed-2014.06.13-15.35.36.qcow2 | 06:30 |
| StevenK | ... | 06:30 |
| lifeless | vigneshvar: interesting - | 06:30 |
| lifeless | "db-password": "unset", is in there | 06:30 |
| lifeless | vigneshvar: you will need to reset your admin token and perhaps others since you've pasted them on the internet :) | 06:31 |
| lifeless | StevenK: huh when did we start doing that? | 06:31 |
| vigneshvar | lifeless, ya sure will do that | 06:31 |
| StevenK | lifeless: I'm not sure. | 06:31 |
| *** jprovazn has joined #tripleo | 06:32 | |
| StevenK | lifeless: I'm getting rather tempted to check, since 10 seed images just wiped out all the free space in /home. | 06:32 |
| marios | StevenK: ouch are you also sending those on the network or at least stored on seed building host? (or attached logical volume?). is that hourly? | 06:32 |
| StevenK | marios: I'm testing something, so rebuilding the seed | 06:32 |
| vigneshvar | lifeless: "db-password" is expected to be unset ? | 06:33 |
| lifeless | vigneshvar: unset is a (poor) default we have | 06:33 |
| lifeless | vigneshvar: its a value | 06:33 |
| StevenK | lifeless: b740c50c7625162a6ba4499a7dfbc7450deac3eb in dib | 06:33 |
| StevenK | Merged in fa9a561819197c136a3740cdd81c8631e78640d4 on the 12th | 06:34 |
| lifeless | wow | 06:35 |
| vigneshvar | lifeless: i can see that there is a key db-password, but why is not getting recognized/parsed | 06:35 |
| lifeless | vigneshvar: can you pastebin the end of os-collect-config.log? From the last time it started a poll | 06:39 |
| vigneshvar | lifeless, http://pastebin.com/QUjEmmvt | 06:40 |
| lifeless | vigneshvar: thats at least half an hour ago | 06:41 |
| lifeless | vigneshvar: it should be retrying every 30 seconds | 06:41 |
| lifeless | vigneshvar: so there is something very wrong :) | 06:43 |
| vigneshvar | lifeless, that server had a time set differently, i manually started os-collect-config and i get the same | 06:44 |
| lifeless | vigneshvar: time skew can break authentication | 06:44 |
| lifeless | vigneshvar: you should fix that before trying anything else | 06:44 |
| *** e0ne has joined #tripleo | 06:45 | |
| vigneshvar | lifeless, undercloud and overcloud are in sync | 06:46 |
| lifeless | hmm | 06:47 |
| lifeless | I guess we need to check you have a new enough os-collect-config | 06:48 |
| lifeless | run /opt/stack/venvs/os-collect-config/bin/pip freeze | 06:48 |
| lifeless | StevenK: you seeing: devtest.sh --trash-my-machine 2>&1 | tail -n 50 | pastebinit | 06:49 |
| lifeless | http://paste.ubuntu.com/7667546/ | 06:49 |
| lifeless | and now I figure it out | 06:49 |
| StevenK | Nope | 06:51 |
| StevenK | Not with my local mirror, at least | 06:51 |
| vigneshvar | lifeless, | 06:51 |
| vigneshvar | p freeze | 06:51 |
| vigneshvar | Babel==1.3 | 06:51 |
| vigneshvar | anyjson==0.3.3 | 06:51 |
| vigneshvar | eventlet==0.14.0 | 06:51 |
| vigneshvar | greenlet==0.4.2 | 06:51 |
| vigneshvar | iso8601==0.1.8 | 06:51 |
| vigneshvar | lxml==3.2.5 | 06:52 |
| vigneshvar | netaddr==0.7.10 | 06:52 |
| vigneshvar | os-collect-config==0.1.10 | 06:52 |
| vigneshvar | oslo.config==1.2.1 | 06:52 |
| lifeless | vigneshvar: uhm pastebint! | 06:52 |
| vigneshvar | pbr==0.5.23 | 06:52 |
| vigneshvar | prettytable==0.7.2 | 06:52 |
| vigneshvar | python-keystoneclient==0.4.2 | 06:52 |
| vigneshvar | pytz==2013.9 | 06:52 |
| vigneshvar | requests==2.2.1 | 06:52 |
| vigneshvar | six==1.5.2 | 06:52 |
| vigneshvar | wsgiref==0.1.2 | 06:52 |
| StevenK | vigneshvar: Pastebin! | 06:52 |
| lifeless | vigneshvar: ok so 0.1.10 is very very old | 06:52 |
| lifeless | vigneshvar: I think it predates sofware config in the heat templates | 06:52 |
| lifeless | vigneshvar: so you'll need a newer os-collect-config | 06:53 |
| *** dtantsur|afk is now known as dtantsur | 06:53 | |
| lifeless | yeah, 16158684a68f8e5c8d1b896f8bc12bbb0e9fc66d is what you need whichisn't in 0.1.10 | 06:53 |
| lifeless | and 831ab0be03f2b5635534321611cd0fb1b5a44127 | 06:53 |
| lifeless | 0.1.13 seems to be the minimum release you need | 06:54 |
| lifeless | vigneshvar: but I recommend 0.1.20 | 06:56 |
| lifeless | since we've been fixing bugs with interactions with heat etc | 06:56 |
| vigneshvar | lifeless, sure i will rebuild a new diskimage and update you | 06:57 |
| vigneshvar | i will rebuild with specified version | 06:58 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-image-elements: Do not use cat in sysctl-set-value https://review.openstack.org/101116 | 06:59 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-image-elements: Add a sysctl-append-value script https://review.openstack.org/101117 | 07:00 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-image-elements: Reserve the keystone high port out of ephemeral https://review.openstack.org/100407 | 07:00 |
| *** xuhaiwei has quit IRC | 07:03 | |
| *** andreaf has quit IRC | 07:03 | |
| jprovazn | can someone approve https://review.openstack.org/#/c/98709/ (enough +2 and tests passed) | 07:03 |
| greghaynes | jprovazn: tests passed a while ago :/ | 07:04 |
| lifeless | jprovazn: CI is wedged | 07:05 |
| *** CaptTofu_ has joined #tripleo | 07:05 | |
| *** lifeless changes topic to "CI DOWN, - can't ping instances | Using OpenStack to deploy OpenStack;meetings Tuesday 1900//0700 UTC in #openstack-meeting-alt" | 07:05 | |
| lifeless | jprovazn: we're all trying to reproduce it atm | 07:05 |
| lifeless | so we can fixenate it | 07:06 |
| *** e0ne has quit IRC | 07:06 | |
| jprovazn | greghaynes: well, there is no policy about tests freshness | 07:07 |
| greghaynes | hehe, true | 07:07 |
| marios | jprovazn: why don't you approve it? | 07:08 |
| *** shardy_afk is now known as shardy | 07:08 | |
| * marios looks at comments | 07:08 | |
| lifeless | jprovazn: actually there is one for openstack as a whole | 07:08 |
| marios | lifeless: i have just seen one, don't know if it the same one you're talkinga bout. am trying to investigate a bit more ( /#/c/95396/ Check for relevant environment variables ) | 07:08 |
| lifeless | marios: jprovazn: please don't land stuff other than fixing CI - last thing we need is more destabilisation | 07:08 |
| marios | lifeless: ah thanks | 07:09 |
| marios | lifeless: you might want to send a firedrill type thing to list for better coverage, if it's like that | 07:09 |
| lifeless | marios: I might tag you to do that, since I'm out for the day - I've got a virus or some such | 07:09 |
| marios | lifeless: basically there the overcloud can't ping the demo box it launches | 07:09 |
| lifeless | marios: yes | 07:09 |
| lifeless | there's a bug for it | 07:09 |
| marios | lifeless: awesome thanks, no? | 07:09 |
| marios | lifeless: (number) | 07:10 |
| lifeless | #1292105 | 07:10 |
| lifeless | I pushed up I0f27e95c63ee4de023a7c8248054f668fa7274cf as one attempt | 07:10 |
| *** CaptTofu_ has quit IRC | 07:10 | |
| lifeless | and I7ed40078f487459dee9055ef41f10a9b60a0e674 as another | 07:10 |
| *** jcoufal has joined #tripleo | 07:10 | |
| marios | lifeless: great thx | 07:10 |
| lifeless | anyhow, me -> out. to sleep perchance feel less ass | 07:10 |
| marios | lifeless: can you set chan topic "please don't land stuff other than fixing CI - last thing we need is more destabilisation | 07:12 |
| marios | lifeless: ah... :) irssi truncates it for me | 07:13 |
| marios | lifeless: have a good rest | 07:13 |
| *** lifeless changes topic to "CI DOWN, - can't ping instances; only land CI fixes | Using OpenStack to deploy OpenStack;meetings Tuesday 1900//0700 UTC in #openstack-meeting-alt" | 07:14 | |
| lifeless | marios: tada; cat food changed, now sleep. | 07:14 |
| *** ajc_ has joined #tripleo | 07:17 | |
| *** athomas has joined #tripleo | 07:21 | |
| *** marios_ has joined #tripleo | 07:24 | |
| *** hewbrocca has joined #tripleo | 07:26 | |
| *** jeremydei has quit IRC | 07:27 | |
| *** marios has quit IRC | 07:27 | |
| *** jeremydei has joined #tripleo | 07:28 | |
| *** pcrews has quit IRC | 07:37 | |
| *** yolanda has joined #tripleo | 07:44 | |
| yolanda | hi, having a problem here: https://review.openstack.org/#/c/99924/ | 07:44 |
| d0ugal | lsmola__: Unless you can provide a complete example of a full template model I'm not sure I know how to fix it based on those comments | 07:44 |
| yolanda | with undercloud-precise and overcloud-f28 jobs | 07:44 |
| d0ugal | lsmola__: I didn't fully understand it which was why I didn't write that bit ;) | 07:44 |
| yolanda | GheRivero, lifeless, can you help me ? | 07:45 |
| d0ugal | lsmola__: So, I'll try making a guess but I expect the spec will need to be updated when somebody can correct me. | 07:45 |
| lsmola__ | d0ugal: so the template definition should start at line 34 | 07:45 |
| lsmola__ | d0ugal: dependencies there should be just some string indexes (like filename or type) | 07:46 |
| lsmola__ | d0ugal: template should not point to other template, you are passing that relation to heat stack create | 07:46 |
| d0ugal | ok | 07:46 |
| lsmola__ | d0ugal: saying this string 'compute.yaml' stands for this json_dumped_object, files={'compute.yaml': json_dump} | 07:47 |
| d0ugal | lsmola__: Where does it point to another template? | 07:48 |
| lsmola__ | d0ugal: the spec? | 07:48 |
| d0ugal | lsmola__: Are you referring to how the dependency has been done? | 07:48 |
| lsmola__ | d0ugal: I am reffering to how tzumainn was defining it in the etherpad | 07:49 |
| d0ugal | Right | 07:49 |
| d0ugal | lsmola__: I can't see the template pointing to another template? | 07:49 |
| d0ugal | I might be reading it wrong, and I've not fully read your additions yet | 07:49 |
| lsmola__ | d0ugal: well is is poiting to some index, so it you have in your template get_file: compute.yaml | 07:50 |
| d0ugal | I think we should wait and have a chat with tzumainn later | 07:50 |
| lsmola__ | d0ugal: it's not pointing to a template compute.yaml, it's just index, and you will define the content in files dictionary passed to stack-create | 07:50 |
| lsmola__ | d0ugal: so you can then send files={'compute:yaml': "dumbed_compute_version2"} or files={'compute:yaml': "some other template"} | 07:52 |
| lsmola__ | d0ugal: I am just making sure you are not planning to relate Template object to other Template objects or Provider resource objects on this level | 07:53 |
| lsmola__ | d0ugal: I've added step to line 166, which should be handling all of these relations | 07:56 |
| d0ugal | lsmola__: Thanks, I'll try and take this into account | 07:57 |
| lsmola__ | d0ugal: ok | 07:57 |
| d0ugal | lsmola__: I'm a bit confused still TBH, but when I get back to this bit I'll ping you | 07:57 |
| d0ugal | lsmola__: It might be easier with tzumainn around too, as he knows exactly what he intended here :) | 07:57 |
| lsmola__ | d0ugal: ok | 07:57 |
| *** xuhaiwei has joined #tripleo | 07:59 | |
| *** gfidente has joined #tripleo | 08:00 | |
| *** nati_ueno has quit IRC | 08:01 | |
| *** e0ne has joined #tripleo | 08:05 | |
| *** markmc has joined #tripleo | 08:05 | |
| yolanda | GheRivero, lifeless, now only one of the jobs fail: http://logs.openstack.org/24/99924/6/check-tripleo/check-tripleo-overcloud-f20/a619b81 | 08:07 |
| *** e0ne has quit IRC | 08:13 | |
| *** e0ne has joined #tripleo | 08:23 | |
| *** jp_at_hp has joined #tripleo | 08:32 | |
| openstackgerrit | Om Kumar proposed a change to openstack/diskimage-builder: Remove unwanted kernel/initrd. https://review.openstack.org/101130 | 08:32 |
| *** pelix has joined #tripleo | 08:35 | |
| *** andreaf has joined #tripleo | 08:37 | |
| *** yolanda has left #tripleo | 08:40 | |
| *** yolanda has joined #tripleo | 08:42 | |
| *** nosnos has quit IRC | 08:44 | |
| *** lucasagomes has joined #tripleo | 08:45 | |
| *** nosnos has joined #tripleo | 08:46 | |
| *** rcarrill` has quit IRC | 08:50 | |
| *** CaptTofu_ has joined #tripleo | 08:53 | |
| *** rakesh_hs2 has quit IRC | 08:54 | |
| *** rakesh_hs2 has joined #tripleo | 08:58 | |
| *** CaptTofu_ has quit IRC | 08:58 | |
| *** untriaged-bot has joined #tripleo | 09:01 | |
| untriaged-bot | Untriaged bugs so far: | 09:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331931 | 09:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331867 | 09:01 |
| *** untriaged-bot has quit IRC | 09:01 | |
| uvirtbot | Launchpad bug 1331931 in tripleo "NeutronControlPlaneID should not be set if building image only" [Undecided,In progress] | 09:01 |
| uvirtbot | Launchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New] | 09:01 |
| *** rcarrillocruz has joined #tripleo | 09:01 | |
| rushiagr | Hi All! | 09:05 |
| rushiagr | Can I create ubuntu 'trusty' images using DIB? | 09:05 |
| rushiagr | I have a precise machine. | 09:05 |
| *** e0ne_ has joined #tripleo | 09:11 | |
| *** e0ne has quit IRC | 09:13 | |
| *** openstackgerrit has quit IRC | 09:14 | |
| *** martyntaylor has joined #tripleo | 09:17 | |
| *** jtomasek has joined #tripleo | 09:20 | |
| gfidente | guys, which version of ubuntu do we use for tripleo-ci ? | 09:28 |
| jprovazn | gfidente: "precise" I suppose - check jobs are names like this: "check-tripleo-undercloud-precise" | 09:35 |
| gfidente | jprovazn, ah yeah :( sorry about the stupid question | 09:35 |
| gfidente | thanks | 09:35 |
| jprovazn | gfidente: np | 09:36 |
| *** e0ne_ has quit IRC | 09:38 | |
| *** e0ne has joined #tripleo | 09:39 | |
| *** nosnos has quit IRC | 09:40 | |
| *** jprovazn has quit IRC | 09:42 | |
| *** e0ne has quit IRC | 09:43 | |
| *** matsuhashi has quit IRC | 09:44 | |
| *** matsuhashi has joined #tripleo | 09:44 | |
| *** matsuhashi has quit IRC | 09:46 | |
| *** matsuhashi has joined #tripleo | 09:47 | |
| *** IvanBerezovskiy1 has joined #tripleo | 09:49 | |
| *** IvanBerezovskiy1 has left #tripleo | 09:50 | |
| *** IvanBerezovskiy1 has joined #tripleo | 09:50 | |
| *** akuznetsov has quit IRC | 09:52 | |
| *** xuhaiwei has quit IRC | 10:03 | |
| *** jprovazn has joined #tripleo | 10:14 | |
| *** yamahata has quit IRC | 10:16 | |
| *** akrivoka has joined #tripleo | 10:17 | |
| *** CaptTofu_ has joined #tripleo | 10:36 | |
| shadower | StevenK: ping | 10:40 |
| shadower | StevenK: I've a few questions about os-cloud-config, looks like you may be the right person to ask | 10:41 |
| *** matsuhashi has quit IRC | 11:00 | |
| *** matsuhashi has joined #tripleo | 11:01 | |
| *** jtomasek has quit IRC | 11:01 | |
| *** e0ne has joined #tripleo | 11:05 | |
| *** matsuhashi has quit IRC | 11:05 | |
| *** dtantsur is now known as dtantsur|lunch | 11:11 | |
| *** jtomasek has joined #tripleo | 11:14 | |
| *** yamahata has joined #tripleo | 11:15 | |
| *** matsuhashi has joined #tripleo | 11:16 | |
| *** CaptTofu_ has quit IRC | 11:22 | |
| *** CaptTofu_ has joined #tripleo | 11:23 | |
| *** CaptTofu_ has quit IRC | 11:27 | |
| *** weshay has joined #tripleo | 11:28 | |
| *** julim has joined #tripleo | 11:36 | |
| *** akuznetsov has joined #tripleo | 11:39 | |
| *** ajc_ has quit IRC | 11:50 | |
| *** ajc_ has joined #tripleo | 11:50 | |
| StevenK | shadower: Sorry, I was off having dinner and such, what can I help you understand? | 11:50 |
| shadower | no worries | 11:52 |
| shadower | StevenK: so if IIUIC, os-cloud-config exists because we can't express some init calls within Heat templates, right? | 11:53 |
| shadower | the keystone role and user config API calls specificaly | 11:54 |
| *** bmahalakshmi has quit IRC | 11:54 | |
| *** ajc_ has quit IRC | 11:55 | |
| StevenK | shadower: Well, we need to do Keystone init and such like before we can invote heat, yes -- and we need to share code between tripleo-incubator (specifically, devtest) and tuskar | 11:55 |
| shadower | StevenK: I meant the undercloud Heat. I.e. if I want to deploy the overcloud, why do I need occ -- why can't I put that stuff in say the software config for the controller node in the overcloud heat template | 11:56 |
| shadower | StevenK: what I'd like to understand is -- is this a limitation of Heat that we should (eventually) try to add there? The stuff that occ does sounds like orchestration to me | 11:57 |
| StevenK | shadower: Right, we will use os-cloud-config for things shared between the seed and the undercloud. | 11:57 |
| StevenK | Registering nodes, that kind of thing. | 11:58 |
| shadower | StevenK: oh. So it's not going to be used for the overcloud? | 11:58 |
| StevenK | shadower: I don't believe it will be used for the overcloud, I think that's all Heat. | 11:58 |
| shadower | right okay | 11:58 |
| shadower | I do understand why we need to have something like this for the seed | 11:58 |
| *** rbrady has joined #tripleo | 12:11 | |
| *** matsuhashi has quit IRC | 12:13 | |
| *** matsuhashi has joined #tripleo | 12:14 | |
| StevenK | greghaynes: Thanks for the pointer to the other review -- I've commented and suggested he follow my work. | 12:16 |
| lsmola__ | stevebaker: I believe you are talking about functionality that even shouldn't be in the os-cloud-config from the beginning | 12:16 |
| lsmola__ | stevebaker: dam, sorry that was for StevenK | 12:16 |
| StevenK | lsmola__: Registering nodes? Initializing Keystone? | 12:17 |
| lsmola__ | StevenK: the main part that is shared everywhere is init-keystone, register-endpoints | 12:17 |
| lsmola__ | StevenK: possibly setup of neutron | 12:17 |
| StevenK | lsmola__: register-nodes is called for both the seed and the undercloud. | 12:18 |
| *** matsuhas_ has joined #tripleo | 12:18 | |
| *** matsuhashi has quit IRC | 12:18 | |
| lsmola__ | StevenK: yea sure, it can be there | 12:19 |
| lsmola__ | StevenK: thought for us the main part was https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest_overcloud.sh#L403 | 12:19 |
| lsmola__ | StevenK: to line 426 probably, we will see | 12:20 |
| lsmola__ | StevenK: though for UI we are currently not using the registering nodes from there, we just call the API's directly, for CLI it will be handy | 12:21 |
| StevenK | Right :-) | 12:22 |
| lsmola__ | StevenK: btw. it was named config, because it was doing these post config stuff, maybe we should rename it to something more general | 12:22 |
| lsmola__ | StevenK: registering_nodes doesn't fit much to config :-) | 12:23 |
| StevenK | lsmola__: Personally, I'm happy with the name os-cloud-config | 12:23 |
| lsmola__ | StevenK: yeah I guess, it's not worth month long naming conversation :-) | 12:24 |
| shadower | right, so occ will handle the init-keystone/ keystone role-create, setup end points, user-config and setup-neutron stuff, too, right? | 12:25 |
| shadower | which we need for the overcloud as well | 12:25 |
| *** CaptTofu_ has joined #tripleo | 12:29 | |
| *** CaptTofu_ has quit IRC | 12:33 | |
| lsmola__ | shadower: yes, all the config should be extracted to occ | 12:37 |
| lazy_prince | hi all.. is there any patch out for uboot support in dib..? i am working on a similar stuff and so need to know. | 12:39 |
| *** vigneshvar has quit IRC | 12:39 | |
| *** tzumainn has joined #tripleo | 12:40 | |
| *** andreaf has quit IRC | 12:46 | |
| *** morazi has joined #tripleo | 12:46 | |
| *** CaptTofu_ has joined #tripleo | 12:53 | |
| *** lucasagomes is now known as lucas-hungry | 12:55 | |
| *** jcoufal has quit IRC | 12:56 | |
| *** vigneshvar has joined #tripleo | 13:04 | |
| *** mestery has joined #tripleo | 13:09 | |
| *** mestery has quit IRC | 13:09 | |
| *** mestery has joined #tripleo | 13:10 | |
| *** dtantsur|lunch is now known as dtantsur | 13:12 | |
| *** dprince has joined #tripleo | 13:14 | |
| *** jcoufal has joined #tripleo | 13:23 | |
| *** andreaf has joined #tripleo | 13:24 | |
| *** lucas-hungry is now known as lucasagomes | 13:33 | |
| *** d0ugal has quit IRC | 13:39 | |
| *** d0ugal_ has joined #tripleo | 13:39 | |
| *** d0ugal_ is now known as d0ugal | 13:40 | |
| *** sballe has joined #tripleo | 13:47 | |
| *** davidlenwell has quit IRC | 13:51 | |
| *** anteaya has quit IRC | 13:51 | |
| *** anteaya has joined #tripleo | 13:52 | |
| *** davidlenwell has joined #tripleo | 13:53 | |
| *** IvanBerezovskiy1 has left #tripleo | 13:56 | |
| *** dguerri has joined #tripleo | 13:57 | |
| *** TravT has joined #tripleo | 13:59 | |
| *** bcrochet has quit IRC | 14:00 | |
| jang1 | BLOODY eventlet. | 14:03 |
| *** bcrochet has joined #tripleo | 14:03 | |
| jang1 | I'm beginning to think that rewriting ironic to be correctly threaded and to drop eventlet would be a faster way to a more robust approach. | 14:04 |
| *** e0ne has quit IRC | 14:10 | |
| rpodolyaka | heh | 14:14 |
| *** bnemec has joined #tripleo | 14:14 | |
| *** e0ne has joined #tripleo | 14:17 | |
| *** matsuhas_ has quit IRC | 14:17 | |
| *** dguerri is now known as _dguerri | 14:19 | |
| *** matsuhashi has joined #tripleo | 14:23 | |
| *** _dguerri is now known as dguerri | 14:24 | |
| *** jdob has joined #tripleo | 14:33 | |
| *** athomas_ has joined #tripleo | 14:33 | |
| *** rakesh_hs2 has quit IRC | 14:33 | |
| *** matsuhashi has quit IRC | 14:33 | |
| *** matsuhashi has joined #tripleo | 14:33 | |
| *** rwsu has joined #tripleo | 14:34 | |
| *** athomas_ has quit IRC | 14:36 | |
| *** cwolferh has joined #tripleo | 14:38 | |
| *** matsuhashi has quit IRC | 14:38 | |
| *** matsuhashi has joined #tripleo | 14:40 | |
| *** dguerri is now known as _dguerri | 14:41 | |
| *** _dguerri is now known as dguerri | 14:41 | |
| *** rcarrill` has joined #tripleo | 14:59 | |
| *** e0ne has quit IRC | 14:59 | |
| *** mrunge has quit IRC | 14:59 | |
| *** e0ne has joined #tripleo | 15:00 | |
| *** untriaged-bot has joined #tripleo | 15:00 | |
| untriaged-bot | Untriaged bugs so far: | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1332021 | 15:00 |
| uvirtbot | Launchpad bug 1332021 in tripleo "check-tripleo-overcloud-precise job fails when collecting host state" [Undecided,New] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331931 | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331867 | 15:00 |
| *** untriaged-bot has quit IRC | 15:00 | |
| uvirtbot | Launchpad bug 1331931 in tripleo "NeutronControlPlaneID should not be set if building image only" [Undecided,In progress] | 15:00 |
| uvirtbot | Launchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New] | 15:00 |
| *** rcarrillocruz has quit IRC | 15:02 | |
| *** ddieterly has joined #tripleo | 15:04 | |
| *** e0ne has quit IRC | 15:04 | |
| *** jprovazn is now known as jprovazn_afk | 15:05 | |
| *** rcarrill` has quit IRC | 15:05 | |
| *** rcarrillocruz has joined #tripleo | 15:07 | |
| *** matsuhashi has quit IRC | 15:07 | |
| *** casanch1 has joined #tripleo | 15:07 | |
| *** matsuhashi has joined #tripleo | 15:07 | |
| jdob | lifeless mentioned in the tripleo meeting "since dib had a regression you might want to see if the regression was released or not; if it wasn't, we should fix it before releasing", but i'm having trouble tracking down what regression he's talking about. anyone know? | 15:11 |
| *** matsuhashi has quit IRC | 15:11 | |
| bnemec | jdob: I _think_ he was referring to https://review.openstack.org/#/c/99505/ | 15:12 |
| *** blamar has joined #tripleo | 15:12 | |
| *** rcarrill` has joined #tripleo | 15:13 | |
| jdob | do you know if the f20 gate is happy again? I'll kick off a recheck on it and try to get that in before I release | 15:13 |
| dprince | jdob: it isn't happy yet. | 15:15 |
| dprince | jdob: also, it isn't Fedora specific | 15:15 |
| *** rcarrillocruz has quit IRC | 15:15 | |
| jdob | it meaning what, the reason it's not happy or that bug? | 15:16 |
| dprince | jdob: we ran check experimental and the ubuntu overcloud job failed in the same way | 15:16 |
| dprince | jdob: the issue is the ping to the overcloud instance (the use image) fails | 15:16 |
| jdob | ah, ok. so i'm new to the release process and looking for advice. should I wait and hope this patch lands before late tomorrow or just release what we have and it'll get into next week's? | 15:18 |
| *** athomas_ has joined #tripleo | 15:18 | |
| *** athomas_ has quit IRC | 15:18 | |
| *** bcrochet has quit IRC | 15:18 | |
| *** athomas has quit IRC | 15:19 | |
| dprince | jdob: not sure, I've reviewed most of our commits and none of them look suspicious. "our" meaning the TripleO specific stuff | 15:19 |
| dprince | jdob: this could either be an environmental thing... or some add neutron/nova change is causing the issue | 15:20 |
| dprince | odd | 15:20 |
| *** jcoufal has quit IRC | 15:21 | |
| gfidente | dprince, I was looking into https://bugs.launchpad.net/tripleo/+bug/1226310 today | 15:23 |
| uvirtbot | Launchpad bug 1226310 in tripleo "Nova bm operations fail when LIBVIRT_DEFAULT_URI not set" [Medium,Triaged] | 15:23 |
| *** bcrochet has joined #tripleo | 15:23 | |
| gfidente | can you help me figure where is that we set the default uri for the CI jobs? it definitely have to be set somewhere | 15:23 |
| gfidente | *libvirt default uri | 15:23 |
| dprince | gfidente: devtest_variables.sh does it | 15:25 |
| *** bcrochet has quit IRC | 15:25 | |
| gfidente | dprince, yeah but that only applies to the scripts launched by devtest | 15:25 |
| *** jtomasek has quit IRC | 15:26 | |
| dprince | gfidente: we call devtest_variables.sh directly in toci_devtest.sh (in the tripleo-ci project) | 15:26 |
| gfidente | the problem we're seeing is that the virtual power driver, when connects to the 'hosting' machine to poweron the baremetal node, doesn't connect to qemu:///system | 15:26 |
| *** TravT has quit IRC | 15:26 | |
| dprince | gfidente: SO I think this is outside of the scope of TripleO. I mean we don't own the seed host entirely... | 15:27 |
| *** bcrochet has joined #tripleo | 15:27 | |
| dprince | gfidente: you'd need to set that in the .bashrc of the host machine, or source it everytime... | 15:27 |
| dprince | gfidente: or perhaps just configure it correctly in the libvirt.conf file (in /etc) | 15:28 |
| dprince | gfidente: lots of ways to do it. perhaps one of those could got into tripleo-incubator as part of the seed host setup | 15:28 |
| gfidente | dprince, indeed | 15:28 |
| *** athomas has joined #tripleo | 15:28 | |
| gfidente | I was trying to figure how is that accomplished for the CI jobs | 15:29 |
| *** TravT has joined #tripleo | 15:35 | |
| *** ramishra has joined #tripleo | 15:38 | |
| *** jml has quit IRC | 15:39 | |
| *** jml has joined #tripleo | 15:40 | |
| *** ramishra_ has joined #tripleo | 15:41 | |
| dprince | gfidente: ah, I see. So I'm not sure exactly why it is working there then. For CI we do setup our host machines a bit differently | 15:42 |
| *** ramishra has quit IRC | 15:43 | |
| gfidente | dprince, heh I'm still learning about it ... but I'd expect the undercloud would at some point poweron the overcloud VMs and that should take the same route | 15:44 |
| *** vigneshvar has quit IRC | 15:44 | |
| gfidente | virtual power driver in the undercloud is going to ssh to the system hosting the test environment and powering up a VM | 15:44 |
| dprince | gfidente: will see if I can trace this and figure it out in a bit. Right now I'm debugging something else | 15:44 |
| gfidente | sure, understand that | 15:45 |
| gfidente | sorry :) | 15:45 |
| *** eghobo has joined #tripleo | 15:52 | |
| *** eghobo has quit IRC | 15:52 | |
| *** eghobo has joined #tripleo | 15:53 | |
| *** eghobo has quit IRC | 15:53 | |
| *** eghobo has joined #tripleo | 15:54 | |
| *** e0ne has joined #tripleo | 15:57 | |
| *** lazy_prince has quit IRC | 16:00 | |
| *** e0ne has quit IRC | 16:01 | |
| *** e0ne has joined #tripleo | 16:03 | |
| *** jcoufal has joined #tripleo | 16:06 | |
| *** blamar has quit IRC | 16:18 | |
| *** martyntaylor has left #tripleo | 16:23 | |
| *** nati_ueno has joined #tripleo | 16:23 | |
| *** regebro|afk is now known as regebro | 16:30 | |
| *** akrivoka has quit IRC | 16:35 | |
| *** jcoufal has quit IRC | 16:41 | |
| *** jcoufal has joined #tripleo | 16:42 | |
| *** e0ne has quit IRC | 16:43 | |
| *** e0ne has joined #tripleo | 16:43 | |
| andreaf | hi - any core reviewer with a sec for a mini-patch https://review.openstack.org/#/c/98817/ with a +2 already? | 16:44 |
| *** e0ne has quit IRC | 16:48 | |
| *** dkehn_ has joined #tripleo | 16:52 | |
| *** dkehn_ is now known as dkehnx2 | 16:53 | |
| *** dkehnx2 is now known as dkehnc | 16:53 | |
| *** dkehn has quit IRC | 16:54 | |
| *** dkehnc has quit IRC | 16:54 | |
| *** dkehnc has joined #tripleo | 16:55 | |
| *** dkehnx has quit IRC | 16:55 | |
| *** dkehnc is now known as dkehnx | 16:56 | |
| *** dkehn has joined #tripleo | 16:56 | |
| *** nati_ueno has quit IRC | 16:59 | |
| *** vinsh_zzz is now known as vinsh | 17:00 | |
| *** vinsh has quit IRC | 17:00 | |
| *** vinsh has joined #tripleo | 17:00 | |
| *** nati_ueno has joined #tripleo | 17:02 | |
| *** pcrews has joined #tripleo | 17:05 | |
| *** jang1 has quit IRC | 17:07 | |
| *** dkehn_ has joined #tripleo | 17:08 | |
| *** lucasagomes has quit IRC | 17:09 | |
| andrearosa | SpamapS: if you are around and you have time, please can you see my reply to your comment on this bug https://bugs.launchpad.net/tripleo/+bug/1324598 | 17:10 |
| uvirtbot | Launchpad bug 1324598 in tripleo "Passthrough-enable for nova.conf mustache template is not correct" [High,In progress] | 17:10 |
| *** dkehnx has quit IRC | 17:11 | |
| *** dkehn has quit IRC | 17:11 | |
| *** dkehn has joined #tripleo | 17:13 | |
| *** blamar has joined #tripleo | 17:15 | |
| *** dguerri has quit IRC | 17:19 | |
| SpamapS | andrearosa: here now. looking. | 17:21 |
| andrearosa | cool, thanks | 17:22 |
| *** athomas has quit IRC | 17:22 | |
| SpamapS | andrearosa: responded, and +2'd | 17:23 |
| andrearosa | SpamapS: thanks for your time. | 17:24 |
| *** proffalken has quit IRC | 17:24 | |
| *** hewbrocca has quit IRC | 17:26 | |
| mordred | SpamapS: hey - are we expecting f20 jobs to be successful in tripleo ci? | 17:27 |
| mordred | (I know we have issues with f20 nodepool, but I haven't really been fully following) | 17:28 |
| mordred | SpamapS: yolanda has been trying to get https://review.openstack.org/#/c/99924/ moving, but I think we're not sure if the errors it's seeing are from her, or are known things or what | 17:28 |
| *** e0ne has joined #tripleo | 17:29 | |
| yolanda | modred, that kind of errors shouldn't come from my change | 17:29 |
| vinsh | /tmp/in_target.d/pre-install.d/05-mysql: line 40: add-apt-repository: command not found | 17:31 |
| vinsh | where is that found usually? | 17:31 |
| bnemec | mordred: No, our overcloud jobs are broken right now. :-( | 17:32 |
| bnemec | Networking issues to the user instance in the overcloud. | 17:32 |
| SpamapS | mordred: yes we are expecting them to work | 17:33 |
| SpamapS | mordred: but I think dprince was working on an issue w/ pinging the end-user booted instance | 17:33 |
| *** ramishra_ has quit IRC | 17:34 | |
| SpamapS | oh add-apt-repository is running on fedora?! | 17:34 |
| vinsh | that^ where is that pulled in from? that tool? | 17:34 |
| * SpamapS looks | 17:34 | |
| SpamapS | oh wait no | 17:34 |
| SpamapS | that is separate issue | 17:34 |
| SpamapS | vinsh: that is in python-software-properties | 17:34 |
| vinsh | hm. not sure where that comes from in this stack. | 17:35 |
| SpamapS | vinsh: it is included by default on Ubuntu cloud images. | 17:35 |
| vinsh | oh.. | 17:35 |
| SpamapS | so probably being assumed for all dpkg based systems | 17:35 |
| vinsh | grr.. wonder if it is included in hlinux. | 17:35 |
| * vinsh goes to see. | 17:35 | |
| dprince | mordred: the ping failures we are currently seeing for the overcloud jobs seem to affect all distros. We ran a 'check experimental' yesterday and saw the same failure w/ Ubuntu as well | 17:35 |
| *** yamahata has quit IRC | 17:36 | |
| *** eghobo has quit IRC | 17:37 | |
| *** eguz has joined #tripleo | 17:37 | |
| mordred | dprince: awesome. well, good to know | 17:37 |
| *** Shrews has quit IRC | 17:37 | |
| *** jcoufal has quit IRC | 17:38 | |
| *** pelix has quit IRC | 17:41 | |
| SpamapS | dprince: need any help debugging/testing on that? | 17:46 |
| *** morazi has quit IRC | 17:46 | |
| dprince | SpamapS: maybe. I'm currently using a vnc console to actually attach to the testenv | 17:47 |
| SpamapS | dprince: nasty | 17:47 |
| SpamapS | dprince: is it not reproducible on a local devtest run? | 17:47 |
| dprince | SpamapS: yeah, well noone can reproduce locally | 17:47 |
| * bnemec did | 17:47 | |
| bnemec | But I don't know enough about multi-node Neutron to figure out what's wrong. | 17:47 |
| dprince | bnemec: oh, so you hit it locally? | 17:47 |
| bnemec | dprince: Yeah, I randomly hit it on a devtest run yesterday. | 17:48 |
| dprince | bnemec: seriously, tell somebody! | 17:48 |
| bnemec | I'm pretty sure I mentioned it multiple times right here. ;-) | 17:48 |
| dprince | bnemec: yeah, I've seen a few CI passes in the past 24 hours as well (2) | 17:48 |
| dprince | bnemec: I missed it. | 17:48 |
| bnemec | Yeah, lot of discussion going on in the channel lately. | 17:49 |
| *** eguz has quit IRC | 17:49 | |
| dprince | bnemec: can you add details here https://bugs.launchpad.net/tripleo/+bug/1292105 | 17:50 |
| uvirtbot | Launchpad bug 1292105 in tripleo "CI failed pinging overcloud instance" [Critical,In progress] | 17:50 |
| *** eghobo has joined #tripleo | 17:50 | |
| bnemec | I'm working on setting up a parallel devtest system to compare with (assuming that one doesn't hit the bug as well). | 17:50 |
| *** Shrews has joined #tripleo | 17:50 | |
| *** proffalken has joined #tripleo | 17:51 | |
| *** e0ne has quit IRC | 17:52 | |
| dprince | bnemec: so have a look at the iptables rules on the controller | 17:52 |
| dprince | bnemec: you'll have to use ip netns exec <namespace> iptables-save | 17:53 |
| *** e0ne has joined #tripleo | 17:53 | |
| dprince | bnemec: to get at them | 17:53 |
| bnemec | dprince: qrouter or qdhcp? | 17:53 |
| dprince | bnemec: qrouter | 17:53 |
| bnemec | dprince: http://paste.openstack.org/show/84516/ | 17:54 |
| *** blamar has quit IRC | 17:57 | |
| *** e0ne has quit IRC | 17:57 | |
| *** dkehn_ is now known as dkehnx | 17:58 | |
| *** andreaf_ has joined #tripleo | 18:05 | |
| *** bcrochet has quit IRC | 18:08 | |
| *** sballe has quit IRC | 18:08 | |
| *** tzumainn has quit IRC | 18:08 | |
| *** jp_at_hp has quit IRC | 18:08 | |
| *** victor_lowther has quit IRC | 18:08 | |
| *** noslzzp has quit IRC | 18:08 | |
| *** regebro has quit IRC | 18:08 | |
| *** yolanda has quit IRC | 18:08 | |
| *** dshulyak has quit IRC | 18:08 | |
| *** ccrouch1 has quit IRC | 18:08 | |
| *** fabbione has quit IRC | 18:08 | |
| *** ohadlevy has quit IRC | 18:08 | |
| *** andreaf has quit IRC | 18:08 | |
| *** dprince has quit IRC | 18:08 | |
| *** julim has quit IRC | 18:08 | |
| *** mkerrin has quit IRC | 18:08 | |
| *** jcooley has quit IRC | 18:08 | |
| *** cody-somerville has quit IRC | 18:08 | |
| *** bkero has quit IRC | 18:08 | |
| *** rushiagr has quit IRC | 18:08 | |
| *** russell_h has quit IRC | 18:08 | |
| *** eghobo has quit IRC | 18:08 | |
| *** vinsh has quit IRC | 18:08 | |
| *** cwolferh has quit IRC | 18:08 | |
| *** wendar has quit IRC | 18:08 | |
| *** agordeev has quit IRC | 18:08 | |
| *** yjiang5 has quit IRC | 18:08 | |
| *** adam_g has quit IRC | 18:08 | |
| *** rainya has quit IRC | 18:08 | |
| *** stevebaker has quit IRC | 18:08 | |
| *** edunham has quit IRC | 18:08 | |
| *** zigo has quit IRC | 18:08 | |
| *** chmouel has quit IRC | 18:08 | |
| *** antonym has quit IRC | 18:08 | |
| *** clarkb has quit IRC | 18:08 | |
| *** uvirtbot has quit IRC | 18:08 | |
| *** eghobo has joined #tripleo | 18:10 | |
| *** vinsh has joined #tripleo | 18:10 | |
| *** bcrochet has joined #tripleo | 18:10 | |
| *** cwolferh has joined #tripleo | 18:10 | |
| *** sballe has joined #tripleo | 18:10 | |
| *** dprince has joined #tripleo | 18:10 | |
| *** tzumainn has joined #tripleo | 18:10 | |
| *** julim has joined #tripleo | 18:10 | |
| *** yolanda has joined #tripleo | 18:10 | |
| *** jp_at_hp has joined #tripleo | 18:10 | |
| *** victor_lowther has joined #tripleo | 18:10 | |
| *** noslzzp has joined #tripleo | 18:10 | |
| *** edunham has joined #tripleo | 18:10 | |
| *** regebro has joined #tripleo | 18:10 | |
| *** dshulyak has joined #tripleo | 18:10 | |
| *** mkerrin has joined #tripleo | 18:10 | |
| *** ccrouch1 has joined #tripleo | 18:10 | |
| *** fabbione has joined #tripleo | 18:10 | |
| *** wendar has joined #tripleo | 18:10 | |
| *** ohadlevy has joined #tripleo | 18:10 | |
| *** jcooley has joined #tripleo | 18:10 | |
| *** agordeev has joined #tripleo | 18:10 | |
| *** cody-somerville has joined #tripleo | 18:10 | |
| *** yjiang5 has joined #tripleo | 18:10 | |
| *** bkero has joined #tripleo | 18:10 | |
| *** rushiagr has joined #tripleo | 18:10 | |
| *** adam_g has joined #tripleo | 18:10 | |
| *** rainya has joined #tripleo | 18:10 | |
| *** russell_h has joined #tripleo | 18:10 | |
| *** uvirtbot has joined #tripleo | 18:10 | |
| *** clarkb has joined #tripleo | 18:10 | |
| *** antonym has joined #tripleo | 18:10 | |
| *** chmouel has joined #tripleo | 18:10 | |
| *** zigo has joined #tripleo | 18:10 | |
| *** stevebaker has joined #tripleo | 18:10 | |
| *** dguerri has joined #tripleo | 18:10 | |
| *** jp_at_hp has quit IRC | 18:11 | |
| *** jp_at_hp has joined #tripleo | 18:11 | |
| *** thegodfather has joined #tripleo | 18:14 | |
| *** bcrochet has quit IRC | 18:14 | |
| *** sballe has quit IRC | 18:14 | |
| *** tzumainn has quit IRC | 18:14 | |
| *** jp_at_hp has quit IRC | 18:14 | |
| *** victor_lowther has quit IRC | 18:14 | |
| *** noslzzp has quit IRC | 18:14 | |
| *** regebro has quit IRC | 18:14 | |
| *** yolanda has quit IRC | 18:14 | |
| *** dshulyak has quit IRC | 18:14 | |
| *** ccrouch1 has quit IRC | 18:14 | |
| *** fabbione has quit IRC | 18:14 | |
| *** ohadlevy has quit IRC | 18:14 | |
| *** dprince has quit IRC | 18:14 | |
| *** julim has quit IRC | 18:14 | |
| *** mkerrin has quit IRC | 18:14 | |
| *** jcooley has quit IRC | 18:14 | |
| *** cody-somerville has quit IRC | 18:14 | |
| *** bkero has quit IRC | 18:14 | |
| *** rushiagr has quit IRC | 18:14 | |
| *** russell_h has quit IRC | 18:14 | |
| *** eghobo has quit IRC | 18:14 | |
| *** vinsh has quit IRC | 18:14 | |
| *** cwolferh has quit IRC | 18:14 | |
| *** wendar has quit IRC | 18:14 | |
| *** agordeev has quit IRC | 18:14 | |
| *** yjiang5 has quit IRC | 18:14 | |
| *** adam_g has quit IRC | 18:14 | |
| *** rainya has quit IRC | 18:14 | |
| *** stevebaker has quit IRC | 18:14 | |
| *** edunham has quit IRC | 18:14 | |
| *** zigo has quit IRC | 18:14 | |
| *** chmouel has quit IRC | 18:14 | |
| *** antonym has quit IRC | 18:14 | |
| *** clarkb has quit IRC | 18:14 | |
| *** uvirtbot has quit IRC | 18:14 | |
| *** eghobo has joined #tripleo | 18:16 | |
| *** vinsh has joined #tripleo | 18:16 | |
| *** bcrochet has joined #tripleo | 18:16 | |
| *** cwolferh has joined #tripleo | 18:16 | |
| *** sballe has joined #tripleo | 18:16 | |
| *** dprince has joined #tripleo | 18:16 | |
| *** tzumainn has joined #tripleo | 18:16 | |
| *** julim has joined #tripleo | 18:16 | |
| *** yolanda has joined #tripleo | 18:16 | |
| *** jp_at_hp has joined #tripleo | 18:16 | |
| *** victor_lowther has joined #tripleo | 18:16 | |
| *** noslzzp has joined #tripleo | 18:16 | |
| *** edunham has joined #tripleo | 18:16 | |
| *** regebro has joined #tripleo | 18:16 | |
| *** dshulyak has joined #tripleo | 18:16 | |
| *** mkerrin has joined #tripleo | 18:16 | |
| *** ccrouch1 has joined #tripleo | 18:16 | |
| *** fabbione has joined #tripleo | 18:16 | |
| *** wendar has joined #tripleo | 18:16 | |
| *** ohadlevy has joined #tripleo | 18:16 | |
| *** jcooley has joined #tripleo | 18:16 | |
| *** agordeev has joined #tripleo | 18:16 | |
| *** cody-somerville has joined #tripleo | 18:16 | |
| *** yjiang5 has joined #tripleo | 18:16 | |
| *** bkero has joined #tripleo | 18:16 | |
| *** rushiagr has joined #tripleo | 18:16 | |
| *** adam_g has joined #tripleo | 18:16 | |
| *** rainya has joined #tripleo | 18:16 | |
| *** russell_h has joined #tripleo | 18:16 | |
| *** uvirtbot has joined #tripleo | 18:16 | |
| *** clarkb has joined #tripleo | 18:16 | |
| *** antonym has joined #tripleo | 18:16 | |
| *** chmouel has joined #tripleo | 18:16 | |
| *** zigo has joined #tripleo | 18:16 | |
| *** stevebaker has joined #tripleo | 18:16 | |
| dprince | bnemec: at least that is what I'm seeing | 18:16 |
| dprince | bnemec: On your nova compute node go have a look in /mnt/state/var/lib/nova/instances/<uuid> | 18:16 |
| dprince | bnemec: you'll have to go to the compute where the instance is running (virsh list...) | 18:16 |
| dprince | bnemec: on mine the instance just plain doesn't get its dhcp address | 18:17 |
| *** fabbione has quit IRC | 18:17 | |
| bnemec | dprince: Yeah, that's what I'm seeing too. I left a note on the bug. | 18:17 |
| bnemec | dprince: (also, I think I missed part of that due to the netsplit. The first line I saw was "at least that is what I'm seeing") | 18:18 |
| *** bmahalakshmi has joined #tripleo | 18:18 | |
| dprince | bnemec: thanks | 18:18 |
| dprince | bnemec: well at least we are seeing the same thing | 18:18 |
| bnemec | I just kicked off my second devtest run. I'm going to go grab lunch and hopefully I'll have something to look at when I'm done. | 18:22 |
| *** dkehn_ has joined #tripleo | 18:25 | |
| *** dkehn_ has quit IRC | 18:27 | |
| *** dkehn has quit IRC | 18:28 | |
| *** dkehnx has quit IRC | 18:28 | |
| *** dkehn_ has joined #tripleo | 18:28 | |
| *** dkehn has joined #tripleo | 18:29 | |
| *** yolanda has quit IRC | 18:33 | |
| *** tzumainn has quit IRC | 18:33 | |
| *** regebro has quit IRC | 18:33 | |
| *** yolanda has joined #tripleo | 18:33 | |
| *** sballe has quit IRC | 18:33 | |
| *** tzumainn has joined #tripleo | 18:33 | |
| *** regebro has joined #tripleo | 18:33 | |
| *** jang has joined #tripleo | 18:33 | |
| SpamapS | dprince: FYI, 'nova show' will tell admins which compute host has an instance. | 18:39 |
| dprince | SpamapS: if you don't have creds then virsh works just fine too ;) | 18:41 |
| *** dtantsur is now known as dtantsur|afk | 18:48 | |
| d0ugal | tzumainn: Would you object if I removed the Provider Resource model? | 18:54 |
| tzumainn | d0ugal, nope, go for it | 18:55 |
| d0ugal | tzumainn: ok, we can maybe add it later, but I'm trying to justify it in the spec, but I can't :) | 18:56 |
| tzumainn | yep, fair enough! | 18:56 |
| d0ugal | tzumainn: cool, thanks | 18:58 |
| SpamapS | dprince: yeah thats fine for when you're testing 1 or 2 compute nodes :) | 19:01 |
| *** dshulyak_ has joined #tripleo | 19:08 | |
| *** nati_uen_ has joined #tripleo | 19:10 | |
| *** nati_ueno has quit IRC | 19:14 | |
| NobodyCam | I have a heat question. Will heat read tempate valuse from enviroment vars? I.E. export UNDERCLOUD_NeutronPublicInterface=blah | 19:16 |
| NobodyCam | values even | 19:16 |
| *** dkehn_ has quit IRC | 19:24 | |
| SpamapS | NobodyCam: Heat? | 19:24 |
| SpamapS | NobodyCam: heat only knows -P and --environment. It ignores the "environment variables" in your shell. | 19:25 |
| NobodyCam | thats what I thought | 19:25 |
| NobodyCam | SpamapS: could you jump in hipchat - ironic channel | 19:26 |
| *** dkehn has quit IRC | 19:28 | |
| *** e0ne has joined #tripleo | 19:28 | |
| *** casanch1_ has joined #tripleo | 19:31 | |
| *** casanch1 has quit IRC | 19:34 | |
| *** dkehn has joined #tripleo | 19:35 | |
| *** dkehn_ has joined #tripleo | 19:35 | |
| *** casanch1_ has quit IRC | 19:35 | |
| *** bmahalakshmi has quit IRC | 19:35 | |
| *** e0ne has quit IRC | 19:35 | |
| *** e0ne has joined #tripleo | 19:36 | |
| *** dkehn_ has quit IRC | 19:36 | |
| *** dkehn_ has joined #tripleo | 19:37 | |
| *** dkehn_ is now known as dkehnx | 19:37 | |
| *** e0ne has quit IRC | 19:38 | |
| *** e0ne has joined #tripleo | 19:41 | |
| *** e0ne has quit IRC | 19:49 | |
| *** e0ne has joined #tripleo | 19:49 | |
| *** akuznetsov has quit IRC | 19:49 | |
| *** e0ne has quit IRC | 19:52 | |
| *** jang1 has joined #tripleo | 19:53 | |
| *** akuznetsov has joined #tripleo | 19:55 | |
| vinsh | hmmmr "/opt/stack/os-config-refresh/configure.d/51-mysql-init: line 13: /usr/local/mysql/scripts/mysql_install_db: No such file or directory" | 19:55 |
| lifeless | o/ | 19:56 |
| vinsh | Should be.. if 10-mysql actually ran during DIB. | 19:56 |
| * vinsh curls up in a ball and screams. | 19:56 | |
| lifeless | SpamapS: what do you think of the idea of heat being allowed to ignore excess parameters rather than failing ? | 19:57 |
| lifeless | vinsh: chmod ? | 19:57 |
| *** e0ne has joined #tripleo | 19:57 | |
| vinsh | elements/mysql/install.d/10-mysql: ln -sf /usr/local/bin/mysql_install_db /usr/bin/mysql_install_db | 19:58 |
| vinsh | elements/mysql/install.d/10-mysql: sed -i -e 's,^basedir="",basedir="/usr/local/mysql",' scripts/mysql_install_db | 19:58 |
| vinsh | elements/mysql/install.d/10-mysql: install -o root -g root -m 0755 scripts/mysql_install_db /usr/local/bin | 19:58 |
| greghaynes | vinsh: are you using the switch to package based install thing? | 19:58 |
| *** dkehn_ has joined #tripleo | 19:58 | |
| vinsh | for some reason.. the install.d job isn't doing its thing correctly | 19:58 |
| lifeless | jdob: bug 1329068 | 19:58 |
| uvirtbot | Launchpad bug 1329068 in trove-integration "The trove guest-image is unable to publish the guest-config via cloud-init" [Critical,In progress] https://launchpad.net/bugs/1329068 | 19:58 |
| vinsh | greghaynes, I dont' know | 19:58 |
| greghaynes | vinsh: well, its a patch youd have to manually pull in, but if you did that stuff is possibly in the wrong place | 19:59 |
| lifeless | jdob: if we release with a regression, its an API break and thats bad | 19:59 |
| *** akuznetsov has quit IRC | 19:59 | |
| vinsh | greghaynes, I just see that during DIB.. 10-mysql says it ran.. yet on the resulting image.. the link it should have made when it ran.. isn't there. | 20:00 |
| *** dkehn_ has quit IRC | 20:00 | |
| greghaynes | lol | 20:00 |
| vinsh | 0_o | 20:00 |
| *** dkehn has quit IRC | 20:00 | |
| *** dkehn_ has joined #tripleo | 20:00 | |
| *** dkehnx has quit IRC | 20:00 | |
| jdob | ok, ya, this is the review that ben linked me earlier. lifeless, what's the right way to do this, since CI is failing. hold off on the release until CI passes for that review? | 20:00 |
| lifeless | jdob: hold off until the bug is fix committed | 20:01 |
| jdob | kk | 20:01 |
| jdob | lifeless: also, when you get a second, are you the one who can add me to tripleo-ptl so I can do the release? | 20:01 |
| lifeless | jdob: when that is is a separate discussion :). | 20:01 |
| lifeless | jdob: sure, whats your gerrit username ? | 20:01 |
| jdob | jdob | 20:02 |
| lifeless | jdob: you've read the docs on the wiki ? | 20:02 |
| jdob | the release management page? yes | 20:02 |
| jang1 | Hrm. I think it's all over... | 20:02 |
| lifeless | jdob: and spoken with e.g. rpodolyaka to get familiar with process? | 20:02 |
| *** dkehn has joined #tripleo | 20:02 | |
| jdob | not yet | 20:02 |
| *** e0ne has quit IRC | 20:02 | |
| lifeless | jdob: ok, so infra really don't like mistakes in tagging :) | 20:02 |
| lifeless | jdob: so I'll hold off until you've got a green light from one of the existing releasers :) | 20:03 |
| *** dprince has quit IRC | 20:03 | |
| lifeless | (thats why there is a separate team to the tripleo-core) | 20:03 |
| jdob | green light in the sense that I've spoken with someone to make sure I'm clear on the process? | 20:03 |
| vinsh | greghaynes, anything that might cause the install.d stuff to not really do what it should during DIB like this? | 20:03 |
| lifeless | jdob: yup | 20:04 |
| jdob | cool, that makes sense to me | 20:04 |
| lifeless | jdob: we don't have a formal test, though that would be cool | 20:04 |
| lifeless | vinsh: it should be set -e, so will error if it tried and failed | 20:04 |
| lifeless | vinsh: did it run it? Is it in the log ? | 20:05 |
| vinsh | 10-os-config-applier 3.300 | 20:05 |
| vinsh | 10-os-collect-config 9.963 | 20:05 |
| vinsh | 10-mysql 59.286 | 20:05 |
| vinsh | 10-locale 1.681 | 20:05 |
| vinsh | appears to be? | 20:05 |
| lifeless | yup | 20:05 |
| jdob | lifeless: what's the timeframe on the release? if the bug doesn't land tomorrow is there a problem with the release sliding? | 20:05 |
| lifeless | certainly looks that way higher up you should see its invocation | 20:05 |
| lifeless | jdob: there's no problem | 20:05 |
| lifeless | jdob: we release to keep inventory low | 20:05 |
| jdob | gotcha, so its an informal thing | 20:06 |
| lifeless | jdob: right now with CI wedged, inventory isn't building up in the fix-committed part :) | 20:06 |
| jdob | haha | 20:06 |
| vinsh | lifeless, I don't see the invocation of any of them in the devtest_overcloud ouput.. just that list showing times taken | 20:06 |
| lifeless | vinsh: look for something like dib-run-parts Fri Jun 20 08:07:41 NZST 2014 Running /tmp/image.l4xdukW4/hooks/extra-data.d/75-inject-pip-manifests | 20:08 |
| *** casanch1 has joined #tripleo | 20:08 | |
| vinsh | Yeah, nadda | 20:10 |
| vinsh | only appears in that list with the time take listed next to it :-/ | 20:10 |
| vinsh | unless its in another log besides devtest_overcloud output | 20:10 |
| lifeless | thats very very odd | 20:11 |
| vinsh | Maybe just deleting the image and everything from triple-root isn't enough | 20:11 |
| lifeless | do you have those 'Running ....' outputs at all ? | 20:11 |
| vinsh | cache=1 might still use old data? | 20:11 |
| vinsh | I do have those.. but are for cleanup.d things only | 20:12 |
| lifeless | I have them for install.d: | 20:12 |
| lifeless | dib-run-parts Thu Jun 19 20:11:29 UTC 2014 100-horizon completed | 20:12 |
| *** CaptTofu_ has quit IRC | 20:13 | |
| vinsh | I am guessing that usecache=1 got me. even with deleting the image from root | 20:14 |
| vinsh | wild guess maybe | 20:14 |
| greghaynes | it shouldnt :/ | 20:14 |
| greghaynes | if its not there | 20:15 |
| lifeless | vinsh: very much doubt that | 20:15 |
| greghaynes | Probably gamma rays | 20:15 |
| vinsh | well, I have the entire console log.. and can see items like 99-up-to-date invoked in output | 20:16 |
| vinsh | but not 10-mysql.. though it shows as completed in the end list | 20:16 |
| vinsh | "/tmp/in_target.d/install.d/99-dkms" | 20:16 |
| lifeless | yet its there with timing data | 20:16 |
| vinsh | doesn't show up as an image like you posted though.. | 20:17 |
| lifeless | its also set -x so you should see its actions | 20:17 |
| lifeless | its missing -u which is bad form | 20:17 |
| vinsh | I see it now! | 20:18 |
| vinsh | AFTer i set | 20:18 |
| vinsh | export DIB_DEBIAN_USE_CACHED_IMAGE=0 | 20:18 |
| vinsh | and USE_CACHE=0 | 20:18 |
| vinsh | not present in out put otherwise. | 20:19 |
| vinsh | back to you non vinsh program. will see what I get in these images. ;) | 20:19 |
| *** CaptTofu_ has joined #tripleo | 20:21 | |
| *** nati_uen_ has quit IRC | 20:25 | |
| *** nati_ueno has joined #tripleo | 20:25 | |
| *** nati_ueno has quit IRC | 20:27 | |
| *** nati_ueno has joined #tripleo | 20:28 | |
| *** blamar has joined #tripleo | 20:28 | |
| *** rcarrillocruz has joined #tripleo | 20:29 | |
| *** rcarrill` has quit IRC | 20:30 | |
| *** akuznetsov has joined #tripleo | 20:34 | |
| *** casanch1 has quit IRC | 20:34 | |
| *** casanch1 has joined #tripleo | 20:34 | |
| *** mestery has quit IRC | 20:42 | |
| *** mestery has joined #tripleo | 20:43 | |
| tchaypo | Jdob: pong | 20:44 |
| jdob | tchaypo: er, sorry, unping | 20:45 |
| tchaypo | Haha | 20:46 |
| *** mestery has quit IRC | 20:47 | |
| *** akuznetsov has quit IRC | 20:51 | |
| *** jomara has quit IRC | 20:51 | |
| *** jomara has joined #tripleo | 20:57 | |
| *** rcarrill` has joined #tripleo | 20:57 | |
| *** jp_at_hp has quit IRC | 20:57 | |
| *** jp_at_hp has joined #tripleo | 20:58 | |
| *** jdob has quit IRC | 20:58 | |
| *** rcarrillocruz has quit IRC | 21:00 | |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | Untriaged bugs so far: | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331931 | 21:00 |
| uvirtbot | Launchpad bug 1331931 in tripleo "NeutronControlPlaneID should not be set if building image only" [Undecided,In progress] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1332321 | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331867 | 21:00 |
| uvirtbot | Launchpad bug 1332321 in tripleo "Undercloud horizon resource tab returns an error due to missing ceilometer components " [Undecided,New] | 21:00 |
| uvirtbot | Launchpad bug 1331867 in tripleo "Building undercloud failed" [Undecided,New] | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| lifeless | any further progress on CI ? | 21:03 |
| SpamapS | lifeless: I believe dprince was finding that DHCP was failing "sometimes" by attaching to the VNC console of the instance | 21:10 |
| SpamapS | oh but you probably know that | 21:10 |
| lifeless | I'm trying to repro myself, currently on an abnormally long overcloud stack-ready wait | 21:12 |
| *** casanch1 has quit IRC | 21:13 | |
| lifeless | Ng: around ? | 21:18 |
| lifeless | tchaypo: when you get up, lets talk about getting CI regions up | 21:18 |
| lifeless | greghaynes: I think I'm going to ask for more volunteers to context switch to this, as capacity matters. | 21:18 |
| lifeless | greghaynes: if you / adam_g are interested (or know of other interested folk) that would rock | 21:19 |
| jogo | lifeless: https://bugs.launchpad.net/nova/+bug/1201656 | 21:20 |
| uvirtbot | Launchpad bug 1201656 in tripleo "nova baremetal ties nova-compute to one architecture" [High,Fix released] | 21:20 |
| jogo | with nova barmetal being deprecated is that bug still valid? | 21:20 |
| lifeless | jogo: depends on nova's position on bugs in deprecated but not deleted code. | 21:21 |
| lifeless | ho we did actually land the three controller patch | 21:22 |
| lifeless | I wonder | 21:22 |
| *** dshulyak_ has quit IRC | 21:22 | |
| jogo | lifeless: well bugs in deprecated code aren't high priority | 21:22 |
| jogo | let me ask a different question, will tripleo being fixing it? | 21:22 |
| lifeless | no, never planned to. Its there because it is a bug, and folk will a) run into it and b) we don't want a hundred and one near-dups filed. | 21:23 |
| bnemec | From 192.0.2.46 icmp_seq=1 Destination Host Unreachable | 21:24 |
| bnemec | Sonofa... | 21:24 |
| bnemec | Why am I the only one who can reproduce this locally? | 21:24 |
| *** morganfainberg is now known as morganfainberg_Z | 21:24 | |
| lifeless | bnemec: have you tried a single node ctl plane ? | 21:24 |
| bnemec | lifeless: That's how I'm running. Single control, single compute. | 21:26 |
| Ng | lifeless: yep. ssl certs for hp2 are ordered, awaiting verification | 21:26 |
| devananda | lifeless: have you seen https://github.com/openstack/ironic-specs/blob/master/specs/juno/add-node-instance-info.rst and https://review.openstack.org/#/c/95551/11 | 21:27 |
| devananda | i'd like to be sure that tripleo-incubator and related tooling starts migrating to use instance_info soon, ideally by J2, so we can finish deprecating the use of driver_info for non-driver-specific things | 21:28 |
| *** jprovazn_afk has quit IRC | 21:32 | |
| *** dkehn_ is now known as dkehnx | 21:32 | |
| *** openstackgerrit has joined #tripleo | 21:34 | |
| *** jang1 has quit IRC | 21:36 | |
| lifeless | devananda: I don't believe we do any direc tmanipulation of instance info | 21:45 |
| lifeless | devananda: the only things we do with ironic's api is register nodes and turn them off | 21:46 |
| tchaypo | lifeless: Have to go to my sister in law's birthday breakfast, but I should be in the office with Steven about 10 | 21:47 |
| lifeless | tchaypo: ok cool. | 21:47 |
| *** dkehn_ has joined #tripleo | 21:48 | |
| *** dkehn_ has quit IRC | 21:49 | |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Split api.py into multiple files https://review.openstack.org/101344 | 21:49 |
| *** dkehn has quit IRC | 21:49 | |
| *** dkehn_ has joined #tripleo | 21:49 | |
| devananda | lifeless: this patch means you will no longer set the pxe_deploy_* options on a flavor,a nd instead pass them in when enrolling a node in ironic | 21:50 |
| lifeless | bnemec: Ah,I had manually set controlscale to 3 | 21:50 |
| *** dkehnx has quit IRC | 21:50 | |
| devananda | lifeless: so i think it will require changes in tripleo-incubator | 21:50 |
| lifeless | devananda: I am confuse, instance_info will be wiped every time | 21:50 |
| lifeless | devananda: so how can that tie into what you just said | 21:51 |
| devananda | er | 21:51 |
| devananda | driver info | 21:51 |
| devananda | jugglign too many meetings :p | 21:51 |
| *** dkehn has joined #tripleo | 21:51 | |
| lifeless | devananda: ok yes we need to a) switch to the occ register-nodes and b) have that fix in there | 21:52 |
| devananda | tldr; you will need to put *deploy* image UUIDs on driver_info instead of on nova flavor | 21:52 |
| devananda | right now, ironic supports both | 21:52 |
| lifeless | devananda: well, backwards compat. We need to set it on both. | 21:52 |
| devananda | and there's a migration script proposed to mvoe live nodes over | 21:52 |
| *** noslzzp has quit IRC | 21:52 | |
| devananda | lifeless: right | 21:52 |
| lifeless | devananda: is there a bug filed on os-cloud-config and tripleo for this? | 21:53 |
| devananda | not yet, no | 21:53 |
| *** ddieterly has quit IRC | 21:53 | |
| lifeless | could you please file one? Since you have the details. I'll point folk at it in next weeks meeting | 21:53 |
| devananda | NobodyCam: we need the tripleo-undercloud test going soon ... | 21:53 |
| devananda | lifeless: ack | 21:53 |
| devananda | thanks | 21:53 |
| *** morganfainberg_Z is now known as morganfainberg | 21:55 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/tripleo-ci: Try reverting Neutron ovs-agent change https://review.openstack.org/101349 | 22:02 |
| bnemec | Restarting l3-agent and ovs-agent on my controller node seems to have fixed the problem in one of my environments. | 22:02 |
| bnemec | lifeless: Figured I would give ^ a shot. | 22:02 |
| bnemec | I think that merged around the time we starting having issues and it looked like about the only thing that could be related. | 22:03 |
| NobodyCam | devananda: its got two +2's | 22:06 |
| NobodyCam | https://review.openstack.org/#/c/100063/ | 22:06 |
| NobodyCam | I'll poke infra | 22:06 |
| *** CaptTofu_ has quit IRC | 22:06 | |
| devananda | NobodyCam: awesome, ty | 22:06 |
| *** CaptTofu_ has joined #tripleo | 22:07 | |
| *** nati_ueno has quit IRC | 22:07 | |
| *** nati_ueno has joined #tripleo | 22:07 | |
| lifeless | bnemec: yeah, longshot, but worth checking | 22:10 |
| lifeless | bnemec: if that isn't it I think we need to write some code to bisect the openstack global timeline | 22:10 |
| *** CaptTofu_ has quit IRC | 22:11 | |
| *** nati_ueno has quit IRC | 22:11 | |
| *** nati_ueno has joined #tripleo | 22:12 | |
| *** davidlenwell is now known as davidlenwell_ | 22:14 | |
| *** davidlenwell_ is now known as davidlenwell | 22:14 | |
| *** ChanServ sets mode: +v davidlenwell | 22:14 | |
| bnemec | lifeless: That should be fun. :-) Probably be useful again in the future though. | 22:16 |
| bnemec | lifeless: Unfortunately I'm not going to be around much tomorrow again. | 22:16 |
| *** CaptTofu_ has joined #tripleo | 22:20 | |
| *** jp_at_hp has quit IRC | 22:24 | |
| lifeless | wat | 22:24 |
| lifeless | sudo tail -F /var/log/upstart/os-collect-config.log | 22:24 |
| lifeless | sudo: error in /etc/sudo.conf, line 0 while loading plugin `sudoers_policy' | 22:24 |
| lifeless | sudo: unable to load /usr/lib/sudo/sudoers.so: (null) | 22:24 |
| lifeless | sudo: fatal error, unable to load plugins | 22:24 |
| *** ccrouch1 has quit IRC | 22:25 | |
| *** jp_at_hp has joined #tripleo | 22:26 | |
| *** andreaf_ has quit IRC | 22:26 | |
| lifeless | now + user-config | 22:29 |
| lifeless | ERROR: HTTPConnectionPool(host='192.0.2.25', port=8774): Max retries exceeded with url: /v2/73a06626a3264a31a82d7075ac2878cf/os-keypairs (Caused by <class 'httplib.BadStatusLine'>: '') | 22:29 |
| lifeless | something verra odd there | 22:29 |
| *** jp_at_hp has quit IRC | 22:32 | |
| *** blamar has quit IRC | 22:34 | |
| *** openstackgerrit has quit IRC | 22:34 | |
| *** 20WAAHXAJ has joined #tripleo | 22:36 | |
| *** ccrouch has joined #tripleo | 22:39 | |
| lifeless | hmm, keystone is whinging | 22:44 |
| lifeless | 2014-06-19 22:23:29.001 6918 WARNING keystoneclient.middleware.auth_token [-] Configuring admin URI using auth fragments. This is deprecated, use 'identity_uri' instead. | 22:44 |
| lifeless | morganfainberg: whats ^ about ? | 22:45 |
| *** openstack has joined #tripleo | 22:51 | |
| *** CaptTofu_ has quit IRC | 22:55 | |
| *** CaptTofu_ has joined #tripleo | 22:55 | |
| SpamapS | lifeless: yeah there's a bug there | 22:56 |
| SpamapS | lifeless: its the middleware | 22:56 |
| SpamapS | lifeless: they want us to just feed a URL in, instead of the component pieces | 22:56 |
| lifeless | SpamapS: shouldn't it look it up in keystone? | 22:56 |
| SpamapS | lifeless: this is _to find keystone_ | 22:57 |
| SpamapS | auth_url | 22:57 |
| morganfainberg | lifeless, uhm | 22:57 |
| morganfainberg | lifeless, uhhhhhhhh | 22:57 |
| morganfainberg | lifeless, wrong config option used? identity_url should be the option instead? | 22:57 |
| morganfainberg | s/url/uri | 22:58 |
| SpamapS | morganfainberg: is that different than auth_uri ? | 22:58 |
| morganfainberg | SpamapS, yeah, but i can't remember off the top of my head the difference, i think this is about the unversioned endpoing discovery | 22:59 |
| morganfainberg | let me check w/ jamielennox | 22:59 |
| *** CaptTofu_ has quit IRC | 23:00 | |
| SpamapS | morganfainberg: for reference, https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements/neutron/os-apply-config/etc/neutron/neutron.conf#n69 | 23:00 |
| morganfainberg | ooooh | 23:01 |
| morganfainberg | no i know what it is | 23:01 |
| morganfainberg | instead of using auth_host, auth_port, auth_protocol, | 23:01 |
| lifeless | ok, second time + user-config | 23:01 |
| lifeless | ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-8a04c55a-38d3-44ee-bb89-c2dc29dd5d77) | 23:01 |
| lifeless | has failed | 23:01 |
| morganfainberg | you can use identity_uri=http://{{keystone.host}}:35357 etc | 23:02 |
| morganfainberg | lifeless, https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L221 | 23:02 |
| SlickNik | SpamapS: Commented on https://review.openstack.org/#/c/99505 | 23:03 |
| morganfainberg | lifeless, the construction from the bits and peices is deprecated, but will still work | 23:03 |
| morganfainberg | SpamapS, ^ | 23:03 |
| SpamapS | SlickNik: ty | 23:03 |
| SlickNik | SpamapS: You were mentioning that we might want to look into changing the defaults for 12.04, so I looked into what was getting set in a 12.04 image (without the element). | 23:05 |
| SlickNik | SpamapS: In that case, there is a 90_dpkg.cfg file which has: | 23:05 |
| SlickNik | "# to update this file, run dpkg-reconfigure cloud-init | 23:05 |
| SlickNik | datasource_list: [ NoCloud, ConfigDrive, OVF, MAAS, Ec2 ]" | 23:05 |
| *** CaptTofu_ has joined #tripleo | 23:05 | |
| SpamapS | morganfainberg: https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements/cinder/os-apply-config/etc/cinder/cinder.conf#n57 .. different fields .. so confusing. :( | 23:06 |
| morganfainberg | SpamapS, *blink* | 23:07 |
| morganfainberg | yeah :( | 23:07 |
| morganfainberg | SpamapS, this is part of what we'll be fixing with the extracted middleware | 23:07 |
| morganfainberg | SpamapS, /me is working on extracting middleware into it's own package | 23:07 |
| lifeless | wat | 23:07 |
| lifeless | /var/log/upstart/nova-api.log:2014-06-19 23:01:02.560 6933 ERROR nova.api.openstack [req-8a04c55a-38d3-44ee-bb89-c2dc29dd5d77 None] Caught error: [Errno 12] Cannot allocate memory | 23:07 |
| morganfainberg | we're going to aim to fix some of this wonkyness with the 1.0 release | 23:08 |
| SpamapS | morganfainberg: cool | 23:08 |
| SpamapS | lifeless: nova's getting old | 23:08 |
| morganfainberg | (first release will be 1.0.0) | 23:08 |
| morganfainberg | SpamapS, but splitting and merging in git repositories from all over is ... uhhhh | 23:08 |
| morganfainberg | "fun" | 23:08 |
| lifeless | SpamapS: this is trunk | 23:08 |
| morganfainberg | i think i'm up to 4 split/merges | 23:08 |
| morganfainberg | SpamapS, it's safe (yeah i know deprecation warning) to use the old style constructed uri | 23:09 |
| lifeless | wtf total used free shared buffers cached | 23:09 |
| lifeless | Mem: 2.0G 1.9G 100M 940K 10M 73M | 23:09 |
| lifeless | -/+ buffers/cache: 1.8G 184M | 23:09 |
| lifeless | openstack not fitting in a 2G VM. Fark. | 23:09 |
| morganfainberg | with the spec to split the middleware we wont be changing the middleware except for security-fixes in the current location(keystoneclient) so don't worry about the options disappearing | 23:09 |
| SpamapS | morganfainberg: just promise you'll run 'check experimental' before you land it :) | 23:13 |
| morganfainberg | SpamapS, before we land the change to convert people over to it, i will absolutely do so | 23:14 |
| SpamapS | morganfainberg: man.. I'm craving shabu shabu again.. :) | 23:18 |
| morganfainberg | SpamapS, that stuff was awesome! | 23:19 |
| bnemec | WTH? The VM pinged fine for about 30 seconds then it died again. | 23:19 |
| SpamapS | yeah.. want | 23:19 |
| * bnemec gives up | 23:19 | |
| morganfainberg | i'm def. up to go again. | 23:19 |
| SpamapS | bnemec: :-/ | 23:19 |
| SpamapS | bnemec: overlay networks are the pits | 23:19 |
| *** yamahata has joined #tripleo | 23:19 | |
| SpamapS | morganfainberg: yeah let's try to go next week some time. | 23:20 |
| bnemec | If you can believe it, I actually have less pleasant things than debugging CI failures that I need to go do. :-/ | 23:20 |
| morganfainberg | SpamapS, sounds like a plan! | 23:20 |
| *** dkehn_ is now known as dkehnx | 23:20 | |
| *** noslzzp has joined #tripleo | 23:21 | |
| 20WAAHXAJ | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Split api.py into multiple files https://review.openstack.org/101344 | 23:25 |
| lifeless | interesting nick there gerritbot | 23:27 |
| lifeless | ok so | 23:29 |
| *** 20WAAHXAJ has quit IRC | 23:29 | |
| lifeless | openstack has got fat | 23:29 |
| *** openstackgerrit has joined #tripleo | 23:31 | |
| lifeless | bnemec: I finally got an end to end run ok | 23:31 |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Do not use DatasourceNone for precise cloud-init https://review.openstack.org/99505 | 23:31 |
| lifeless | bnemec: by increasing my node size to 3G | 23:32 |
| lifeless | bnemec: I'm just looking to see what arch we use in CI | 23:32 |
| ccrouch | on fedora it should be amd64 | 23:32 |
| bnemec | lifeless: I was already running 3G or my undercloud OOMs every time. | 23:32 |
| lifeless | bnemec: CI is running default node size | 23:33 |
| lifeless | erm, no its not we pull it in from gearman-worker.mem-per-env | 23:33 |
| bnemec | lifeless: CI never uses the undercloud though. | 23:33 |
| bnemec | I think it boots fine, but when you try to load all the overcloud stuff it dies. | 23:33 |
| lifeless | bnemec: that was overcloud control config that fails for me | 23:33 |
| lifeless | bnemec: with 2G nodes | 23:33 |
| lifeless | we make amd64 arch nodes | 23:34 |
| SpamapS | on purpose? | 23:35 |
| lifeless | SpamapS: I think so, fidelity. | 23:35 |
| SpamapS | Just spun up a devstack today.. | 23:35 |
| SpamapS | total used free shared buffers cached | 23:35 |
| SpamapS | Mem: 7985 3987 3997 0 94 1963 | 23:35 |
| SpamapS | -/+ buffers/cache: 1929 6055 | 23:35 |
| lifeless | so I'm wondering | 23:35 |
| lifeless | 4G lol | 23:35 |
| SpamapS | ubuntu 12171 0.4 1.1 289744 96164 pts/8 S+ 23:26 0:02 /usr/bin/python /usr/local/bin/nova-api | 23:36 |
| uvirtbot | Launchpad bug 12171 in gaim "/usr/share/doc/gaim is empty" [Unknown,Fix released] https://launchpad.net/bugs/12171 | 23:36 |
| SpamapS | ubuntu 12172 0.4 1.1 289712 96188 pts/8 S+ 23:26 0:02 /usr/bin/python /usr/local/bin/nova-api | 23:36 |
| uvirtbot | Launchpad bug 12172 in ubuntu "Keyboard language menu in the installer is not sorted (dup-of: 9520)" [Low,Invalid] https://launchpad.net/bugs/12172 | 23:36 |
| uvirtbot | Launchpad bug 9520 in kbd-chooser "Ubuntu should have a better method of selecting the keyboard layout" [Medium,Fix released] https://launchpad.net/bugs/9520 | 23:36 |
| SpamapS | top two processes | 23:36 |
| SpamapS | (rss anyway) | 23:36 |
| lifeless | oh uvirtbot no you didn't | 23:36 |
| SpamapS | hahahaha | 23:36 |
| SpamapS | ubuntu 666 | 23:37 |
| uvirtbot | Launchpad bug 666 in launchpad "can't file a bug on Ubuntu" [Medium,Invalid] https://launchpad.net/bugs/666 | 23:37 |
| SpamapS | irony | 23:37 |
| * SpamapS needs to take a break before diving back into heat dev | 23:37 | |
| lifeless | ok so we grab 'free' output | 23:38 |
| lifeless | hmm, I'd prefer free -h | 23:38 |
| openstackgerrit | lifeless proposed a change to openstack-infra/tripleo-ci: Use free -h for us mere mortals. https://review.openstack.org/101366 | 23:39 |
| *** xuhaiwei has joined #tripleo | 23:44 | |
| *** mestery has joined #tripleo | 23:46 | |
| lifeless | ok so the CI fail isn't memory | 23:48 |
| lifeless | total used free shared buffers cached | 23:48 |
| lifeless | Mem: 4048704 3148732 899972 444 71808 352396 | 23:48 |
| lifeless | -/+ buffers/cache: 2724528 1324176 | 23:48 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Warn about 2GB w/amd64. https://review.openstack.org/101369 | 23:50 |
| *** dkehn_ has joined #tripleo | 23:50 | |
| *** dkehn_ has quit IRC | 23:51 | |
| *** dkehn has quit IRC | 23:51 | |
| lifeless | bnemec: two runs now, works both times. | 23:51 |
| *** dkehn_ has joined #tripleo | 23:51 | |
| *** dkehnx has quit IRC | 23:51 | |
| lifeless | bnemec: AIUI its fedora and ubuntu right? I don't need to build a fedora set of images? | 23:51 |
| bnemec | lifeless: Yeah, the check experimental on Ubuntu failed too. | 23:52 |
| lifeless | bnemec: I'm looping on overcloud only now. | 23:52 |
| lifeless | bnemec: cached image, again and again | 23:52 |
| *** dkehn has joined #tripleo | 23:52 | |
| *** nati_ueno has quit IRC | 23:54 | |
| bnemec | lifeless: Yeah, I seem to be the only one hitting this in local devtest. I've gotten it every time I've run it in the past two days though. | 23:54 |
| lifeless | wtf | 23:54 |
| *** nati_ueno has joined #tripleo | 23:55 | |
| *** nati_ueno has quit IRC | 23:55 | |
| *** nati_ueno has joined #tripleo | 23:56 | |
| lifeless | maybe we're missing something | 23:58 |
| lifeless | like firewall rules | 23:58 |
| lifeless | on the jenkins slave | 23:58 |
| lifeless | or seed | 23:58 |
| lifeless | ip forwarding | 23:58 |
| * lifeless throws ideas out | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!