| *** jcooley_ has quit IRC | 00:13 | |
| *** jcooley_ has joined #tripleo | 00:14 | |
| *** jcooley_ has quit IRC | 00:28 | |
| *** jcooley_ has joined #tripleo | 00:29 | |
| *** jcooley_ has quit IRC | 00:34 | |
| *** jcooley_ has joined #tripleo | 00:35 | |
| *** jcooley_ has quit IRC | 00:57 | |
| *** jcooley_ has joined #tripleo | 00:58 | |
| *** jcooley_ has quit IRC | 01:03 | |
| peoplemerge | SpamapS: Just looking at the schedule at SCALE, noticed you're giving a talk. Looking forward to it! | 01:10 |
|---|---|---|
| *** cd-undercloud has joined #tripleo | 01:11 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 01:11 |
| *** cd-undercloud has quit IRC | 01:11 | |
| *** NearlyFunctional has quit IRC | 01:17 | |
| *** NearlyFunctional has joined #tripleo | 01:19 | |
| *** michchap_ has joined #tripleo | 01:20 | |
| *** yamahata_ has joined #tripleo | 01:22 | |
| *** jrist has quit IRC | 01:29 | |
| *** rwsu has quit IRC | 01:29 | |
| *** openstackgerrit has quit IRC | 01:29 | |
| *** yamahata__ has quit IRC | 01:29 | |
| *** michchap has quit IRC | 01:30 | |
| *** uvirtbot has quit IRC | 01:30 | |
| *** rwsu has joined #tripleo | 01:37 | |
| *** jrist has joined #tripleo | 01:37 | |
| lifeless | peoplemerge: gl! | 01:42 |
| lifeless | SpamapS: ohhai? | 01:42 |
| lifeless | InstanceDeployFailure: Timeout reached while waiting for PXE deploy of instance b2bf8601-240d-4ee8-b19d-93c10ddb26d1 | 01:44 |
| *** uvirtbot has joined #tripleo | 01:50 | |
| lifeless | StevenK: ah, you don't mirror sources I'm guessing ? | 02:31 |
| lifeless | StevenK: since my shiny new apt-mirror is many more GB than 50 :) | 02:32 |
| *** cd-undercloud has joined #tripleo | 02:36 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 02:36 |
| *** cd-undercloud has quit IRC | 02:36 | |
| SpamapS | lifeless: I thought StevenK said 600G? | 02:48 |
| SpamapS | for "all" | 02:48 |
| peoplemerge | lifeless: thanks. Last time I got stuck on testenv https://gist.github.com/peoplemerge/8863542 | 02:48 |
| peoplemerge | This time I did it all more thoughtfully. Rebooting after devtest_setup and before testenv fixed that (libvirt permissions err?) | 02:50 |
| peoplemerge | I was able to build seed without all that complicated manual bridging step SpamapS and I did | 02:51 |
| peoplemerge | it looks good so far. Gotta go buy some ski gear with wifey, be back after dinner :) | 02:51 |
| lifeless | SpamapS: oh hi! see my question back before you disappeared ;) | 02:52 |
| SpamapS | lifeless: ah, so a) :( more bad machines | 02:53 |
| SpamapS | lifeless: b) The "without heat's help" solution just requires that we infer the change that is coming and react accordingly... | 02:53 |
| lifeless | SpamapS: you say potato I saw wtf you talking about willis? | 02:54 |
| SpamapS | lifeless: I'm worried about getting things into Heat actually. I feel that there is hyper-conservative behavior dominating the discussions so it may take a while to land things. :-/ | 02:55 |
| SpamapS | lifeless: Could just be my perceptions. | 02:55 |
| *** CaptTofu has joined #tripleo | 02:55 | |
| lifeless | SpamapS: perhaps. still, I want to understand your idea | 02:55 |
| lifeless | I don't, atm | 02:55 |
| SpamapS | lifeless: so we'd just have a pre-server waitcondition and launchconfig that exposes the same things as the current post-server launchconfig and waitcondition (except the server's address) ... | 02:57 |
| SpamapS | lifeless: if we see image ID changing, we'd know that a rebuild is coming, and do any appropriate actions, then ping back that wait condition. | 02:57 |
| SpamapS | lifeless: this is pure evil, but it would provide ammunition to counter the conservative "don't make Heat a workflow engine" folks. | 02:58 |
| lifeless | SpamapS: I see, so the evil bit is that we're duplicating the entire config descriptor | 02:58 |
| SpamapS | and inferring the action | 02:59 |
| lifeless | SpamapS: we only need the image property tough | 02:59 |
| lifeless | none of the rest - just a second cfn source for occ | 02:59 |
| SpamapS | Yeah that is true | 02:59 |
| lifeless | what about deletes? | 02:59 |
| SpamapS | nothing implicit I can think of.. we'd have to actually set a "you're about to be deleted" flag | 03:00 |
| lifeless | ok | 03:00 |
| lifeless | so we can workaround heat not doing this intrinsically for rebuild,but not scale down | 03:00 |
| SpamapS | right | 03:00 |
| lifeless | so, I think I'd rather carry a patch than invest in a poor-mans solution that is incomplete like tis | 03:01 |
| lifeless | what do you think ? | 03:01 |
| SpamapS | Likewise | 03:04 |
| SpamapS | I'm working through the options | 03:04 |
| SpamapS | what about the hundreds of updates solution? | 03:04 |
| SpamapS | I _hate_ that one. | 03:04 |
| SpamapS | but I'm tired of arguing | 03:04 |
| lifeless | 2014-02-09 03:02:17.385 21895 ERROR nova.virt.libvirt.driver [-] [instance: 45daff91-84dd-4596-95c2-023afc6436c1] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://overcloud-novacompute1-bxq4qdzw7bif/system: Unable to resolve address 'overcloud-novacompute1-bxq4qdzw7bif' service '16509': Name or service not known | 03:04 |
| lifeless | SpamapS: hundreds of updates solution? | 03:04 |
| SpamapS | lifeless: oh sorry that is for rolling | 03:05 |
| lifeless | SpamapS: so I'm not seeing these arguments | 03:05 |
| lifeless | SpamapS: I'm entirely happy to go to battle for sanity for you :) | 03:05 |
| lifeless | SpamapS: if you want to focus on making the world a better place through code | 03:06 |
| SpamapS | dunno, have found it hard to even write code for this | 03:07 |
| SpamapS | trying to make a spec that is perfect :=P | 03:07 |
| lifeless | SpamapS: ouch | 03:07 |
| lifeless | SpamapS: why | 03:07 |
| SpamapS | lifeless: not sure. I feel the same about Heat as I did about Juju.. lots of smart people not caring much for their real users. | 03:09 |
| lifeless | SpamapS: ok, so lets fix that. | 03:10 |
| SpamapS | It could just be that I'm losing weight and projecting my negativity on my current problem set too. | 03:10 |
| lifeless | you're dieting> | 03:10 |
| lifeless | ? | 03:10 |
| SpamapS | Yeah a lot | 03:10 |
| lifeless | cool | 03:10 |
| lifeless | I'm trying but man | 03:10 |
| lifeless | with kids, so hard | 03:11 |
| lifeless | s/s// | 03:11 |
| SpamapS | I hit 260lbs. last week.. which caused me to throw out my back... so I'm on 1600 calories a day | 03:11 |
| lifeless | SpamapS: ah, so you had been anti-dieting | 03:11 |
| SpamapS | also called "stress eating" | 03:11 |
| lifeless | whats got you stressed? | 03:11 |
| SpamapS | That's a profound question that probably has a complicated answer. | 03:12 |
| lifeless | ok | 03:12 |
| lifeless | so zaneb seems to agree that having servers optionally notified of coming changes to the server itself via metadata is good short term answer to a bunch of things | 03:12 |
| lifeless | AFAICT we just need to show up with code now | 03:12 |
| SpamapS | Yeah I think that could be an easy one to get done. | 03:12 |
| lifeless | unless stevebaker wants a cordon-bleu-print | 03:13 |
| SpamapS | It's basically my poor-man's solution, but encoded in an argument to the server which is something like "metadata_notification_key: xxx" which will be a key in metadata where we put a dict with a pending action/callback url, etc. | 03:14 |
| *** mestery__ is now known as mestery | 03:14 | |
| lifeless | yes | 03:15 |
| lifeless | though since we use a decoupled metadata struct | 03:15 |
| SpamapS | metadata_notification_resource: too :-P | 03:15 |
| SpamapS | or unravel the circular dependency bits | 03:15 |
| lifeless | one thing at a time | 03:17 |
| SpamapS | the former seems simpler since the circular dep thing will be controversial and possibly complicated | 03:17 |
| SpamapS | lifeless: I think what is getting me is that it feels like updates are the red headed step child of Heat. | 03:17 |
| SpamapS | lifeless: anywa... airing doubts makes them feel less important. Thanks. | 03:18 |
| lifeless | SpamapS: so, I think that CFN (and thus Heat) were built on the basis of not needing what we need, because there are other tools like trove for db's | 03:21 |
| lifeless | but we're deploying with itself | 03:21 |
| lifeless | so we need to handle scenarios that the older design was able to just avoid | 03:25 |
| SpamapS | agreed | 03:33 |
| lifeless | anyhow, I think we have a simple, and easy answer here. | 03:48 |
| lifeless | so sometime monday it should be working and we can monkey patch it on :) | 03:49 |
| *** CaptTofu has quit IRC | 03:52 | |
| *** cody-somerville has joined #tripleo | 03:55 | |
| *** cody-somerville has joined #tripleo | 03:55 | |
| *** cd-undercloud has joined #tripleo | 03:57 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 03:57 |
| *** cd-undercloud has quit IRC | 03:57 | |
| StevenK | lifeless: Yeah, I don't mirror sources. That should only count as another arch, really. | 04:07 |
| * lifeless tries live block migration with qemi | 04:07 | |
| lifeless | qemu | 04:07 |
| lifeless | call me a daredevil | 04:07 |
| *** ohadlevy has quit IRC | 04:09 | |
| *** ohadlevy has joined #tripleo | 04:11 | |
| *** ohadlevy is now known as Guest29150 | 04:11 | |
| lifeless | hmm | 04:37 |
| lifeless | virNetTLSContextCheckCertFile:117 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory | 04:37 |
| lifeless | but I didn't turn tls on | 04:37 |
| lifeless | yay :/ https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/979212 | 04:40 |
| uvirtbot | Launchpad bug 979212 in libvirt "libvirtd --listen fails with: Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory" [Medium,Won't fix] | 04:40 |
| *** cd-undercloud has joined #tripleo | 05:19 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 05:19 |
| *** cd-undercloud has quit IRC | 05:19 | |
| *** killer_prince has joined #tripleo | 05:32 | |
| *** edmund1 has joined #tripleo | 05:37 | |
| *** edmund1 has quit IRC | 05:42 | |
| lifeless | I get the feeling I'm manually reproducing all the neutron races right now | 05:43 |
| lifeless | ../ | 05:43 |
| *** jcooley_ has joined #tripleo | 05:46 | |
| *** jcooley_ has quit IRC | 05:50 | |
| *** CaptTofu has joined #tripleo | 05:53 | |
| *** CaptTofu has quit IRC | 05:58 | |
| *** vkozhukalov has joined #tripleo | 06:23 | |
| *** cd-undercloud has joined #tripleo | 06:39 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 06:39 |
| *** cd-undercloud has quit IRC | 06:39 | |
| *** Guest29150 is now known as ohadlevy | 06:46 | |
| *** ohadlevy is now known as ohadlevy_ | 06:46 | |
| *** ohadlevy_ is now known as ohadlevy | 06:46 | |
| *** ohadlevy has joined #tripleo | 06:46 | |
| *** noslzzp has quit IRC | 06:58 | |
| *** akuznetsov has quit IRC | 07:07 | |
| *** akuznetsov has joined #tripleo | 07:12 | |
| *** akuznetsov has quit IRC | 07:22 | |
| *** akuznetsov has joined #tripleo | 07:37 | |
| *** mrunge has joined #tripleo | 07:51 | |
| *** e0ne has joined #tripleo | 07:53 | |
| *** CaptTofu has joined #tripleo | 07:54 | |
| *** CaptTofu has quit IRC | 07:59 | |
| *** cd-undercloud has joined #tripleo | 08:00 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 08:00 |
| *** cd-undercloud has quit IRC | 08:00 | |
| lifeless | StevenK: https://review.openstack.org/59699 merged - congrats | 08:10 |
| *** rlandy has joined #tripleo | 08:11 | |
| lifeless | greghaynes: that thing with rootwrap | 08:13 |
| lifeless | greghaynes: a thought - neutron knows where it's being installed (realpath etc) | 08:13 |
| lifeless | greghaynes: so, during install, it should specify that location as a valid path for binaries. | 08:13 |
| peoplemerge | OK The seed step is done, following devtest. Here's where it gets a bit hazy for me. I guess undercloud is optional here due to the modest size of my effort. | 08:22 |
| peoplemerge | do I run devtest_overcloud.sh on seed? | 08:23 |
| lifeless | peoplemerge: you're using real machines right? | 08:24 |
| peoplemerge | Also I don't see the part in docs where run `nova baremetal-node-create seed 2 4096 128 $NIC` but that does appear to happen on seed | 08:24 |
| peoplemerge | lifeless: yes | 08:24 |
| peoplemerge | lifeless: desktops, using fake power | 08:24 |
| lifeless | how many machines? | 08:25 |
| peoplemerge | lifeless: 5 | 08:25 |
| lifeless | so for that, you could do one undercloud, then 1 overcloud control and three hypervisors | 08:26 |
| lifeless | peoplemerge: I wouldn't skip undercloud, because deploy from real machines is much faster | 08:26 |
| lifeless | its nice to have an actual real box for the undercloud | 08:26 |
| lifeless | so for that, you're registering a single machine with the seed, then you source the seed rc, and then you can run devtest_undercloud | 08:26 |
| *** mrunge has quit IRC | 08:27 | |
| lifeless | the last few commits added support for metadata in the test environment describing real hardware | 08:27 |
| lifeless | its still a bit rough, but you may want to look at their diffs to get your head around it | 08:27 |
| lifeless | also, apropos of onthing | 08:28 |
| lifeless | HTF did this ever work : ./etc/neutron/rootwrap.d/l3.filters~:metadata_proxy_local: CommandFilter, /usr/local/bin/neuton-ns-metadata-proxy, root | 08:28 |
| peoplemerge | lifeless: sounds good. better not over think this | 08:39 |
| peoplemerge | lifeless: hm I guess my confusion is which commands do run where, ie run devtet #1-5 on host containing seed, #6 on seed, then run baremetalnodecreate, then boot a box and pxe should fire which should make the undercloud. | 08:45 |
| *** mrunge has joined #tripleo | 08:45 | |
| peoplemerge | will review last few commits | 08:49 |
| * peoplemerge -> bed | 08:50 | |
| *** e0ne has quit IRC | 08:58 | |
| lifeless | peoplemerge: all the commands are run from the same shell | 08:59 |
| *** e0ne has joined #tripleo | 08:59 | |
| *** cd-undercloud has joined #tripleo | 09:21 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 09:21 |
| *** cd-undercloud has quit IRC | 09:21 | |
| *** e0ne has quit IRC | 09:22 | |
| *** mrunge has quit IRC | 09:37 | |
| *** CaptTofu has joined #tripleo | 09:55 | |
| *** CaptTofu has quit IRC | 10:00 | |
| *** SpamapS has quit IRC | 10:06 | |
| *** mestery_ has joined #tripleo | 10:08 | |
| *** mestery has quit IRC | 10:10 | |
| *** SpamapS has joined #tripleo | 10:13 | |
| *** cd-undercloud has joined #tripleo | 10:42 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 10:42 |
| *** cd-undercloud has quit IRC | 10:42 | |
| *** jrist has quit IRC | 10:52 | |
| *** jrist has joined #tripleo | 11:05 | |
| *** boris-42_ has joined #tripleo | 11:11 | |
| *** boris-42 has quit IRC | 11:11 | |
| StevenK | lifeless: Woo. I owe Zhi Yan Liu at one beer, possibly more. | 11:26 |
| *** CaptTofu has joined #tripleo | 11:55 | |
| *** CaptTofu has quit IRC | 11:56 | |
| *** CaptTofu_ has joined #tripleo | 11:56 | |
| *** CaptTofu_ has quit IRC | 12:01 | |
| *** cd-undercloud has joined #tripleo | 12:03 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 12:03 |
| *** cd-undercloud has quit IRC | 12:03 | |
| *** CaptTofu has joined #tripleo | 12:14 | |
| *** CaptTofu has quit IRC | 13:12 | |
| *** nijaba_ has quit IRC | 13:18 | |
| *** nijaba has joined #tripleo | 13:18 | |
| *** nijaba has quit IRC | 13:18 | |
| *** nijaba has joined #tripleo | 13:18 | |
| *** cd-undercloud has joined #tripleo | 13:24 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 13:24 |
| *** cd-undercloud has quit IRC | 13:24 | |
| *** tserong has quit IRC | 13:33 | |
| *** tserong has joined #tripleo | 13:39 | |
| *** weshay has joined #tripleo | 14:12 | |
| *** nijaba has quit IRC | 14:28 | |
| *** noslzzp has joined #tripleo | 14:31 | |
| *** nijaba has joined #tripleo | 14:32 | |
| *** nijaba has quit IRC | 14:32 | |
| *** nijaba has joined #tripleo | 14:32 | |
| *** cd-undercloud has joined #tripleo | 14:45 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 14:45 |
| *** cd-undercloud has quit IRC | 14:45 | |
| mordred | StevenK: neat. what did Zhi Yan Liu do? | 14:56 |
| mordred | o m g | 14:56 |
| mordred | you got glance moved to testrepository? | 14:56 |
| *** nijaba has quit IRC | 14:56 | |
| * mordred hands StevenK a beer too | 14:57 | |
| *** nijaba has joined #tripleo | 14:57 | |
| *** nijaba has quit IRC | 14:57 | |
| *** nijaba has joined #tripleo | 14:57 | |
| mordred | are you tackling keystone too? | 14:57 |
| SpamapS | ok, time to pull latest heat | 15:08 |
| SpamapS | into undercloud | 15:08 |
| SpamapS | we'll get fixes for 500 error handling and event-list will finally be sane again | 15:08 |
| mordred | woot | 15:12 |
| *** nijaba has quit IRC | 15:13 | |
| *** nijaba has joined #tripleo | 15:16 | |
| *** e0ne has joined #tripleo | 15:17 | |
| *** ftcjeff has joined #tripleo | 15:54 | |
| *** openstackgerrit has joined #tripleo | 15:55 | |
| *** ftcjeff has quit IRC | 16:04 | |
| *** edmund has joined #tripleo | 16:25 | |
| *** nijaba has quit IRC | 16:44 | |
| *** nijaba has joined #tripleo | 16:45 | |
| *** nijaba has quit IRC | 16:45 | |
| *** nijaba has joined #tripleo | 16:45 | |
| *** e0ne has quit IRC | 16:56 | |
| *** rlandy has quit IRC | 17:03 | |
| *** cd-undercloud has joined #tripleo | 17:09 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 17:09 |
| *** cd-undercloud has quit IRC | 17:09 | |
| *** tserong has quit IRC | 17:12 | |
| *** tserong has joined #tripleo | 17:13 | |
| *** mestery_ is now known as mestery | 17:19 | |
| *** d0ugal has joined #tripleo | 17:26 | |
| *** d0ugal has joined #tripleo | 17:26 | |
| openstackgerrit | Richard Su proposed a change to openstack/tripleo-image-elements: Add service restart on tgtd for Fedora https://review.openstack.org/72154 | 17:31 |
| *** rwsu has quit IRC | 17:32 | |
| *** bauzas has joined #tripleo | 17:34 | |
| *** bauzas has quit IRC | 18:25 | |
| *** cd-undercloud has joined #tripleo | 18:33 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 18:33 |
| *** cd-undercloud has quit IRC | 18:33 | |
| lifeless | oh joy, now ovs-vswitchd segfaults. what did I do? | 18:35 |
| lifeless | also, morning | 18:35 |
| *** dkehn__ has joined #tripleo | 19:03 | |
| *** dkehn__ has quit IRC | 19:05 | |
| *** dkehn__ has joined #tripleo | 19:05 | |
| *** dkehn_ has quit IRC | 19:05 | |
| *** dkehn has quit IRC | 19:06 | |
| *** dkehn has joined #tripleo | 19:07 | |
| *** tserong has quit IRC | 19:14 | |
| lifeless | ahahahahaha | 19:24 |
| lifeless | [ 4.013765] init: mountall main process (210) killed by FPE signal | 19:24 |
| lifeless | General error mounting filesystems. | 19:24 |
| lifeless | A maintenance shell will now be started. | 19:24 |
| lifeless | CONTROL-D will terminate this shell and reboot the system. | 19:24 |
| lifeless | root@demo:~# | 19:24 |
| *** tserong has joined #tripleo | 19:24 | |
| *** tserong has joined #tripleo | 19:24 | |
| *** d0ugal has quit IRC | 19:40 | |
| *** cd-undercloud has joined #tripleo | 19:55 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 19:55 |
| *** cd-undercloud has quit IRC | 19:55 | |
| SpamapS | wtf | 19:58 |
| SpamapS | FPE?!!? | 19:58 |
| SpamapS | for mountall? | 19:58 |
| SpamapS | lifeless: is this on one of our failed boxes? | 19:59 |
| *** marun has joined #tripleo | 20:04 | |
| SpamapS | peoplemerge: sitting at computer for about 10 more minutes if you need help debugging | 20:08 |
| *** e0ne has joined #tripleo | 20:09 | |
| lifeless | SpamapS: thats on a vm inside the ci-overcloud | 20:16 |
| lifeless | SpamapS: whats more perplexing is this freaking gre tunnel not decapsulating /anything/ in my local vm based testbed | 20:17 |
| lifeless | now its not | 20:17 |
| lifeless | it was working | 20:17 |
| SpamapS | lifeless: can you please explain again how to attach to the shared screens for ilos btw? | 20:17 |
| SpamapS | it is not explained in /home/shared | 20:17 |
| SpamapS | and I don't use screen, so I am about to throw things at it. | 20:17 |
| lifeless | hah | 20:18 |
| lifeless | sec | 20:18 |
| SpamapS | screen -x ? screen -foad | 20:18 |
| lifeless | ssh to the bastion | 20:18 |
| lifeless | screen -Ax ilos{1,2} | 20:18 |
| SpamapS | also I'm not listed in the acl in the screen_ilorc1 | 20:18 |
| SpamapS | so thats probably half the problem | 20:18 |
| lifeless | SpamapS: neither am I | 20:19 |
| lifeless | SpamapS: so try | 20:19 |
| lifeless | screen -Ax ilos1 | 20:19 |
| lifeless | and see what happens | 20:19 |
| SpamapS | oh aclgrp users | 20:19 |
| SpamapS | n/m | 20:19 |
| SpamapS | no screen to be attached matching ilos1 | 20:19 |
| SpamapS | it has never worked | 20:19 |
| lifeless | SpamapS: try now | 20:21 |
| SpamapS | lifeless: we've been around this merry go round a few times | 20:21 |
| SpamapS | Must run suid root for multiuser support | 20:21 |
| SpamapS | it only works for the user running the screens | 20:21 |
| lifeless | SpamapS: /dont/ fiddle it | 20:21 |
| SpamapS | hah no I still have burn marks from the last time | 20:22 |
| lifeless | SpamapS: no, it has worked, last time you 'fixed' it by breaking it. | 20:22 |
| lifeless | SpamapS: so you no touchy. | 20:22 |
| SpamapS | this is why I don't use the shared screen btw | 20:22 |
| lifeless | sure, if it doesn't work thats a good reason not to use it | 20:22 |
| SpamapS | I'd love to fix it | 20:22 |
| lifeless | but not to not fix it | 20:22 |
| SpamapS | but I'm like ---> tmux holmes | 20:22 |
| lifeless | SpamapS: try now | 20:23 |
| SpamapS | same | 20:24 |
| lifeless | SpamapS: whats the specific error ? | 20:24 |
| SpamapS | clint@bm-aw1az2-freecloud0001:/home/shared$ screen -Ax robertc/ilos1 | 20:24 |
| SpamapS | Must run suid root for multiuser support. | 20:24 |
| SpamapS | (without your username, "There is no screen to be attached matching ilos1") | 20:24 |
| lifeless | hmmm | 20:24 |
| lifeless | screen is setuid already | 20:26 |
| lifeless | 6865.ilos2 (12/17/2013 01:29:01 AM) (Multi, detached) | 20:26 |
| lifeless | 5187.ilos1 (11/22/2013 01:07:55 AM) (Multi, detached) | 20:26 |
| SpamapS | lifeless: would it be terrible to just use a semaphore rather than use shared screens? | 20:27 |
| lifeless | they are marked multiuser | 20:27 |
| lifeless | Ng: around ? | 20:27 |
| SpamapS | screen is not suid root btw | 20:27 |
| lifeless | no ? | 20:27 |
| SpamapS | sgid utmp | 20:27 |
| SpamapS | I believe that's a dpkg-reconfigure thing | 20:28 |
| lifeless | sudo dpkg-reconfigure screen -p low finds nowthing | 20:28 |
| lifeless | nothing | 20:28 |
| *** CaptTofu has joined #tripleo | 20:29 | |
| SpamapS | gah | 20:29 |
| SpamapS | ok well I tried.. can I just ilo into freecloud 0030 and see why it can't pxe? | 20:29 |
| lifeless | Debian Modifications | 20:29 |
| lifeless | -------------------- | 20:29 |
| lifeless | * added Debian package maintenance files | 20:29 |
| lifeless | * Use /var/run/screen as socket directory | 20:29 |
| lifeless | * Make it set-gid "utmp" instead of setuid root | 20:29 |
| lifeless | ] dpkg-statoverride --update --add root utmp 4755 /usr/bin/screen | 20:29 |
| lifeless | ] chmod 0755 /var/run/screen | 20:29 |
| SpamapS | Debian brokeded it | 20:29 |
| lifeless | debian, I love you. SOMETIMES. | 20:29 |
| lifeless | so, try now | 20:30 |
| SpamapS | works | 20:32 |
| SpamapS | ugh | 20:32 |
| SpamapS | if you ctrl-c the sleep the window is lost | 20:32 |
| *** noslzzp has quit IRC | 20:32 | |
| lifeless | haha, yeah | 20:32 |
| lifeless | SpamapS: sec restarting it | 20:33 |
| lifeless | screen -S ilos1 -c ilo_screenrc1 for reference | 20:34 |
| SpamapS | aye | 20:34 |
| SpamapS | Access to session denied now | 20:35 |
| lifeless | orly | 20:36 |
| lifeless | try now? | 20:36 |
| SpamapS | mind if I add 'defzombie rd' to the rcs? | 20:37 |
| lifeless | gimme a sec | 20:37 |
| lifeless | did you manage to connect? | 20:37 |
| SpamapS | that will just make the ctrl-c try again if an ilo exits | 20:37 |
| SpamapS | I did | 20:37 |
| SpamapS | though.. fun.. 'ctrl-A d' cannot detach | 20:38 |
| lifeless | ! | 20:38 |
| lifeless | ok, i've jus ttweaked the rcs to get you access on startup | 20:38 |
| lifeless | hopefully | 20:38 |
| lifeless | make your changes | 20:38 |
| mordred | SpamapS: tmux ftw | 20:41 |
| lifeless | bah | 20:42 |
| lifeless | I bet you like systemd too | 20:42 |
| mordred | SpamapS: btw - I tried spinning up a new raring node in hpcloud to host my IRC stuff (Which is on precise) and SOMETHING about the byobu/tmux config in raring is different from precise and the keybindings are all wonky now | 20:42 |
| lifeless | mordred: you use byobu? | 20:42 |
| mordred | as in, pg-up/pg-dn don't work anymore | 20:42 |
| mordred | lifeless: hell yes | 20:42 |
| *** vkozhukalov has quit IRC | 20:42 | |
| mordred | lifeless: screen is a pile of poop | 20:43 |
| SpamapS | byobu is tmux now ;) | 20:43 |
| lifeless | mordred: byobu used to be based on screen... so thats a orthogonal answer. | 20:43 |
| mordred | screws with all of my keybindings - so I avoided it until byobu came along and unbroke it | 20:43 |
| SpamapS | but I have given up using it.. it wastes so much load | 20:43 |
| lifeless | mordred: I'm just surprised, since i didn't think I knew any byobu users. | 20:43 |
| mordred | me | 20:43 |
| mordred | quite aggressively | 20:43 |
| lifeless | existence proof | 20:43 |
| lifeless | who knew | 20:43 |
| mordred | :) | 20:43 |
| SpamapS | I've gotten used to the tmux keys | 20:44 |
| mordred | SpamapS: does tmux break existing keybindings out of the box like screen does? | 20:44 |
| mordred | ctrl-a is the big one | 20:44 |
| mordred | the asstastic fail of screen using that is amazing to me | 20:44 |
| mordred | I'm sorry | 20:44 |
| lifeless | mordred: emacs user? | 20:44 |
| SpamapS | they use ctrl-b | 20:44 |
| mordred | emacs has been aroudn for SO MANY MORE YEARS | 20:44 |
| SpamapS | which I don't use for anything | 20:44 |
| mordred | lifeless: well, yeah - also bash command line editing | 20:44 |
| lifeless | mordred: if you don't have vim mode turned on :P | 20:45 |
| mordred | typing ctrl-a a is stupid | 20:45 |
| SpamapS | screen's upstream is, IIRC, completely dead | 20:45 |
| mordred | lifeless: I do not. I use emacs keybindings | 20:45 |
| lifeless | I should man up and switch to tmux at some point I guess | 20:45 |
| lifeless | I just haven't enjoyed my experiences with tmux so far | 20:45 |
| mordred | because they work exceptionally well - unless you're using screen, in which case the world breaks | 20:45 |
| mordred | what I like about my weechat-in-tmux-in-byobu experience is that all of the keys do what I expect them to and I did not do extra config anywhere | 20:46 |
| lifeless | ok, so back to this openvswitch headf*ck | 20:46 |
| *** bauzas has joined #tripleo | 20:46 | |
| mordred | :) | 20:46 |
| lifeless | mordred: btw, you have mail (that work thread) | 20:46 |
| lifeless | mordred: we have confusion somewhere | 20:46 |
| mordred | lifeless: I'm sure we do | 20:46 |
| lifeless | plus I'm having to reboot the n-o-a every few hours on my undercloud or dhcp goes lalalalala | 20:49 |
| lifeless | sheese | 20:49 |
| SpamapS | lifeless: doesn't it just HUP dnsmasq a lot? | 20:50 |
| SpamapS | wait thats n-d-a | 20:50 |
| mordred | lifeless: responded | 20:50 |
| SpamapS | n-o-a runs ovs-vsctl all the time | 20:50 |
| *** tserong has quit IRC | 20:51 | |
| lifeless | SpamapS: yea, and its doing something that breaks the br-int -> tap glue | 20:51 |
| lifeless | nda HUPs dnsmasq, which has its own issue | 20:52 |
| lifeless | SpamapS: https://bugs.launchpad.net/neutron/+bug/1271344 | 20:52 |
| uvirtbot | Launchpad bug 1271344 in tripleo "neutron-dhcp-agent doesn't hand out leases for recently used addresses" [Critical,Triaged] | 20:52 |
| *** CaptTofu has quit IRC | 20:56 | |
| *** jrist has quit IRC | 20:58 | |
| lifeless | mordred: reresponded :P | 20:58 |
| SpamapS | ok.. back to Sunday | 21:00 |
| mordred | lifeless: I rereresponded - but I did so to Clint | 21:04 |
| lifeless | yah, I got the beep :P | 21:06 |
| lifeless | its very gentle, OWA. | 21:06 |
| lifeless | 'behp' | 21:06 |
| lifeless | oh wow, bad bad bad rules on the compute node vswitch | 21:10 |
| *** jrist has joined #tripleo | 21:10 | |
| lifeless | that was it | 21:12 |
| lifeless | sheeeese | 21:12 |
| lifeless | looks like the ovs agent isn't resyncing properly after rabbit disconnects | 21:13 |
| *** e0ne has quit IRC | 21:20 | |
| * lifeless stabs | 21:23 | |
| lifeless | overcloud-novacompute1-bxq4qdzw7bif.novalocal | 21:23 |
| *** tzumainn has joined #tripleo | 21:32 | |
| mordred | any redhat people aroudn who know where the heck haproxy is on a rhel install? | 21:42 |
| *** Perfectknoppsand has joined #tripleo | 21:42 | |
| lifeless | tzumainn probably does | 21:42 |
| lifeless | qemu slow | 21:43 |
| lifeless | the good news I have migration working, but don't think its live yet | 21:43 |
| lifeless | for all that its called 'live-migration' | 21:43 |
| *** e0ne has joined #tripleo | 21:51 | |
| *** CaptTofu has joined #tripleo | 21:53 | |
| mordred | lifeless: :) | 21:55 |
| *** noslzzp has joined #tripleo | 21:58 | |
| *** CaptTofu has quit IRC | 21:59 | |
| SpamapS | 2014-02-09 22:08:43,071.071 1484 TRACE nova.openstack.common.threadgroup NovaException: Baremetal node: 32 has no available physical interface for virtual interface c6192a37-b6d9-4434-8bd5-739a6513cb47 | 22:08 |
| lifeless | SpamapS: fun, new one on me | 22:09 |
| SpamapS | yeah... | 22:09 |
| lifeless | SpamapS: ci-overcloud ? | 22:09 |
| SpamapS | lifeless: looking now not sure | 22:09 |
| SpamapS | on cd-undercloud | 22:09 |
| SpamapS | nova-compute was showing 0 anything available | 22:09 |
| SpamapS | restarted it and it started spewing this trace | 22:09 |
| SpamapS | | 3198f460-ebd6-43ed-8e8b-4aa9142dd624 | testenv-testenv3-wn42v7iwboxr | ACTIVE | None | Running | ctlplane=10.10.16.152, 10.10.16.156 | | 22:10 |
| SpamapS | that's node 32 | 22:10 |
| *** cd-undercloud has joined #tripleo | 22:10 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 22:10 |
| *** cd-undercloud has quit IRC | 22:10 | |
| *** Perfectknoppsand has quit IRC | 22:12 | |
| SpamapS | weird weird weird | 22:13 |
| SpamapS | neutron port c6192a37-b6d9-4434-8bd5-739a6513cb47 has mac 78:e7:d1:21:66:76 | 22:13 |
| SpamapS | which is not for bm node 32, but for 4 | 22:13 |
| SpamapS | which is an unallocated machine | 22:13 |
| SpamapS | doh | 22:17 |
| *** hewbrocca has quit IRC | 22:17 | |
| lifeless | misregistered node ? | 22:17 |
| SpamapS | two ports for 3198f460-ebd6-43ed-8e8b-4aa9142dd624 | 22:17 |
| SpamapS | I already pasted it even | 22:17 |
| SpamapS | 10.10.16.152 and 10.10.16.156 | 22:18 |
| SpamapS | No idea why nova wants to plug node 32 into that | 22:18 |
| *** e0ne has quit IRC | 22:20 | |
| SpamapS | 2014-02-09 22:23:17,854.854 32448 WARNING nova.compute.manager [req-d6d585cc-eeb0-470a-b574-ab6d316bfd48 None None] Found 35 in the database and 27 on the hypervisor. | 22:23 |
| lifeless | SpamapS: I mean in nova baremetal's registry | 22:23 |
| SpamapS | lifeless: no the macs are right I re-checked against the info text | 22:23 |
| SpamapS | lifeless: and that node is working fine. nova_bm just tried to plug that port into the wrong instance from what I can tell. | 22:24 |
| SpamapS | 2014-02-09 22:29:51,258.258 32448 INFO nova.compute.resource_tracker [-] Compute_service record updated for undercloud:fbe3a166-af9e-4caa-9008-09988efec77c | 22:30 |
| SpamapS | 2014-02-09 22:29:51,259.259 32448 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources | 22:30 |
| SpamapS | 2014-02-09 22:29:51,307.307 32448 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 0 | 22:30 |
| SpamapS | lifeless: seems like something else is broken.. | 22:30 |
| SpamapS | lifeless: no baremetal resources are available | 22:31 |
| SpamapS | 2014-02-09 22:29:51,425.425 32448 ERROR nova.network.neutronv2.api [-] *************************************************************************************************************foo {'tenant_id': u'4956c533154c476799c688eda7ed65ab', 'device_id': '6bd2573d-1e19-4baa-96ff-0ca240e1a074'} | 22:31 |
| SpamapS | also .. that.. ?? | 22:31 |
| lifeless | the foo is debug cruft I added, ignore it | 22:31 |
| SpamapS | ah ok | 22:31 |
| SpamapS | lifeless: so I deleted that port from neutron, as the machine that mac is associated with is unallocated. That made the weird trace go away. | 22:32 |
| lifeless | right | 22:33 |
| lifeless | only one interface row in bm_interfaces. weird. | 22:34 |
| lifeless | SpamapS: you did see that I booted all the undercloud machines | 22:34 |
| lifeless | SpamapS: to find broken ones | 22:34 |
| lifeless | right ? | 22:34 |
| SpamapS | Oh so there really are no available nodes? | 22:34 |
| lifeless | [nova boot --num=instances=.... | 22:34 |
| lifeless | well | 22:34 |
| SpamapS | I totally did not see that | 22:35 |
| lifeless | there should be enough for the overcloud still | 22:35 |
| lifeless | check nova list:) | 22:35 |
| SpamapS | we should drop retries to 0 and do that | 22:35 |
| SpamapS | but yeah we can use nova-compute.log to find the borken ones | 22:35 |
| SpamapS | lifeless: ok so anyway, overcloud is still getting errors | 22:37 |
| SpamapS | lifeless: and the one I booted got an error | 22:37 |
| lifeless | what was the error ? | 22:37 |
| SpamapS | no available hosts | 22:37 |
| SpamapS | but immediately | 22:37 |
| SpamapS | not after pxe timeouts | 22:37 |
| lifeless | ok so | 22:37 |
| lifeless | nova hypervisor-stats | 22:37 |
| lifeless | | count | 35 | | 22:37 |
| lifeless | | running_vms | 35 | | 22:37 |
| SpamapS | yeah | 22:38 |
| lifeless | nova baremetal-node-list | grep undercloud | wc -l | 22:38 |
| lifeless | 35 | 22:38 |
| lifeless | we have 5 marked bad | 22:38 |
| lifeless | nova baremetal-node-list | grep -v undercloud | 22:38 |
| SpamapS | ok | 22:38 |
| SpamapS | so, do I delete two of the ACTIVE find brokens, and the overcloud, and then try the stack creation again? | 22:39 |
| lifeless | the overcloud is 3 of the 35 | 22:39 |
| SpamapS | ERROR,ERROR | 22:39 |
| SpamapS | 1 active only | 22:39 |
| lifeless | nova list| grep ACTIVE | wc -l | 22:39 |
| lifeless | 27 | 22:39 |
| lifeless | nova list| grep None | wc -l | 22:40 |
| lifeless | 35 | 22:40 |
| lifeless | the ERRORS count against a machine it would seem | 22:40 |
| SpamapS | errrrrrrrr | 22:40 |
| lifeless | now, whether thats a bug or not, dunno. | 22:40 |
| lifeless | But | 22:40 |
| *** CaptTofu has joined #tripleo | 22:41 | |
| lifeless | Iw ould try this - delete an ERROR findbroken | 22:41 |
| lifeless | and try booting one machine with nova boot | 22:41 |
| lifeless | (once nova hypervisor-stats shows running vms=34 | 22:41 |
| lifeless | SpamapS: whats your goal - make overcloud work ? | 22:42 |
| SpamapS | TIL 'nova hypervisor-stats' | 22:42 |
| SpamapS | lifeless: yes | 22:42 |
| lifeless | ok | 22:42 |
| lifeless | so delete the overcloud stack | 22:42 |
| lifeless | and two of 'findbroken ACTIVE' | 22:42 |
| lifeless | then boot --num-instance 5 | 22:42 |
| lifeless | that should flush out the two broken ones from the overcloud | 22:42 |
| SpamapS | thats what I said, I'm setting booty traps | 22:43 |
| lifeless | then delete 3 findbroken ACTIVE | 22:43 |
| lifeless | then run tripleo-cd again | 22:43 |
| lifeless | yeah with you now; was being slow | 22:43 |
| SpamapS | lifeless: freecloud0016 looks like not getting PXE | 22:48 |
| lifeless | SpamapS: mac is right? I find tcpdump on the undercloud br-ctlplane is useful to see if the machines DHCP gets out | 22:49 |
| *** jackrabbit has joined #tripleo | 22:50 | |
| *** jackrabbit has quit IRC | 22:50 | |
| lifeless | stab | 22:51 |
| lifeless | 2014-02-09 21:51:20.327 3459 ERROR nova.virt.disk.mount.nbd [req-288461c9-9645-47f4-9e62-da5355f0ffa2 554875e48592436e96725c48b070d737 f2dd987b34584852b6a824d32d290a1c] nbd module not loaded | 22:51 |
| lifeless | its not meant to be -trying- | 22:51 |
| SpamapS | noooo | 22:51 |
| SpamapS | nbd: http://memecrunch.com/meme/26FC/why-won-t-you-die/image.png | 22:51 |
| lifeless | def _allocate_nbd(self): | 22:52 |
| lifeless | raise Exception("FOAD") | 22:52 |
| lifeless | cloud-init-nonet[73.79]: waiting 120 seconds for network device | 22:52 |
| lifeless | sheese | 22:52 |
| SpamapS | no I was wrong it is booing into deploy ramdisk | 22:53 |
| SpamapS | booting even | 22:53 |
| SpamapS | lifeless: it used to wait __forever__ | 22:54 |
| SpamapS | had a bit of a knock down drag out (but always friendly) with smoser over that | 22:54 |
| lifeless | SpamapS: my sheese wasn't at cloud-init | 22:54 |
| SpamapS | oh | 22:54 |
| lifeless | it was at this qemu cloud flakiness on networking | 22:54 |
| lifeless | which I'm slowly debugging | 22:55 |
| SpamapS | OH hahahaha I totally missed the FOAD | 22:55 |
| SpamapS | hmmm... boot into deploy RD shows it just stalling while booting cpus | 22:55 |
| SpamapS | so might be a bad CPU | 22:55 |
| lifeless | ahhhh | 22:55 |
| lifeless | why can't we have nice things? | 22:55 |
| SpamapS | [ 0.591937] smpboot: Booting Node 0, Processors #22 | 22:55 |
| lifeless | so, mark bad? | 22:56 |
| SpamapS | [ 0.578597] smpboot: Booting Node 1, Processors #21 OK | 22:56 |
| SpamapS | Yeah and lets get a ticket open | 22:56 |
| lifeless | for tht I'm opening a trello card, recording in -tab.txt and changing the service host in bm_nodes. | 22:56 |
| lifeless | and pinging someone that has cycles to liase with -ops | 22:56 |
| lifeless | e.g. Ng (though I must read his docs now!) | 22:56 |
| SpamapS | Yeah I think he did try to teach us all | 22:58 |
| SpamapS | lifeless: I'm being pulled back to sunday stuff.. | 22:58 |
| SpamapS | freecloud0016 .. bad... confirmed | 22:58 |
| SpamapS | wait no | 22:59 |
| SpamapS | textcons may just be confused | 22:59 |
| SpamapS | because it is pingable | 22:59 |
| lifeless | heh | 23:01 |
| SpamapS | weeeiirrdd | 23:01 |
| SpamapS | it worked | 23:02 |
| SpamapS | so.... | 23:02 |
| SpamapS | -> Sunday | 23:02 |
| SpamapS | lifeless: test-clint1 is totally deletable | 23:02 |
| SpamapS | I may leave it a bit just bcuz | 23:02 |
| lifeless | ssh on qemu sllooww | 23:05 |
| *** e0ne has joined #tripleo | 23:12 | |
| *** CaptTofu has quit IRC | 23:18 | |
| lifeless | omg | 23:21 |
| lifeless | nbd needs such heavy boots to avoid | 23:21 |
| lifeless | if size: | 23:22 |
| lifeless | disk.extend(target, size, use_cow=True) | 23:22 |
| lifeless | uses qemu-nbd | 23:22 |
| *** noslzzp has quit IRC | 23:22 | |
| *** noslzzp has joined #tripleo | 23:24 | |
| *** cd-undercloud has joined #tripleo | 23:33 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 23:33 |
| *** cd-undercloud has quit IRC | 23:33 | |
| *** edmund has joined #tripleo | 23:37 | |
| *** jeremydei has quit IRC | 23:39 | |
| *** jeremydei has joined #tripleo | 23:42 | |
| greghaynes | lifeless: re the neutron rootwrap thing: the full venv path idea works for me | 23:48 |
| lifeless | greghaynes: btw did you spot the typo in l3.filters? | 23:49 |
| lifeless | greghaynes: I submitted a patch for it | 23:49 |
| greghaynes | oh, no | 23:49 |
| lifeless | neuton | 23:50 |
| lifeless | greghaynes: that might possibly be the cause, I haven't checked though | 23:50 |
| greghaynes | ah, ill check it out | 23:50 |
| lifeless | SpamapS: whats this about: cloud-init-nonet[70.26]: waiting 120 seconds for network device | 23:51 |
| greghaynes | if it works with that would be kind of.. odd | 23:51 |
| lifeless | cloud-init-nonet[78.65]: static networking is now up | 23:51 |
| lifeless | SpamapS: the instance has dhcp configured... | 23:51 |
| lifeless | and wtf | 23:52 |
| lifeless | live-migrate reboots the instance. | 23:52 |
| lifeless | I don't know *where* to start | 23:52 |
| lifeless | oh | 23:53 |
| lifeless | --block-migration does not do whwat you might think | 23:53 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
