Sunday, 2014-03-30

« Saturday, 2014-03-29 Index Monday, 2014-03-31 »
*** sballe_ has quit IRC00:35
*** nati_ueno has quit IRC00:35
openstackgerritA change was merged to openstack-infra/tripleo-ci: Remove unneeded setup from toci_gate_test.sh  https://review.openstack.org/8179601:13
*** CaptTofu has joined #tripleo01:46
*** CaptTofu has quit IRC02:14
*** julim has joined #tripleo02:23
*** julim has quit IRC02:28
*** CaptTofu has joined #tripleo02:32
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/129048803:00
uvirtbotLaunchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete]03:00
*** untriaged-bot has quit IRC03:00
*** eghobo has joined #tripleo03:14
*** CaptTofu has quit IRC03:16
*** CaptTofu has joined #tripleo03:17
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Add some clarity to the first-time user experience  https://review.openstack.org/8329403:20
*** CaptTofu has quit IRC03:21
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Add some clarity to the first-time user experience  https://review.openstack.org/8329403:28
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Standardise location of environment password files.  https://review.openstack.org/8325003:54
*** vkozhukalov has joined #tripleo04:36
*** julim has joined #tripleo05:24
*** julim has quit IRC05:28
*** tchaypo has quit IRC05:46
*** tchaypo has joined #tripleo05:47
*** vkozhukalov has left #tripleo06:27
*** yamahata has quit IRC06:37
*** yamahata has joined #tripleo06:39
*** eghobo has quit IRC06:45
*** zigo has quit IRC06:56
*** zigo has joined #tripleo06:58
*** akuznetsov has joined #tripleo07:20
*** julim has joined #tripleo07:25
*** julim has quit IRC07:33
*** akuznetsov has quit IRC07:36
*** jang1 has joined #tripleo08:10
*** akuznetsov has joined #tripleo08:11
*** akuznetsov has quit IRC08:12
*** akuznetsov has joined #tripleo08:50
*** slagle has quit IRC08:58
*** slagle has joined #tripleo08:59
*** untriaged-bot has joined #tripleo09:00
untriaged-botUntriaged bugs so far:09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/129048809:00
uvirtbotLaunchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete]09:00
*** untriaged-bot has quit IRC09:00
*** jang1 has quit IRC09:03
*** yamahata has quit IRC09:11
*** akuznetsov has quit IRC09:16
*** akuznetsov has joined #tripleo09:19
*** jang1 has joined #tripleo09:19
*** CaptTofu has joined #tripleo11:06
*** CaptTofu has quit IRC11:14
*** CaptTofu_ has joined #tripleo11:15
*** CaptTofu_ has quit IRC11:26
*** Guest77682 is now known as ohadlevy11:29
*** ohadlevy has joined #tripleo11:29
*** CaptTofu has joined #tripleo11:53
*** CaptTofu has quit IRC12:04
openstackgerritA change was merged to openstack-infra/tripleo-ci: Allow ironic to use ssh commands  https://review.openstack.org/8390612:07
*** shardy has quit IRC12:13
*** killer_prince has joined #tripleo12:55
*** Shrews has joined #tripleo13:19
*** rlandy has joined #tripleo14:45
*** rlandy has quit IRC14:51
*** untriaged-bot has joined #tripleo15:00
untriaged-botUntriaged bugs so far:15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/129048815:00
*** untriaged-bot has quit IRC15:00
*** akuznetsov has quit IRC15:00
uvirtbotLaunchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete]15:00
*** morazi has joined #tripleo15:48
*** jroll has quit IRC16:10
*** slagle has quit IRC16:10
*** jroll has joined #tripleo16:14
*** yamahata has joined #tripleo16:15
*** eghobo has joined #tripleo16:28
*** eghobo has quit IRC16:55
*** nati_ueno has joined #tripleo16:55
*** jang1 has quit IRC16:58
*** jang1 has joined #tripleo16:59
*** sdake_ has quit IRC17:13
*** akuznetsov has joined #tripleo17:23
*** nati_ueno has quit IRC17:25
*** e0ne has joined #tripleo17:32
*** rlandy has joined #tripleo17:39
*** e0ne has quit IRC17:51
*** e0ne has joined #tripleo18:06
*** beekneemech has quit IRC18:18
*** panda has quit IRC18:25
*** panda has joined #tripleo18:25
lifelessStevenK: IIRC ai saw you and yang talking about os-cloud-config - this https://etherpad.openstack.org/p/tuskar_init_after_create - was the initial request about it18:58
*** e0ne has quit IRC18:58
*** marun has quit IRC18:58
*** e0ne has joined #tripleo18:59
*** akuznetsov has quit IRC19:10
*** flashgordon is now known as jogo19:15
lifelesswow19:52
lifelessneutron port-list | awk '/te_testenv/ {print $2}'19:52
lifeless{"error": {"message": "An unexpected error prevented the server from fulfilling your request. (OperationalError) (2002, \"Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)\") None None", "code": 500, "title": "Internal Server Error"}}19:52
lifelesson ci-overcloud HP19:52
lifeless/dev/sda3       470G   30G  420G   7% /19:53
lifeless/dev/sda1       985G   51G  884G   6% /mnt19:53
lifelessmysql.err is empty19:53
lifelessah, wrong logs, we should fix that19:54
lifeless140330 19:53:32  InnoDB: Operating system error number 0 in a file operation.19:55
lifelessInnoDB: Error number 0 means 'Success'.19:55
lifelesswtf19:55
SpamapSahhahaha19:55
SpamapSnice19:55
lifelessSpamapS: 'help'19:55
SpamapSlifeless: is mysql broken?19:55
lifelessyes19:55
SpamapSlifeless: did we have an unexpected crash?19:56
lifelessSpamapS: http://paste.openstack.org/show/74624/19:56
lifeless# uptime19:56
lifeless 19:56:18 up 26 days, 9 min,  3 users,  load average: 0.97, 0.95, 1.0019:56
SpamapSof mysqld?19:57
SpamapSlooks like libaio problems19:57
lifelessSpamapS: I don't know, I'm not a mysql expert ;)19:57
lifelessSpamapS: can you login as well and look ?19:57
*** sdake_ has joined #tripleo19:57
SpamapSInnoDB: File operation call: 'Linux aio'.19:57
SpamapSlifeless: yeah looking now19:58
lifelesshttp://bugs.mysql.com/bug.php?id=54430 perhaps ?19:58
SpamapSlifeless: looks like an error happening over and over19:59
*** e0ne has quit IRC19:59
SpamapS140330  4:03:20 [Note] Recovering after a crash using /mnt/state/var/lib/mysql/mysql-bin19:59
SpamapS140330  4:03:21  InnoDB: Operating system error number 0 in a file operation.19:59
lifelessSpamapS: indeed!19:59
SpamapS[2247207.917782] Sense Key : Medium Error [current]20:00
lifelessurgh20:00
SpamapSlifeless: so the error code looks to be 0 because it is AIO20:00
lifelessbad disk ?20:00
SpamapScould be20:00
SpamapSwe're not RAID1 at least?20:00
lifelesswe're RAID120:01
lifeless2x2TB disks in these boxes20:01
lifelesslsblk shows no sdb, so raid120:01
*** e0ne has joined #tripleo20:01
lifelessdriver fail perhaps? try a reboot?20:02
SpamapS[2246403.081090] end_request: critical target error, dev sda, sector 721112020:03
SpamapS120:03
*** lifeless changes topic to "FIREDRILL: ci-overcloud mysql down, disk errors | tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt"20:03
lifelesslooking for the first occurence20:06
lifelessSpamapS: http://paste.openstack.org/show/74625/20:07
lifeless06:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)20:09
lifeless is the raid card I think20:09
lifelessSpamapS: thoughts?20:10
*** killer_prince has quit IRC20:11
*** e0ne has quit IRC20:11
SpamapSlifeless: we can run diagnostics on it from the ilo web interface IIRC20:11
*** e0ne has joined #tripleo20:12
*** nati_ueno has joined #tripleo20:13
*** lazy_prince has joined #tripleo20:13
*** lazy_prince is now known as killer_prince20:14
SpamapSlifeless: looking at ilo now20:21
SpamapShm no ...20:23
SpamapSlifeless: installing hpacucli .. should let us interrogate the raid controller20:27
lifelesspresumably its a bit error on one disk20:27
lifelessjust can't tell which disk20:27
lifelessso it shows bad to the kernel20:27
lifelesswtf20:33
lifeless 3310 ?        S      0:05 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new20:33
lifeless/usr/sbin/hpacucli: line 18: /opt/compaq/hpacucli/bld/.hpacucli: No such file or directory20:33
SpamapStyeah20:38
SpamapSlifeless: linked against a libstdc++ that Ubuntu doesn't have (this is aliened from the RPMS)20:38
SpamapSlifeless: can't find hpacucli for Debian or Ubuntu20:39
SpamapSwhy you would distribute a closed source thing w/ shared linking.. I don't know20:39
SpamapSlifeless: I'm at a loss, and have to go for family time stuff20:40
SpamapSlifeless: I do think that we need to be careful about rebooting before we know whats up20:40
lifelessSpamapS: there is a BIOS configurator for raid20:41
SpamapSyeah20:41
lifelessSpamapS: here's my decision tree20:41
lifelesseither a) its recoverable or b) its not20:41
lifelesswe have no backs20:41
lifelessif b) we rebuild on a new host.20:41
lifelessif a) the path doesn't matter too much20:41
lifelesswe've captured the sense code in a pastebin20:41
lifelesss/backs/backups/20:42
SpamapSSure sounds good to me.20:42
SpamapSWill check back in in about 2 hours.20:42
lifelessack20:42
lifelessmore [10401347.008399] hpsa 0000:06:00.0: cmd_alloc returned NULL!20:42
lifelessmeh, wrong host20:43
*** nati_ueno has quit IRC20:59
*** cody-somerville has quit IRC21:03
*** e0ne has quit IRC21:08
*** eghobo has joined #tripleo21:11
tchaypomorning21:26
lifelessoh hai21:31
*** e0ne has joined #tripleo21:39
*** e0ne_ has joined #tripleo21:40
*** e0ne has quit IRC21:40
tchaypoStill no action on https://bugs.launchpad.net/tripleo/+bug/1290486 from the neutron end21:41
uvirtbotLaunchpad bug 1290486 in tripleo "neutron-openvswitch-agent must be restarted after ovsdb-server failure in order to pass traffic" [Critical,In progress]21:41
tchaypoI think it's time to jump into irc and start asking what we need to do to get someone to take a look21:42
tchaypoor start taking a look myself, I guess21:42
*** jang1 has quit IRC21:42
*** e0ne_ has quit IRC21:44
*** cody-somerville has joined #tripleo21:57
*** cody-somerville has joined #tripleo21:57
*** lifeless has quit IRC21:59
*** lifeless has joined #tripleo21:59
*** e0ne has joined #tripleo22:01
*** e0ne has quit IRC22:03
lifelessSpamapS: finally got into the raid bios22:20
lifelessSpamapS: only has one drive22:20
mordredlifeless: that doesn't sound much like RAID22:28
mordredlifeless: that soudns like D22:28
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Standardise location of environment password/rc files.  https://review.openstack.org/8325022:30
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Standardise location of environment password/rc files.  https://review.openstack.org/8325022:31
lifelessmordred: yeah22:34
lifelessmordred: server 'monty005'22:34
tchaypomordred: possibly "ID"22:35
mordredlifeless: are we naming all of our servers after me now/22:37
tchaypomordred: no no, it's named after monty from mysql22:40
tchaypo*the* monty, not the *other* monty22:40
lifelessmordred: this is one of the ones you had reserved22:49
lifelesswhich appear to be uniformly messed up22:49
mordredlifeless: AWESOME22:51
lifelesse,g, less ram, and as of today less disks...22:54
lifelesshah, or - perhaps, nova bm was forcing the machine off. STAB22:57
*** e0ne has joined #tripleo23:01
*** e0ne has quit IRC23:05
cody-somervillelifeless: What machine is this?23:07
lifelesscody-somerville: monty00523:11
cody-somervillelifeless: Where did "monty005" come from? Where does it live?23:12
lifelesscody-somerville: where are we going with these questions, we've a cloud down and I'm focused on that23:12
lifelesscody-somerville: if you're interested in joining tripleo-cd-admins, cool - but now isn't the best time23:13
cody-somervillelifeless: Apologies. I think we're good. Thought you were provisioning something new based on Monty's comment. Wanted to make sure you weren't talking about machines from Vlad.23:14
lifelesscody-somerville: nope23:14
*** ci-overcloud has joined #tripleo23:48
ci-overcloud************** ci-overcloud complete status=128 ************23:48
*** ci-overcloud has quit IRC23:48
*** ci-overcloud has joined #tripleo23:53
ci-overcloud************** ci-overcloud complete status=130 ************23:53
*** ci-overcloud has quit IRC23:53
*** ci-overcloud has joined #tripleo23:55
ci-overcloud************** ci-overcloud complete status=7 ************23:55
*** ci-overcloud has quit IRC23:55
lifelesswhy is packet forwarding so hard for these machines23:56
SpamapSlifeless: greedy machines?23:57
SpamapSlifeless: how's it going? disk failed for sure?23:57
lifelessSpamapS: CPU/MB I think23:58
lifelessSpamapS: see trello23:58
lifelessSpamapS: just trying to build a new overcloud image to deploy23:58
« Saturday, 2014-03-29 Index Monday, 2014-03-31 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!