| tchaypo | timestamp on the message from heat event-list is 2014-09-22T08:35:08Z | 00:01 |
|---|---|---|
| tchaypo | which is before the log shows the new value being read.... because I forgot to delete it. | 00:01 |
| tchaypo | well done james. | 00:01 |
| *** dsneddon has joined #tripleo | 00:13 | |
| *** lucas-dinner has quit IRC | 00:17 | |
| *** cwolferh_ has quit IRC | 00:19 | |
| *** nosnos has joined #tripleo | 00:24 | |
| *** ChuckC_ has joined #tripleo | 00:29 | |
| *** noslzzp has quit IRC | 00:29 | |
| *** dvossel has quit IRC | 00:30 | |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Updated tuskar-ui install to setup roles and various parameters https://review.openstack.org/123503 | 00:32 |
| *** ChuckC has joined #tripleo | 00:33 | |
| *** ramishra has joined #tripleo | 00:33 | |
| *** ChuckC_ has quit IRC | 00:33 | |
| *** sseago has quit IRC | 00:36 | |
| *** Abner_Zulauf83 has quit IRC | 00:42 | |
| lifeless | tchaypo: where did you end up booking accom for paris? | 00:43 |
| tchaypo | le meridien | 00:43 |
| tchaypo | I didn't check too deeply, but stevenk told me it was cheaper than the options CWT were offering | 00:44 |
| *** Ada32 has joined #tripleo | 00:44 | |
| *** penick has joined #tripleo | 00:46 | |
| *** saurabhs has left #tripleo | 00:46 | |
| *** pradeep has joined #tripleo | 00:48 | |
| *** jang has quit IRC | 01:10 | |
| *** jang has joined #tripleo | 01:11 | |
| tchaypo | with the higher retry count I'm still not getting anything built. | 01:15 |
| tchaypo | I'm seeing lots of things in BUILD | spwaning | 01:16 |
| tchaypo | but then when I check back they're just in BUILD | - | 01:16 |
| tchaypo | and have no IP assigned any more.. and the IPs assigned are creeping up through the scope. | 01:16 |
| *** spzala has quit IRC | 01:19 | |
| stevebaker | tchaypo: that "attempts 3" is from heat, not nova | 01:21 |
| stevebaker | tchaypo: no, ignore me. heat.conf action_retry_limit defaults to 5, and that message is from nova | 01:23 |
| lifeless | tchaypo: thats expected | 01:25 |
| lifeless | tchaypo: its retrying | 01:25 |
| *** Ada32 has quit IRC | 01:25 | |
| lifeless | tchaypo: it allocates the ip before handing the instance to the compute process | 01:25 |
| lifeless | tchaypo: then that raises ane rror, deallocates the ip and hands it back to the conductor | 01:26 |
| lifeless | tchaypo: which reschedules,now onto a node hopefully free, and we rinse and repeat | 01:26 |
| *** panda has quit IRC | 01:26 | |
| lifeless | luncthime | 01:27 |
| *** panda has joined #tripleo | 01:27 | |
| tchaypo | that's a late lunch | 01:28 |
| tchaypo | it's nearly lunchtime for me too | 01:28 |
| tchaypo | still seeing this | 01:28 |
| tchaypo | Sep 24 01:27:53 undercloud-undercloud-meprf37xkzvy nova-compute: 2014-09-24 01:27:53.480 34864 TRACE nova.openstack.common.loopingcall InstanceDeployFailure: Failed to provision instance 9043289c-6463-4df4-af55-68854150328d: Failed to deploy. Error: Failed to set DHCP BOOT options for any port on node 1b8b7dec-d93b-483a-958f-354dbb4f7c94. | 01:28 |
| *** pradeep has quit IRC | 01:29 | |
| lifeless | ok so thats going to fail tings out | 01:29 |
| lifeless | we need to fix tht :) | 01:30 |
| lifeless | have you checked for time skew | 01:30 |
| lifeless | between your uc nodes | 01:30 |
| tchaypo | which seems to be because of http://paste.openstack.org/show/114775/ | 01:31 |
| tchaypo | time as far as I can tell both machines are within 1 second | 01:34 |
| tchaypo | the bastion host reports time in BST rather than UTC which puts it an hour out, but I can't see how that would matter | 01:35 |
| *** cwolferh_ has joined #tripleo | 01:35 | |
| tchaypo | not when it's explicitly reporting the timezone. Not unless keystoneclient ignores the timezone | 01:35 |
| tchaypo | https://bugs.launchpad.net/python-keystoneclient/+bug/1195924 shows that it did used to do silly things | 01:36 |
| uvirtbot | Launchpad bug 1195924 in python-keystoneclient "expiration check is using different timezone other than UTC" [Medium,Fix released] | 01:36 |
| *** sseago has joined #tripleo | 01:40 | |
| *** Antonetta37 has joined #tripleo | 01:42 | |
| tchaypo | okay, so I've just realised this is similar to what I've seen before - first build after restarting things gets 5 up, later builds get none. This one should have had its 5 successes by now | 01:50 |
| *** morazi has quit IRC | 01:50 | |
| lifeless | tchaypo: both machines? | 01:50 |
| lifeless | tchaypo: there aren't three? | 01:50 |
| tchaypo | so I restarted ironic-conductor to see if that would do whatever happens to make the first 5 work... but it seems that nova gets very sad if the ironic-conductor it's talking to goes away | 01:51 |
| tchaypo | this is still a non-ha undercloud | 01:51 |
| lifeless | when you restart the conductor all in-progress requests get discarded | 01:51 |
| lifeless | tchaypo: ah, ok. so one machine is all that matters | 01:51 |
| lifeless | I think | 01:51 |
| lifeless | so we're back to looking at the token verification fail | 01:52 |
| tchaypo | discarding in-progress requests makes sense, but now nova gives me a 500 when I try to get a nova-list | 01:52 |
| lifeless | what doth keystone say? | 01:52 |
| lifeless | 500 .. thats new, file bug | 01:52 |
| *** rhallisey has quit IRC | 01:57 | |
| *** Yanyanhu has joined #tripleo | 01:58 | |
| tchaypo | This as far as I can see the keystone logs only show "RBAC: Authorization granted" | 01:59 |
| tchaypo | not seeing anything being rejected | 01:59 |
| morganfainberg | tchaypo, that typically indicates unless *keystone* is throwing a 500 (it should have an exception) something is happening in nova. | 02:00 |
| morganfainberg | tchaypo, you can enable debug in keystone to dig in futher. | 02:00 |
| tchaypo | debug is enabled in keystone :) | 02:00 |
| tchaypo | the error in neutron is http://paste.openstack.org/show/114775/ | 02:01 |
| morganfainberg | tchaypo, hm. ok. | 02:01 |
| tchaypo | I notice that the error is coming from _cache_get which makes me wonder if neutron is having problems reading its cached token | 02:01 |
| morganfainberg | it could be. | 02:01 |
| tchaypo | just in case I mised something, let me grep for 500 in keystone's log | 02:02 |
| morganfainberg | thats an error in keystonemiddleware | 02:02 |
| morganfainberg | the .cache_get | 02:02 |
| tchaypo | yeah, not the keystone backend | 02:02 |
| tchaypo | which it doesn't seem to be talking to | 02:02 |
| morganfainberg | no .cache_get is called in all cases | 02:02 |
| morganfainberg | tchaypo, is this uuid or PKI tokens? | 02:03 |
| tchaypo | confirmed, no 500 or 4xx in the logs - except a few 400s two days ago | 02:05 |
| morganfainberg | tchaypo, legitimately this is cached as an invalid token: https://github.com/openstack/keystonemiddleware/blob/1.1.1/keystonemiddleware/auth_token.py#L1493 | 02:05 |
| morganfainberg | the exception is coming from line 1495 | 02:05 |
| tchaypo | I'm not sure, but I think it's uuid. keystone.conf doesn't have any pki setup that I can see, and my memory says that pki is still something we're working on | 02:06 |
| morganfainberg | tchaypo, ok, thats fine. just 2x checking caus ePKI is much more complex :P | 02:06 |
| *** pcrews has quit IRC | 02:06 | |
| tchaypo | I assume that if we were seeing pki, there should be something very visible in keystone.conf, right? | 02:07 |
| morganfainberg | tchaypoh, so at some point the token from Keystone is invalid, either it's a timeout (token has been around for longer than the token TTL) or the token has been invalidated | 02:07 |
| morganfainberg | tchaypo, if you look at the keystone config you'd see [token]/provider=keystone.token.provider.backends.[pki|pkiz].Token | 02:07 |
| tchaypo | what I'm actually seeing is keystone.token.backends.sql.Token | 02:08 |
| tchaypo | so that confirms it | 02:08 |
| morganfainberg | tchaypo, that is the driver | 02:08 |
| morganfainberg | not the provider | 02:08 |
| tchaypo | true. | 02:08 |
| *** ramishra has quit IRC | 02:08 | |
| tchaypo | but there is no provider listed that I can see | 02:08 |
| morganfainberg | tchaypo, hm. this i assume is a very new master keystone? | 02:09 |
| *** ramishra has joined #tripleo | 02:09 | |
| tchaypo | I would have built it a few days ago | 02:10 |
| morganfainberg | ok so that is def. UUID then | 02:10 |
| morganfainberg | https://github.com/openstack/keystone/blob/master/etc/keystone.conf.sample#L1534-L1538 | 02:10 |
| morganfainberg | to show the default | 02:11 |
| tchaypo | Yep. In this case the expiration period is set to 14400 for $reasons | 02:11 |
| tchaypo | sorry, something just twigged | 02:12 |
| tchaypo | if the _get_cache call can return a cached "this is an invalid token" answer, I should be able to search up the log and find the result that it cached, right? | 02:12 |
| morganfainberg | so that is telling maybe one of 4 things could be happening: 1) Token has been removed explicitly (DELETE called for the token itself), 2) auth_token middleware is doing something dumb (possible) and mis-marking the token as valid, 3) MySQL is mismatched on time from keystone server, 4) keystone is silently failing (unlikely) | 02:12 |
| tchaypo | The behaviour I see is that after restarting things (haven't narrowed down precisely what, but last time I only restarted heat-api and ironic-conductor so it should be one of those two) I'm able to create 5 nodes | 02:13 |
| tchaypo | after that, auth failures | 02:13 |
| morganfainberg | we don't have the logging debug level for auth_token on. or you'd see something like https://github.com/openstack/keystonemiddleware/blob/1.1.1/keystonemiddleware/auth_token.py#L1494 right above that error in the neutron log | 02:13 |
| tchaypo | always precisely 5 nodes, which suggests to me the token is expiring after 5 uses | 02:14 |
| morganfainberg | tchaypo, that really sounds like a clock mismatch... or if you're getting a new token each time, limited use trust with only 5 uses? | 02:14 |
| morganfainberg | tchaypo, OR something is explicitly revoking the token. | 02:14 |
| morganfainberg | tchaypo, in keystone are you seeing a DELETE call to /v3/auth/tokens or /v2.0/tokens ? | 02:15 |
| morganfainberg | can you take a look at the mysql instance and see how the time in mysql compares to the time on the server keystone is running on [in UTC]? | 02:17 |
| tchaypo | well here's something I haven't seen before | 02:17 |
| tchaypo | http://paste.openstack.org/show/114787/ | 02:17 |
| tchaypo | keystone returning 404s | 02:18 |
| morganfainberg | ok that means the token is invalid in a few ways | 02:18 |
| morganfainberg | possibly | 02:19 |
| morganfainberg | it expired, was revoked, user is no longer valid (suspect because you've probably created a new token and it works again with same creds) | 02:19 |
| morganfainberg | project was "deleted" (again same reason i'd say suspect) | 02:19 |
| tchaypo | not seeing any time skew | 02:20 |
| tchaypo | http://paste.openstack.org/show/114788/ | 02:20 |
| tchaypo | I seperately checked SELECT TIME_FORMAT( NOW() - UTC_TIMESTAMP(), '%H%i' ) AS tz_offset; | 02:20 |
| tchaypo | and it reports an offset of 0000 so it doesnt seem to be a TZ difference either | 02:21 |
| tchaypo | There's a lot of the 404s int he logs; it looks like neutron usually handles this by getting a new token (or got one already) | 02:21 |
| morganfainberg | it sounds like something is explicitly revoking the token. | 02:21 |
| morganfainberg | or.. the token id is getting mangled somehow ... | 02:22 |
| morganfainberg | in the keystone log if it's set to debug you should see all the route calls (e.g. GET, POST, DELETE) | 02:22 |
| *** yamahata has joined #tripleo | 02:23 | |
| morganfainberg | do you see any deletes to <uri path>/tokens ? | 02:23 |
| morganfainberg | or in the case of v2 token/<id> | 02:23 |
| tchaypo | Ah - http://paste.openstack.org/show/114775/ was specifically greeping for WARNING or TRACE | 02:24 |
| tchaypo | http://paste.openstack.org/show/114789/ has fuller context | 02:24 |
| * morganfainberg looks | 02:24 | |
| morganfainberg | sorry sometimes hard to debug looking just at logs like this, doin my best to help ya out | 02:25 |
| tchaypo | no problem :) | 02:25 |
| tchaypo | you certainly have a lot more domain-specific knowledge than I do | 02:25 |
| lifeless | one hopes | 02:25 |
| lifeless | morganfainberg: thanks :) | 02:25 |
| morganfainberg | tchaypo, is this just the neutron log or is this also keystone's log? | 02:26 |
| morganfainberg | this does look all pretty "normal" [i mean except failing unexpectedly] | 02:26 |
| *** ci-overcloud has joined #tripleo | 02:27 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 02:27 |
| *** ci-overcloud has quit IRC | 02:27 | |
| tchaypo | just neutron's logs | 02:28 |
| tchaypo | http://paste.openstack.org/show/114790/ is from "grep -i delete keystone.log", cut down ot just around that time | 02:28 |
| morganfainberg | well then huh | 02:28 |
| tchaypo | There's nothing until Sep 24 01:38:13 which is a bit later | 02:28 |
| morganfainberg | tchaypo, ooh wait a sec here. let me check something | 02:30 |
| * tchaypo waits | 02:30 | |
| morganfainberg | no looks sane, looks like you've deleted the user 6bf69434353740ea8b4ff161a5d0bff0 which we then do delete_user_tokens_callback for event identity user deleted for{'resource_info': u'6bf69434353740ea8b4ff161a5d0bff0'} | 02:31 |
| morganfainberg | so that user's tokens should be deleted | 02:31 |
| morganfainberg | but not user u'c3b15dfb0eaf4221b9826cb475d6875d which is the one who's tokens are failing for | 02:32 |
| morganfainberg | tchaypo, can you set routes.middleware=WARN to routes.middleware=DEBUG in keystone? [this is the default_log_levels option] | 02:34 |
| morganfainberg | tchaypo, i assume this is under Eventlet? | 02:35 |
| morganfainberg | tchaypo (keystone) | 02:35 |
| *** sdake_ has joined #tripleo | 02:35 | |
| morganfainberg | tchaypo, hopefully that can show us if something is saying HEY revoke my token | 02:36 |
| morganfainberg | obviously we'd need to try it again and wait for it to fail. | 02:38 |
| *** Antonetta37 has quit IRC | 02:38 | |
| tchaypo | I'm guessing I'll need to restart keystone to pick that up? | 02:39 |
| morganfainberg | yeah unfortunately | 02:40 |
| tchaypo | this just goes in the DEFAULT section? | 02:40 |
| morganfainberg | tchaypo, yeah let me get you the full option set | 02:40 |
| morganfainberg | sec | 02:40 |
| tchaypo | if I'm reading it correctly, u'c3b15dfb0eaf4221b9826cb475d6875d' deleted user '4ff652c8663b4ba1ad754d228fbd65f4' at 01:38 | 02:42 |
| tchaypo | that | 02:42 |
| tchaypo | s 8 minutes after the failure in http://paste.openstack.org/show/114789/ | 02:42 |
| *** Alena66 has joined #tripleo | 02:42 | |
| tchaypo | and *let me check* | 02:43 |
| morganfainberg | yeah. | 02:43 |
| morganfainberg | and actually you should see logs from like wsgi.server | 02:43 |
| morganfainberg | with things like POST <uri> etc in the keystone log | 02:43 |
| morganfainberg | you shouldn't need to set routes.middleware to debug | 02:43 |
| morganfainberg | tchaypo, something like 34514 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [24/Sep/2014 02:43:52] "POST /v3/auth/tokens HTTP/1.1" 201 6835 0.23655 | 02:45 |
| tchaypo | If I'm reading http://paste.openstack.org/show/114789/ correctly, the "Received request from user: c3b15dfb0eaf4221b9826cb475d6875d" line at the bottom shows the user that initiated the request that failed, right? | 02:45 |
| morganfainberg | tchaypo, reading now | 02:45 |
| tchaypo | but that's the same user id that 8 minutes later was able to delete 4ff652c8663b4ba1ad754d228fbd65f4 - which suggests that the user was still valid at 01:30 | 02:46 |
| morganfainberg | tchaypo, correct | 02:46 |
| morganfainberg | tchaypo, it means either new token was accquired or for some reason the old token "worked" | 02:46 |
| openstackgerrit | Motohiro Otsuka (yuanying) proposed a change to openstack/os-cloud-config: Set iboot_relay_id and iboot_port https://review.openstack.org/122042 | 02:47 |
| morganfainberg | ugh, /me wishes we had audit ids for these tokens in all cases | 02:47 |
| StevenK | yuanying: Is that change just a rebase? | 02:48 |
| tchaypo | uwsgi logs | 02:49 |
| tchaypo | http://paste.openstack.org/show/114796/ | 02:49 |
| morganfainberg | ah ok | 02:49 |
| morganfainberg | yeah that looks clean, no DELETE calls | 02:50 |
| tchaypo | lots of GET and POST, no DELETEs that second. | 02:50 |
| morganfainberg | tchaypo, also uwsgi, awesome i'm going to need to steal that and get it documented in keystone (and tested) as a valid deployment method | 02:50 |
| morganfainberg | you know.. "hey guys you can do this" | 02:51 |
| tchaypo | but http://paste.openstack.org/show/114790/ shows there was nothing for quite a while before that or for 8 minutes afterwards | 02:51 |
| lifeless | we're using uwsgi? huh | 02:51 |
| tchaypo | did i say uwsgi? | 02:51 |
| morganfainberg | lifeless, aparently, and it's on my list of "We need to make sure we support it for keystone" | 02:51 |
| tchaypo | i meant wsgi | 02:51 |
| tchaypo | as in eventlet.wsgi | 02:51 |
| morganfainberg | tchaypo, oh darn it | 02:51 |
| lifeless | hahaha | 02:51 |
| morganfainberg | i was all excited! | 02:51 |
| tchaypo | apparently I can't type wsgi without prepending u | 02:51 |
| lifeless | uwsgi is not wsgi | 02:51 |
| morganfainberg | ok so the only thing that i could see this being is if the user being deleted is somehow involved with the whole stack in a trust? | 02:52 |
| morganfainberg | i don't *think* that is the case. | 02:52 |
| morganfainberg | i saw trust_id = None in the keystone debugs earlier | 02:52 |
| lifeless | morganfainberg: would you like to get handson? | 02:53 |
| morganfainberg | lifeless, might be easier. though ... i try *very* hard to not get my hands into servers i don't want access to :P | 02:53 |
| morganfainberg | lifeless, yeah probably best to take a gander directly | 02:53 |
| lifeless | morganfainberg: so if you do, I'm ok with it; | 02:53 |
| tchaypo | sorry, phonecall | 02:54 |
| lifeless | tchaypo: can you add morgans key to the account you log in on hp2 with, and then do a shared screen session (knowledge transfer) | 02:54 |
| lifeless | after your call :P | 02:54 |
| morganfainberg | cool. | 02:55 |
| morganfainberg | do i need to be on VPN for that? | 02:55 |
| morganfainberg | lifeless, ? | 02:55 |
| lifeless | morganfainberg: npoe | 02:55 |
| morganfainberg | lifeless, phew, easier if i don't need to get that wired up too | 02:55 |
| morganfainberg | and once done i'm totally asking for my key to be removed, i work hard to not have access to things :) | 02:56 |
| lifeless | sure sure | 02:58 |
| lifeless | removing it is in cron in fact | 02:58 |
| *** untriaged-bot has joined #tripleo | 03:00 | |
| untriaged-bot | Untriaged bugs so far: | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1373063 | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 03:00 |
| uvirtbot | Launchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 03:00 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 03:00 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 03:00 |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 03:00 |
| *** untriaged-bot has quit IRC | 03:00 | |
| morganfainberg | lifeless, ah cool | 03:02 |
| morganfainberg | lifeless, i assume ssh-key is what you'll need (cc tchaypo) ? | 03:04 |
| *** sdake_ has quit IRC | 03:05 | |
| *** wendar has quit IRC | 03:07 | |
| *** wendar has joined #tripleo | 03:08 | |
| lifeless | yay | 03:11 |
| lifeless | yah | 03:11 |
| tchaypo | back | 03:12 |
| tchaypo | morganfainberg: indeed. | 03:12 |
| tchaypo | The thing I was wanting to try is to kick off the stack build again now that I've restarted ironic-conductor and see if I start getting a few successes | 03:14 |
| tchaypo | and hopefully catch the logs as it transitions from those 5 successes to consistent failure | 03:14 |
| morganfainberg | ah. | 03:16 |
| morganfainberg | ok | 03:16 |
| morganfainberg | http://paste.openstack.org/show/114797/ | 03:16 |
| *** cwolferh_ has quit IRC | 03:19 | |
| *** penick has quit IRC | 03:19 | |
| tchaypo | try stack@bastion.hp2.tripleo.org | 03:21 |
| tchaypo | are you more familiar with tmux or screen? | 03:23 |
| morganfainberg | screen | 03:25 |
| morganfainberg | yeah logged in | 03:25 |
| tchaypo | let me switch form tmux to screen | 03:26 |
| morganfainberg | k | 03:26 |
| morganfainberg | yeah haven't used tmux at this point just never used it. | 03:26 |
| clarkb | morganfainberg: just ^b instead of ^a | 03:27 |
| tchaypo | I map mine to M-e but that's a different story | 03:28 |
| tchaypo | on the bastion it's ^b | 03:28 |
| tchaypo | but I've switched to screen anyway - standard screen -x should get you in | 03:28 |
| morganfainberg | yeah there now | 03:28 |
| clarkb | I use screen simply because it is everywhere and peopel know how to use it and it is everywhere | 03:28 |
| morganfainberg | clarkb, ++ same reason i use vi instead of emacs... | 03:28 |
| tchaypo | i use tmux because vertical-split and mouse-resize and stored layouts and much win | 03:29 |
| clarkb | tmux does vertical splits | 03:29 |
| tchaypo | but only after using screen for years, so I'm comfortable in it | 03:29 |
| clarkb | wow much lag | 03:29 |
| clarkb | ya everyone says the vertical split is the reason to tmux | 03:29 |
| tchaypo | huh. I was about to say that screen died at 4.0 | 03:30 |
| tchaypo | but http://git.savannah.gnu.org/cgit/screen.git/log/ shows lots of recent activity including prep for 4.2.1 | 03:31 |
| tchaypo | anyway | 03:31 |
| morganfainberg | tchaypo, before i do anything you want to do stuff ? | 03:31 |
| clarkb | screen is still alive and I thin kit vertical splits properly now :) | 03:31 |
| morganfainberg | tchaypo, i do't want to interrupt anything you're working on | 03:31 |
| tchaypo | morganfainberg: I was going to suggest restarting heat and ironic, then I'll kick off a deploy and we can wait for failures to happen | 03:31 |
| morganfainberg | tchaypo ok sounds good go for it (since it's shared screen) | 03:31 |
| morganfainberg | sorry phone call just came in. so i'm looking can't type as much (family) | 03:32 |
| *** jrist has quit IRC | 03:33 | |
| tchaypo | while we're waiting | 03:34 |
| morganfainberg | hehe | 03:34 |
| tchaypo | I'll mentioned that I recently started using emacs, but only after almost 20 years of learning vim, so I'm now fairly comfortable in either | 03:34 |
| morganfainberg | ... waiting ... waiting .... | 03:35 |
| morganfainberg | ahh | 03:35 |
| tchaypo | and vim is more useful for the same rasons - it's everywhere, it's lightweight, everyone knows it | 03:35 |
| morganfainberg | yep | 03:35 |
| morganfainberg | emacs, great operating system if only it had a text editor | 03:35 |
| morganfainberg | >.> | 03:35 |
| tchaypo | yeah | 03:35 |
| morganfainberg | :P | 03:35 |
| tchaypo | the one thing i haven't figured out yet is how to replicate my setup on another machine | 03:36 |
| tchaypo | vim is easy, i have a git repo with submodules for all my plugins | 03:36 |
| tchaypo | emacs is not so simple | 03:36 |
| *** Alena66 has quit IRC | 03:39 | |
| *** jrist has joined #tripleo | 03:40 | |
| *** Chao_Yan has quit IRC | 03:42 | |
| *** Adriana46 has joined #tripleo | 03:48 | |
| *** ramishra has quit IRC | 03:51 | |
| *** pradeep has joined #tripleo | 03:55 | |
| morganfainberg | tchaypo, kinda slow sometimes. | 03:56 |
| morganfainberg | tchaypo, i'm going to go grab foo, its like 9pm and haven't had dinner | 03:56 |
| morganfainberg | tchaypo, i'll be back in ~15-20min if that works? | 03:57 |
| *** lazy_prince is now known as killer_prince | 04:00 | |
| tchaypo | Yeah, I need to grab food too | 04:00 |
| tchaypo | just took a look at the nova-conductor log, it seems like it's very busy sleeping for 60 seconds and not much else | 04:01 |
| tchaypo | which probably explains why the stack-delete is taking so long | 04:01 |
| morganfainberg | hehe, ok back in 20-30mins we'll pick it up again (worst case we can continue tomorrow) | 04:01 |
| *** Chao_Yan has joined #tripleo | 04:03 | |
| yuanying | StevenK: Sorry, I just AFK to lunch. That's different was simply rebase and commit message | 04:10 |
| *** ramishra has joined #tripleo | 04:11 | |
| *** Adriana46 has quit IRC | 04:18 | |
| lifeless | so thats about a week of backlog caught up on | 04:23 |
| *** rushiagr_away is now known as rushiagr | 04:23 | |
| tchaypo | wb | 04:23 |
| clarkb | and its only tuesday | 04:24 |
| clarkb | oh wait you are on the other side of that date line thing | 04:24 |
| tchaypo | I was convinced it was thursday until about 1pm | 04:25 |
| tchaypo | despite the 5am wakeup, which should have been a hint | 04:25 |
| *** nikunj2512 has joined #tripleo | 04:27 | |
| lifeless | I'm now a contributor to importlib2. yay | 04:30 |
| *** Chao_Yan has quit IRC | 04:34 | |
| *** killer_prince has quit IRC | 04:36 | |
| *** Chao_Yan has joined #tripleo | 04:37 | |
| xuhaiwei | lifeless, tchaypo: hi, I noticed that when deploying tripleo to baremetal machine, user can't decide which machine to be undercloud, and which one to be controller node, compute node. I think we should make users decide as they like, what do you think? | 04:38 |
| *** ramishra has quit IRC | 04:39 | |
| *** ramishra has joined #tripleo | 04:39 | |
| morganfainberg | tchaypo ok i'm back | 04:42 |
| morganfainberg | oh look delete failed | 04:42 |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Fix for ironic node creation https://review.openstack.org/123631 | 04:58 |
| *** pensu has joined #tripleo | 05:00 | |
| *** tzumainn has quit IRC | 05:09 | |
| *** tzumainn has joined #tripleo | 05:10 | |
| *** ramishra has quit IRC | 05:13 | |
| *** ramishra has joined #tripleo | 05:14 | |
| *** bvivek has joined #tripleo | 05:15 | |
| tchaypo | I'm back too | 05:15 |
| morganfainberg | tchaypo, wb! | 05:16 |
| tchaypo | well this is messy. | 05:17 |
| morganfainberg | tchaypo, wow that isn't pretty is it | 05:17 |
| tchaypo | My usual approach at this point is to rebuild the undercloud | 05:17 |
| morganfainberg | lol | 05:17 |
| lifeless | tchaypo: uhm, wat | 05:17 |
| tchaypo | BFI FTW ;) | 05:17 |
| lifeless | so upgrade and take a cattle mentality +1 | 05:18 |
| lifeless | but you aren't in ha mode at this layer yet | 05:18 |
| lifeless | so wat | 05:18 |
| *** pensu has quit IRC | 05:19 | |
| tchaypo | if this cloud was ha, restarting ironic-conductor (one hopes) wouldn't have brought the stack-create to its knees | 05:19 |
| tchaypo | I wonder if i can delete a failed-to-delete stack | 05:19 |
| morganfainberg | hehe | 05:20 |
| *** pensu has joined #tripleo | 05:23 | |
| morganfainberg | doesn't exactly look promising | 05:23 |
| tchaypo | nope. | 05:24 |
| tchaypo | I'm wondering if some of the debug logging is slowing things down to the point that it's a problem | 05:24 |
| morganfainberg | it might be :( | 05:24 |
| tchaypo | ah, there we go, delete_failed rather than delete_in_progress | 05:25 |
| *** ramishra has quit IRC | 05:31 | |
| *** ramishra has joined #tripleo | 05:35 | |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Fix for ironic node creation https://review.openstack.org/123631 | 05:50 |
| *** Yanyanhu has quit IRC | 05:51 | |
| morganfainberg | tchaypo, mind if I circle back up with you tomorrow on this front? | 05:52 |
| morganfainberg | tchaypo, i'll get some sleep and see where we are tomorrow see if we can hammer out *what* is actually going on | 05:52 |
| tchaypo | morganfainberg: no problem | 05:59 |
| tchaypo | lifeless: I'm suddenly tempted to go get those things that stop seats reclining | 06:00 |
| *** pensu has quit IRC | 06:00 | |
| morganfainberg | tchaypo, hopefully we can get this all sorted :) | 06:00 |
| morganfainberg | see you tomorrow. | 06:00 |
| *** tzumainn has quit IRC | 06:01 | |
| *** Yanyanhu has joined #tripleo | 06:01 | |
| *** jcoufal has joined #tripleo | 06:01 | |
| lifeless | tchaypo: heh | 06:03 |
| lifeless | tchaypo: I'm tempted to diet so that I can sleep on the plane and not care | 06:04 |
| StevenK | tchaypo: Knee Defender | 06:04 |
| *** ramishra has quit IRC | 06:04 | |
| StevenK | I'm tempted to upgrade to business so I can sleep most of the way to DXB. But I don't think I have the points. | 06:04 |
| *** ramishra has joined #tripleo | 06:07 | |
| *** killer_prince has joined #tripleo | 06:07 | |
| *** killer_prince is now known as lazy_prince | 06:07 | |
| *** k4n0 has joined #tripleo | 06:09 | |
| lifeless | sadly I'm not flying with airnz | 06:14 |
| lifeless | singapore are star alliance, but not the native carrier | 06:14 |
| StevenK | lifeless: So you get status, but can't upgrade? | 06:16 |
| *** ramishra has quit IRC | 06:18 | |
| *** sdake_ has joined #tripleo | 06:20 | |
| *** sdake_ has quit IRC | 06:21 | |
| *** sdake_ has joined #tripleo | 06:21 | |
| *** pensu has joined #tripleo | 06:21 | |
| tchaypo | iirc lifeless will get to use the Silverkris lounge and I'll be stuck in Krisflyer Gold | 06:21 |
| tchaypo | which, despite the name, is far less superior | 06:21 |
| tchaypo | it's the "here's where we can stick the bulk of our nasty partner people so they don't pollute our lovely lounge" lounge | 06:22 |
| * tchaypo re-discovers that "nova force-delete" doesn't mean anything remotely similar to "forcibly delete" | 06:22 | |
| lifeless | StevenK: more or less | 06:23 |
| lifeless | tchaypo: novva reset-status | 06:23 |
| openstackgerrit | Nikunj Aggarwal proposed a change to openstack/diskimage-builder: Fixes element-info -d ubuntu errors https://review.openstack.org/111980 | 06:24 |
| StevenK | ERROR (CommandError): 'reset-status' is not a valid subcommand | 06:24 |
| tchaypo | nor is novva | 06:24 |
| StevenK | Oh, reset-state | 06:24 |
| tchaypo | yeah, that's great | 06:25 |
| tchaypo | then I go to delete and it goes back into ERROR | deleting | 06:25 |
| StevenK | Is there a traceback in the logs? | 06:26 |
| tchaypo | I spoke to soon. | 06:26 |
| tchaypo | of course it goes into | deleting | 06:26 |
| tchaypo | that's becaues i told it to delete. shortly thereafter it's gone. | 06:26 |
| StevenK | Hahaha | 06:26 |
| *** yamahata has quit IRC | 06:27 | |
| *** yamahata has joined #tripleo | 06:27 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-incubator: Use basename when setting PS4 https://review.openstack.org/123334 | 06:28 |
| *** mrunge has joined #tripleo | 06:32 | |
| StevenK | tchaypo: However, that's better than what I get which is no lounge at all | 06:32 |
| openstackgerrit | Ramakrishnan G proposed a change to openstack/diskimage-builder: Add support to build iso image https://review.openstack.org/123642 | 06:33 |
| *** jprovazn has joined #tripleo | 06:41 | |
| *** ramishra has joined #tripleo | 06:41 | |
| *** regebro has joined #tripleo | 06:42 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-incubator: Switch to new setup-neutron https://review.openstack.org/117401 | 06:43 |
| *** athomas has joined #tripleo | 06:43 | |
| *** jistr has joined #tripleo | 06:46 | |
| pensu | Hi all, please share your views: https://ask.openstack.org/en/question/48293/tripleo-error-in-baremetal-installation/ | 06:47 |
| *** chmouel has joined #tripleo | 06:51 | |
| openstackgerrit | yolanda.robla proposed a change to openstack/diskimage-builder: Install debootstrap in debian element https://review.openstack.org/123542 | 06:51 |
| *** Yanyanhu has quit IRC | 06:51 | |
| *** jtomasek has joined #tripleo | 06:56 | |
| tchaypo | "So, as I understand there is no IMAGE_NAME set in the system." - I don't think that's right | 06:57 |
| tchaypo | I would interpret that as meaning that the script running dib-run-parts had set that environment variable | 06:57 |
| tchaypo | but since you haven't set it in your shell the script isn't working | 06:58 |
| *** ramishra has quit IRC | 06:58 | |
| tchaypo | or, possibly, the script doesn't set it and that's why running DIB-RUN-PARTS failed | 06:58 |
| tchaypo | but I think you figured that out already and that's why you tried to give it a value | 06:59 |
| *** sdake_ has quit IRC | 06:59 | |
| *** ramishra_ has joined #tripleo | 07:00 | |
| *** Yanyanhu has joined #tripleo | 07:02 | |
| *** ifarkas has joined #tripleo | 07:02 | |
| openstackgerrit | Christian Berendt proposed a change to openstack/tuskar: Stop using intersphinx https://review.openstack.org/123645 | 07:05 |
| openstackgerrit | Christian Berendt proposed a change to openstack/tuskar-ui: Stop using intersphinx https://review.openstack.org/123647 | 07:06 |
| openstackgerrit | Christian Berendt proposed a change to openstack/python-tuskarclient: Stop using intersphinx https://review.openstack.org/123648 | 07:07 |
| *** GheRivero_ has joined #tripleo | 07:11 | |
| *** GheRivero has quit IRC | 07:14 | |
| *** GheRivero_ is now known as GheRivero | 07:15 | |
| *** ChanServ sets mode: +v GheRivero | 07:15 | |
| nikunj2512 | Please review this patch - https://review.openstack.org/#/c/111980/6 | 07:18 |
| openstackgerrit | Dougal Matthews proposed a change to openstack/tripleo-image-elements: Update Tuskar element https://review.openstack.org/122183 | 07:20 |
| d0ugal | StevenK: removed the blank line :) | 07:20 |
| *** ramishra_ has quit IRC | 07:26 | |
| *** tuvenen has joined #tripleo | 07:28 | |
| *** gfidente has joined #tripleo | 07:31 | |
| *** ramishra has joined #tripleo | 07:32 | |
| *** jp_at_hp has joined #tripleo | 07:35 | |
| *** ramishra_ has joined #tripleo | 07:41 | |
| *** ramishra has quit IRC | 07:44 | |
| *** athomas has quit IRC | 07:51 | |
| *** xuhaiwei has quit IRC | 07:53 | |
| *** athomas has joined #tripleo | 07:54 | |
| *** ramishra_ has quit IRC | 08:05 | |
| *** pblaho has joined #tripleo | 08:07 | |
| *** mrunge_ has joined #tripleo | 08:16 | |
| *** lucasagomes has joined #tripleo | 08:17 | |
| *** mrunge has quit IRC | 08:18 | |
| *** jang1 has joined #tripleo | 08:19 | |
| *** bvivek has quit IRC | 08:21 | |
| *** ramishra has joined #tripleo | 08:23 | |
| *** dtantsur|afk is now known as dtantsur | 08:25 | |
| regebro | Sigh. If two runs of devtest would at least end in the same error message... | 08:25 |
| shadower | regebro: what's the latest? | 08:25 |
| *** bvivek has joined #tripleo | 08:26 | |
| regebro | shadower: "ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500)" | 08:27 |
| *** mrunge_ is now known as mrunge | 08:28 | |
| shadower | regebro: would you add a few preceding lines for context, pls? | 08:29 |
| regebro | shadower: http://pastebin.com/LQJ7Yguk | 08:30 |
| regebro | But it seems pointless in trying to figure out what an error is unless I get it twice in a row... | 08:31 |
| shadower | yea | 08:31 |
| shadower | regebro: are you rebuilding the images at every run? | 08:31 |
| regebro | shadower: yes. | 08:32 |
| regebro | shadower: It rarely fails during image building. Unless I run out of diskspace, which has happened. :) | 08:32 |
| shadower | regebro: but it pulls in the latest checkouts of everything which makes the run much less deterministic | 08:33 |
| shadower | not to mention taking forever | 08:33 |
| shadower | regebro: I'd recommend doing a `devtest.sh --build-only` run once (this will just build the images) and then follow it up with `devtest.sh -c` | 08:34 |
| shadower | the latter will use the existing images | 08:34 |
| regebro | shadower: OK, I'll try that. | 08:34 |
| shadower | which means you should at least arrive to the same error every time | 08:34 |
| regebro | shadower: I haven't tried that for a while, since then --build-only wouldn't work. | 08:34 |
| regebro | I also don't know how to run it "step by step" this way.... | 08:35 |
| jprovazn | regebro: you might find more details when ssh-ing to the undercloud machine - it seems that user-config script failed (which calls "nova keypair-add" on the server) | 08:35 |
| regebro | jprovazn: Yeah, but that's for when I actually get the same error twice. ;-) | 08:35 |
| jprovazn | regebro: there should be the reason of the 500-error in logs | 08:35 |
| regebro | Which were weeks ago, I think. :-P | 08:35 |
| shadower | regebro: build-only has been working for me fine recently. I'd go away from manual step-by-step. It's more error prone | 08:36 |
| shadower | at least I tend to make a lot of mistakes that way | 08:37 |
| regebro | shadower: I usually cut an paste from this: http://docs.openstack.org/developer/tripleo-incubator/devtest.html | 08:37 |
| shadower | yea I used to do that too. Maybe you're better but I've always forgotten a step or made a mistake | 08:38 |
| * shadower has been very happy with the devtest.sh scripts | 08:38 | |
| shadower | well as happy as someone can be with such a complex thing | 08:39 |
| *** ramishra has quit IRC | 08:39 | |
| openstackgerrit | Ramakrishnan G proposed a change to openstack/diskimage-builder: Add support to build iso image https://review.openstack.org/123642 | 08:42 |
| *** ramishra has joined #tripleo | 08:44 | |
| *** jistr has quit IRC | 08:45 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Mandate double-quoting of vars and command substitutions https://review.openstack.org/118652 | 08:47 |
| *** andreaf_ is now known as andreaf | 08:47 | |
| *** ianblenke has quit IRC | 08:58 | |
| openstackgerrit | A change was merged to openstack/os-cloud-config: Add a new blocking argument to register_all_nodes https://review.openstack.org/110555 | 08:59 |
| *** untriaged-bot has joined #tripleo | 09:00 | |
| untriaged-bot | Untriaged bugs so far: | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1373063 | 09:00 |
| uvirtbot | Launchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 09:00 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 09:00 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 09:00 |
| *** untriaged-bot has quit IRC | 09:00 | |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 09:00 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/os-cloud-config: Updated from global requirements https://review.openstack.org/122651 | 09:01 |
| *** ramishra has quit IRC | 09:02 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/tuskar-ui: Updated from global requirements https://review.openstack.org/123559 | 09:03 |
| *** ramishra has joined #tripleo | 09:05 | |
| *** jistr has joined #tripleo | 09:08 | |
| *** derekh has joined #tripleo | 09:09 | |
| openstackgerrit | Eamonn O'Toole proposed a change to openstack/tripleo-image-elements: Add swift.proxy-memcache metadata to proxy-server.conf https://review.openstack.org/86312 | 09:11 |
| *** Yanyanhu has quit IRC | 09:13 | |
| *** jml has quit IRC | 09:17 | |
| *** mrunge has quit IRC | 09:18 | |
| *** jml has joined #tripleo | 09:19 | |
| *** loki184 has joined #tripleo | 09:20 | |
| *** mrunge has joined #tripleo | 09:22 | |
| *** openstack has joined #tripleo | 09:24 | |
| *** ramishra has quit IRC | 09:34 | |
| *** ramishra_ has joined #tripleo | 09:36 | |
| *** pradeep has quit IRC | 09:44 | |
| *** pradeep1 has joined #tripleo | 09:44 | |
| *** rlandy has joined #tripleo | 09:51 | |
| *** kbyrne has joined #tripleo | 09:52 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Reduce resource requirements set aside for testenvs https://review.openstack.org/106352 | 09:56 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Create the h1.ci flavor on ci-overclouds https://review.openstack.org/109630 | 09:56 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-image-elements: Fix RabbitMQ element clustering start and stop https://review.openstack.org/115524 | 10:01 |
| d0ugal | lsmola: if you get time could you re-review https://review.openstack.org/#/c/122183/? | 10:06 |
| lsmola | d0ugal: just a minute | 10:06 |
| d0ugal | lsmola: thanks - no huge rush :) | 10:06 |
| lsmola | d0ugal: debuging non working javascript in tuskar-ui | 10:06 |
| *** jml has quit IRC | 10:23 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-image-elements: Handle double path registration https://review.openstack.org/113546 | 10:23 |
| *** jml has joined #tripleo | 10:26 | |
| *** nikunj2512 has quit IRC | 10:30 | |
| openstackgerrit | Alexis Lee proposed a change to openstack/tripleo-image-elements: Symlink to backup dir until ephemeral storage available https://review.openstack.org/113957 | 10:34 |
| *** rushiagr is now known as rushiagr_away | 10:40 | |
| *** rushiagr_away is now known as rushiagr | 10:43 | |
| *** pensu has quit IRC | 10:45 | |
| *** akrivoka has joined #tripleo | 10:46 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Updated from global requirements https://review.openstack.org/123559 | 10:48 |
| regebro | seriously? It worked now? | 10:56 |
| regebro | Did devtest actually finish? That only happened once before in three months. | 10:57 |
| shadower | regebro: \o/ | 11:06 |
| shadower | regebro: now always use the `-c` option and never touch anything and you'll be fine :-) | 11:07 |
| regebro | shadower: Well, I'm setting up horizon now to verify this. :-) | 11:07 |
| regebro | shadower: OK, will do! | 11:07 |
| shadower | good luck | 11:07 |
| *** rhallisey has joined #tripleo | 11:09 | |
| *** lucasagomes is now known as lucas-hungry | 11:09 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-image-elements: Fix RabbitMQ element clustering start and stop https://review.openstack.org/115524 | 11:14 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-image-elements: Fix RabbitMQ element clustering start and stop https://review.openstack.org/115524 | 11:18 |
| regebro | Hmm. login failed for user admin. | 11:23 |
| *** pensu has joined #tripleo | 11:34 | |
| *** pradeep1 has quit IRC | 11:34 | |
| *** yamahata has quit IRC | 11:35 | |
| *** pradeep has joined #tripleo | 11:36 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/tuskar-ui: Updated from global requirements https://review.openstack.org/123704 | 11:40 |
| *** weshay has joined #tripleo | 11:46 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tripleo-image-elements: Define separate horizon.conf for tuskar-ui https://review.openstack.org/123709 | 11:54 |
| *** morazi has joined #tripleo | 11:56 | |
| *** loki184 has quit IRC | 12:04 | |
| *** lucas-hungry is now known as lucasagomes | 12:08 | |
| *** julim has joined #tripleo | 12:11 | |
| openstackgerrit | A change was merged to openstack/python-tuskarclient: Adds plan-templates command to CLI https://review.openstack.org/120780 | 12:13 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Updated tuskar-ui install to setup roles and various parameters https://review.openstack.org/123503 | 12:13 |
| *** pensu has quit IRC | 12:15 | |
| *** bashok has joined #tripleo | 12:19 | |
| *** jprovazn has quit IRC | 12:26 | |
| openstackgerrit | Peter Belanyi proposed a change to openstack/tripleo-heat-templates: Add converted version of block and object storage https://review.openstack.org/123713 | 12:26 |
| *** pmyers has quit IRC | 12:27 | |
| *** pensu has joined #tripleo | 12:30 | |
| *** jdob has joined #tripleo | 12:31 | |
| regebro | A bottle of Whisky for whoever can finish setting up my tripleo test environment. | 12:33 |
| *** ramishra_ has quit IRC | 12:37 | |
| *** noslzzp has joined #tripleo | 12:37 | |
| *** ramishra has joined #tripleo | 12:41 | |
| *** rhallisey has quit IRC | 12:44 | |
| *** rhallisey has joined #tripleo | 12:48 | |
| *** pblaho has quit IRC | 12:52 | |
| *** ramishra has quit IRC | 12:57 | |
| *** morazi has quit IRC | 12:57 | |
| *** morazi has joined #tripleo | 12:58 | |
| *** tzumainn has joined #tripleo | 13:02 | |
| regebro | The whisky bottle goes to ME! | 13:07 |
| *** jprovazn has joined #tripleo | 13:20 | |
| *** jml has quit IRC | 13:21 | |
| *** yamahata has joined #tripleo | 13:21 | |
| *** yamahata_ has joined #tripleo | 13:23 | |
| *** yamahata has quit IRC | 13:24 | |
| *** jml has joined #tripleo | 13:24 | |
| *** nosnos has quit IRC | 13:27 | |
| *** nosnos has joined #tripleo | 13:27 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Fix for ironic node creation https://review.openstack.org/123631 | 13:30 |
| *** dprince has joined #tripleo | 13:31 | |
| *** nosnos has quit IRC | 13:32 | |
| *** jdob has quit IRC | 13:34 | |
| *** jdob has joined #tripleo | 13:34 | |
| *** pradeep has quit IRC | 13:39 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Horizon temprevert https://review.openstack.org/123730 | 13:42 |
| *** ramishra has joined #tripleo | 13:42 | |
| *** jdob has quit IRC | 13:46 | |
| *** jdob has joined #tripleo | 13:46 | |
| *** rhallisey has quit IRC | 13:53 | |
| gfidente | regebro, maybe you think you did yourself but ... | 13:59 |
| gfidente | ... I'd share this bottle | 14:00 |
| *** noslzzp has quit IRC | 14:01 | |
| *** yamahata_ has quit IRC | 14:02 | |
| *** yamahata_ has joined #tripleo | 14:03 | |
| *** yamahata_ has quit IRC | 14:04 | |
| *** pradeep has joined #tripleo | 14:04 | |
| *** yamahata_ has joined #tripleo | 14:05 | |
| *** rhallisey has joined #tripleo | 14:06 | |
| toabctl | slagle: for https://review.openstack.org/#/c/117704/ . the rootfs is back now. you can retest it if you have time... | 14:11 |
| *** rhallisey has quit IRC | 14:17 | |
| *** Chao_Yan has quit IRC | 14:22 | |
| slagle | toabctl: thanks, i'll have another look | 14:27 |
| *** rlandy has quit IRC | 14:27 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Updated from global requirements https://review.openstack.org/123704 | 14:28 |
| *** andreaf_ has joined #tripleo | 14:36 | |
| *** andreaf has quit IRC | 14:36 | |
| *** andreaf_ is now known as andreaf | 14:36 | |
| *** andreaf_ has joined #tripleo | 14:36 | |
| *** rhallisey has joined #tripleo | 14:37 | |
| *** jp_at_hp has quit IRC | 14:39 | |
| *** ianblenke has joined #tripleo | 14:42 | |
| *** andreaf has quit IRC | 14:43 | |
| openstackgerrit | Tomas Sedovic proposed a change to openstack/tripleo-incubator: Fix --heat-env in devtest_overcloud.sh https://review.openstack.org/123747 | 14:47 |
| *** pcrews has joined #tripleo | 14:47 | |
| *** jp_at_hp has joined #tripleo | 14:47 | |
| *** jp_at_hp has quit IRC | 14:48 | |
| *** jp_at_hp has joined #tripleo | 14:51 | |
| *** cwolferh_ has joined #tripleo | 14:52 | |
| *** jprovazn is now known as jprovazn_afk | 14:56 | |
| *** andreaf has joined #tripleo | 14:57 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Set pki_setup False https://review.openstack.org/123752 | 14:58 |
| *** ifarkas has quit IRC | 14:59 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Set pki_setup False https://review.openstack.org/123752 | 14:59 |
| derekh | So looks like https://review.openstack.org/#/c/123730/ fixes our CI problem , the f20 job is still running but the other 2 have passed http://goodsquishy.com/downloads/tripleo-jobs.html | 14:59 |
| derekh | I'm going to update the commit message to something more sane, then I think we should just merge it, (without waiting another 2 hours | 15:00 |
| derekh | sound ok ? | 15:00 |
| *** untriaged-bot has joined #tripleo | 15:01 | |
| untriaged-bot | Untriaged bugs so far: | 15:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1373063 | 15:01 |
| uvirtbot | Launchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress] | 15:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 15:01 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 15:02 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1373455 | 15:02 |
| uvirtbot | Launchpad bug 1373455 in tripleo "horizon's bootstrap-datepicker..js could not be found?" [Undecided,In progress] | 15:02 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 15:02 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 15:02 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 15:02 |
| *** untriaged-bot has quit IRC | 15:02 | |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 15:02 |
| *** cwolferh_ has quit IRC | 15:03 | |
| openstackgerrit | Tomas Sedovic proposed a change to openstack/tripleo-incubator: Drop stack name from the neutron agent-list check https://review.openstack.org/123754 | 15:05 |
| *** bvivek has quit IRC | 15:18 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: temprevert "Datepicker locale support" https://review.openstack.org/123730 | 15:18 |
| derekh | ok, somebody A+ ^^ to unblock CI | 15:20 |
| *** jistr has quit IRC | 15:21 | |
| *** jml has quit IRC | 15:21 | |
| *** jml has joined #tripleo | 15:22 | |
| derekh | dprince: slagle https://review.openstack.org/#/c/123730/2 | 15:24 |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: temprevert "Datepicker locale support" https://review.openstack.org/123730 | 15:27 |
| slagle | derekh: ^ | 15:27 |
| derekh | slagle: thanks | 15:27 |
| openstackgerrit | Tomas Sedovic proposed a change to openstack/tripleo-incubator: Pass resource registry file to Heat if it exists https://review.openstack.org/123761 | 15:29 |
| *** marun has joined #tripleo | 15:29 | |
| *** rushiagr is now known as rushiagr_away | 15:29 | |
| *** loki184 has joined #tripleo | 15:30 | |
| *** ramishra has quit IRC | 15:31 | |
| *** lazy_prince is now known as killer_prince | 15:34 | |
| *** rushiagr_away is now known as rushiagr | 15:37 | |
| *** jml has quit IRC | 15:40 | |
| *** jml has joined #tripleo | 15:41 | |
| *** ramishra has joined #tripleo | 15:47 | |
| *** tuvenen has quit IRC | 15:52 | |
| *** saurabhs has joined #tripleo | 15:55 | |
| *** ramishra has quit IRC | 15:56 | |
| *** killer_prince is now known as lazy_prince | 15:59 | |
| *** dshulyak_ has joined #tripleo | 16:01 | |
| *** k4n0 has quit IRC | 16:03 | |
| *** jmoffitt_ has quit IRC | 16:03 | |
| *** dshulyak_ has quit IRC | 16:06 | |
| *** jml has quit IRC | 16:09 | |
| *** jml has joined #tripleo | 16:14 | |
| *** rhallisey has quit IRC | 16:16 | |
| *** dshulyak_ has joined #tripleo | 16:22 | |
| *** dsneddon has quit IRC | 16:24 | |
| *** dsneddon has joined #tripleo | 16:25 | |
| *** pradeep has quit IRC | 16:27 | |
| *** openstackgerrit has quit IRC | 16:35 | |
| *** rhallisey has joined #tripleo | 16:44 | |
| *** dsneddon has quit IRC | 16:45 | |
| *** jistr has joined #tripleo | 16:45 | |
| *** dsneddon has joined #tripleo | 16:46 | |
| *** marun has quit IRC | 16:49 | |
| *** yamahata_ has quit IRC | 16:52 | |
| *** pensu has quit IRC | 16:52 | |
| *** noslzzp has joined #tripleo | 16:55 | |
| *** bvivek has joined #tripleo | 16:56 | |
| *** derekh has quit IRC | 16:56 | |
| *** jp_at_hp has quit IRC | 17:01 | |
| *** rwsu has quit IRC | 17:03 | |
| *** rwsu has joined #tripleo | 17:05 | |
| *** dsneddon has quit IRC | 17:06 | |
| *** dsneddon has joined #tripleo | 17:06 | |
| *** rushiagr is now known as rushiagr_away | 17:08 | |
| *** akrivoka has quit IRC | 17:15 | |
| *** bashok has quit IRC | 17:15 | |
| *** andreaf has quit IRC | 17:17 | |
| *** andreaf has joined #tripleo | 17:18 | |
| *** lucasagomes is now known as lucas-dinner | 17:20 | |
| *** mrunge has quit IRC | 17:24 | |
| *** openstackgerrit has joined #tripleo | 17:24 | |
| *** dsneddon has quit IRC | 17:26 | |
| *** dsneddon has joined #tripleo | 17:27 | |
| *** dtantsur is now known as dtantsur|afk | 17:28 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tripleo-image-elements: Handle static files properly https://review.openstack.org/123796 | 17:29 |
| *** dtantsur|afk is now known as dtantsur|nearly- | 17:32 | |
| *** dtantsur|nearly- is now known as dtantsur|nearlya | 17:32 | |
| *** dtantsur|nearlya is now known as dtantsur|afk | 17:32 | |
| *** jcoufal has quit IRC | 17:51 | |
| *** openstackgerrit has quit IRC | 17:51 | |
| *** openstackgerrit has joined #tripleo | 17:52 | |
| *** bashok has joined #tripleo | 17:57 | |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-incubator: Support passing x86_64 to create-nodes https://review.openstack.org/123803 | 18:03 |
| *** marun has joined #tripleo | 18:05 | |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-incubator: Support passing x86_64 to create-nodes https://review.openstack.org/123803 | 18:11 |
| openstackgerrit | patrick-crews proposed a change to openstack/os-cloud-config: ssl_for_create_admin_client https://review.openstack.org/123806 | 18:14 |
| *** pradeep has joined #tripleo | 18:15 | |
| *** jistr has quit IRC | 18:16 | |
| EmilienM | lifeless: I'm not sure you are online, but if yes, could you see the conversation on #openstack-infra about http://lists.openstack.org/pipermail/openstack-dev/2014-September/045395.html ? | 18:21 |
| *** spzala has joined #tripleo | 18:26 | |
| *** zigo has quit IRC | 18:26 | |
| *** zigo has joined #tripleo | 18:28 | |
| *** sdake_ has joined #tripleo | 18:29 | |
| *** lsmola has quit IRC | 18:31 | |
| *** lsmola has joined #tripleo | 18:31 | |
| *** pradeep has quit IRC | 18:37 | |
| *** andreaf has quit IRC | 18:38 | |
| *** andreaf has joined #tripleo | 18:39 | |
| *** pcrews has quit IRC | 18:42 | |
| *** bvivek has quit IRC | 18:49 | |
| *** andreaf has quit IRC | 18:50 | |
| *** andreaf has joined #tripleo | 18:51 | |
| *** jang1 has quit IRC | 18:57 | |
| *** morganfainberg is now known as morgan | 19:02 | |
| *** morgan is now known as morganfainberg | 19:03 | |
| *** bashok_ has joined #tripleo | 19:06 | |
| *** bashok has quit IRC | 19:06 | |
| *** jmoffitt has joined #tripleo | 19:07 | |
| *** bashok has joined #tripleo | 19:14 | |
| *** bashok_ has quit IRC | 19:14 | |
| *** ChuckC has quit IRC | 19:18 | |
| *** athomas has quit IRC | 19:19 | |
| *** jprovazn_afk has quit IRC | 19:20 | |
| *** loki184 has quit IRC | 19:21 | |
| *** bashok has quit IRC | 19:28 | |
| *** andreaf has quit IRC | 19:31 | |
| *** andreaf has joined #tripleo | 19:32 | |
| *** sdake_ has quit IRC | 19:36 | |
| *** lazy_prince is now known as killer_prince | 19:38 | |
| *** openstackgerrit has quit IRC | 19:46 | |
| *** openstackgerrit has joined #tripleo | 19:47 | |
| *** pcrews has joined #tripleo | 19:50 | |
| *** ChuckC has joined #tripleo | 19:54 | |
| *** sballe__ has joined #tripleo | 20:00 | |
| *** gfidente has quit IRC | 20:02 | |
| *** jtomasek has quit IRC | 20:11 | |
| *** jml has quit IRC | 20:33 | |
| *** jml has joined #tripleo | 20:34 | |
| *** dprince has quit IRC | 20:37 | |
| *** bashok has joined #tripleo | 20:42 | |
| *** dividehex has quit IRC | 20:44 | |
| *** dividehex has joined #tripleo | 20:47 | |
| *** jdob has quit IRC | 20:51 | |
| *** bashok has quit IRC | 20:51 | |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | Untriaged bugs so far: | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1373063 | 21:00 |
| uvirtbot | Launchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1369256 | 21:00 |
| uvirtbot | Launchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1366164 | 21:00 |
| uvirtbot | Launchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1367742 | 21:00 |
| uvirtbot | Launchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New] | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| *** noslzzp has quit IRC | 21:02 | |
| *** sballe__ has quit IRC | 21:05 | |
| *** spzala has quit IRC | 21:05 | |
| *** marun has quit IRC | 21:52 | |
| morganfainberg | tchaypo, let me know when you want to pick up working on the identity stuff again | 21:56 |
| -openstackstatus- NOTICE: The openstack-infra/config repo will be frozen for project-configuration changes starting at 00:01 UTC. If you have a pending configuration change that has not merged or is not in the queue, please see us in #openstack-infra. | 21:59 | |
| openstackgerrit | Richard Su proposed a change to openstack/diskimage-builder: Install openstack-selinux on RHEL https://review.openstack.org/123865 | 21:59 |
| tchaypo | tchaypo: I'm just getting breakfast now, give me 30 minutes or so | 22:04 |
| *** marun has joined #tripleo | 22:15 | |
| morganfainberg | tchaypo, no rush | 22:18 |
| morganfainberg | tchaypo, take your time. :) seriously breakfast is important | 22:18 |
| openstackgerrit | Richard Su proposed a change to openstack/diskimage-builder: Install openstack-selinux on RHEL https://review.openstack.org/123865 | 22:19 |
| tchaypo | indeed. I can't think without it. | 22:24 |
| *** julim has quit IRC | 22:25 | |
| tchaypo | oh good | 22:25 |
| tchaypo | i think to myself "I'll just delete the undercloud and quickly build a fresh one while I'm cooking breakfast | 22:25 |
| tchaypo | " | 22:26 |
| tchaypo | so naturally that is now in DELETE_FAILED too | 22:26 |
| *** nijaba_ has joined #tripleo | 22:35 | |
| *** nijaba has quit IRC | 22:35 | |
| *** otoolee has quit IRC | 22:41 | |
| *** spzala has joined #tripleo | 22:43 | |
| *** openstackgerrit has quit IRC | 22:47 | |
| *** openstackgerrit_ has joined #tripleo | 22:47 | |
| *** morazi has quit IRC | 22:48 | |
| *** openstackgerrit_ is now known as openstackgerrit | 22:48 | |
| * tchaypo is done with breakfast | 22:49 | |
| tchaypo | Error: Server undercloud-undercloud-meprf37xkzvy delete failed: (500) No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 404) | 22:50 |
| cinerama | wow, that's different - i thought it was going to be the usual thing where you delete the underlying bits, hope heat resolves itself, and poke the database as needed if not | 22:54 |
| tchaypo | wow. | 22:56 |
| tchaypo | coffee might be an important part of breakfast. | 22:57 |
| tchaypo | i just ran "top" when i meant "head" | 22:57 |
| tchaypo | okay | 22:59 |
| tchaypo | restart the -conductor services, reset-state on the node, now a delete works | 22:59 |
| tchaypo | undercloud build try #2 commences now.. | 22:59 |
| *** Chao_Yan has joined #tripleo | 23:11 | |
| tchaypo | and we have an undercloud | 23:21 |
| tchaypo | morganfainberg: I'm going to enable debugging on all the things then kick off an overcloud build | 23:21 |
| morganfainberg | tchaypo, cool | 23:21 |
| morganfainberg | tchaypo, just joined the screen sessiin | 23:22 |
| morganfainberg | session* | 23:22 |
| morganfainberg | tchaypo, you didn't need the middleware line but it wont hurt anything | 23:26 |
| *** dsneddon has quit IRC | 23:27 | |
| tchaypo | also setting nova's retry count higher (not that that seems to be related but we're going to need it eventually) | 23:28 |
| *** dsneddon has joined #tripleo | 23:29 | |
| morganfainberg | ah ok | 23:29 |
| *** marun has quit IRC | 23:31 | |
| morganfainberg | i'm logging into the same box you're on in the screen session so i can look at the logs directly as well | 23:34 |
| morganfainberg | yay i can track token reuse: <KeystoneToken (audit_id=hlP7b2paQpa6TtddoC7DPA, audit_chain_id=hlP7b2paQpa6TtddoC7DPA) at 0x7f281b8ccd08> | 23:36 |
| tchaypo | I just got the 20 minute warning from my driving instructor, and it usually takes about 20 minutes between kicking this off and the heat stack-create | 23:37 |
| * tchaypo has been meaning to tweak the script to reuse * and jump straight to heat stack-create | 23:37 | |
| morganfainberg | tchaypo, doh! | 23:37 |
| morganfainberg | hehe | 23:37 |
| lifeless | tchaypo: how many hours of driving instruction are you getting ? | 23:43 |
| tchaypo | I think it'll be about 30 by the time I'm done | 23:44 |
| *** dsneddon has quit IRC | 23:48 | |
| *** jml has quit IRC | 23:50 | |
| *** dsneddon has joined #tripleo | 23:51 | |
| morganfainberg | tchaypo, where does the failure usually occur? | 23:51 |
| *** jml has joined #tripleo | 23:52 | |
| *** openstackstatus has quit IRC | 23:55 | |
| *** openstack has joined #tripleo | 23:56 | |
| *** openstackstatus has joined #tripleo | 23:57 | |
| *** ChanServ sets mode: +v openstackstatus | 23:57 | |
| tchaypo | driving, back in an hour | 23:57 |
| tchaypo | stack-create will happen, `nova list` will show ~23 compute nodes and 3 control nodes. ips will be assigned to all of them. 5 will build successfully, then the rest hit the problem where ironic-conduct tells neutron to allocate a port but neutron decides the token is invalid | 23:59 |
| tchaypo | ironic reports this as "could not configure port for pxe boot" or something similar | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!