Thursday, 2014-09-25

morganfainberg it's ironic -> neutron issue00:00
morganfainberggreat i know what to look for then.00:00
morganfainbergi'm keeping an eye on the keystone side seeing what might be going on there.00:00
*** lucas-dinner has quit IRC00:01
jogowe are adding os-collect-config.txt logs into lostash but they aren't getting parsed properly00:02
jogolifeless: ^00:03
jogoso we are wasting space for nothingness00:06
jogofiled a bug: bug 137367200:09
uvirtbotLaunchpad bug 1373672 in tripleo "overcloud-controller0_logs/os-collect-config.txt isn't getting parsed correctly in elasticsearch" [Undecided,New]
*** dsneddon has quit IRC00:10
*** dsneddon has joined #tripleo00:12
*** bkero has quit IRC00:12
*** bkero has joined #tripleo00:15
*** openstackgerrit has quit IRC00:16
lifelessjogo: agree, we should fix the filter00:16
lifelesstchaypo: ironic doesn't add a port, it updates one.00:17
*** openstackgerrit has joined #tripleo00:17
lifelesstchaypo: (minor but key nit :))00:17
*** spzala has quit IRC00:25
*** jml has quit IRC00:28
*** dsneddon has quit IRC00:31
*** jml has joined #tripleo00:32
*** dsneddon has joined #tripleo00:32
*** yamahata_ has joined #tripleo00:36
*** dkehn has quit IRC00:49
*** ChuckC has quit IRC00:49
*** dkehn has joined #tripleo00:51
*** dsneddon has quit IRC00:53
*** dsneddon has joined #tripleo00:53
*** dshulyak_ has quit IRC01:00
*** dshulyak_ has joined #tripleo01:07
*** dshulyak_ has quit IRC01:07
morganfainbergtchaypo, i don't think i've seen a failed01:12
morganfainbergbut i honestly have been looking limitedly at keystone01:12
morganfainbergand ironic01:12
morganfainbergwhoopse. that was a mistake01:13
morganfainbergtchaypo, well that doesn't look like failures01:14
tchaypoI think the failures are being disguised by the fact that nova is retrying 50x each node01:15
morganfainbergbah we don't have an index on 'valid' tokens.01:15
tchaypobut there's only 4 running, I would have expected 5 by now01:15
*** jang has quit IRC01:17
*** jang has joined #tripleo01:18
morganfainbergtchaypo, is it ok if I do a couple ugly queries against the DB?01:19
morganfainbergtchaypo, or would you prefer i didn't (might cause some issues with tokens)01:19
lifelessmorganfainberg: selects orupdates?01:21
morganfainberglifeless, selects01:21
lifelessmorganfainberg: how would selects cause issues?01:21
morganfainberglifeless, but they're whole table scans (32k rows)01:21
lifelessstill, thats tiny01:21
morganfainberglifeless, unindexed columns.01:21
lifelessstill tiny01:21
morganfainbergyeah might cause token read locks which are unfun some times.01:21
morganfainbergbut shouldn't be awful01:21
morganfainbergmysql> select id, expires, valid, trust_id, user_id from token where valid = 0;01:24
morganfainbergEmpty set (1 min 30.46 sec)01:24
morganfainbergthat took a lot longer than i expected.01:24
*** ci-overcloud has joined #tripleo01:25
ci-overcloud************** ci-overcloud complete status=1 ************01:25
*** ci-overcloud has quit IRC01:25
morganfainbergok so, spot check, no tokens should be invalid01:25
morganfainbergand no 404s returned by keystone (according to the log)01:27
lifelessthere may be a mysqlconfig issue then01:28
lifelesstchaypo: check the mysql memory param is set in the deploy config for hp2 ;)01:28
tchaypoif this run ever ends, I'm going to set max retrys back to 3, since I think the value of 50 is helping to mask the problem01:29
tchaypobut heck maybe the value of 50 is what we need to make it work and this whole thing is going to magically coalesce into a working cloud one day01:29
lifelessruh roh01:31
morganfainbergtchaypo, uh i think we need to find out why retry=50 is needed then :P01:31
lifelessthat bash exploit is rather bad01:31
morganfainberglifeless, ?01:31
uvirtbotlifeless: ** RESERVED ** This candidate has been reserved by an organization or individual that will use it when announcing a new security problem.  When the candidate has been publicized, the details for this candidate will be provided. (
tchaypoI've heard rumours that XSA-108 is going to be bad as well01:31
tchaypobut still just says it01:31
tchaypos "pre-released but embargoed"01:32
lifelesstchaypo: I think you'll need to upgrade bash on all the externally listening things asap :)01:32
morganfainberglifeless, that looks like a rough one01:32
* morganfainberg goes and checks bash on his server01:33
* tchaypo apt-get upgrades01:35
morganfainbergphew mine is good01:35
morganfainbergtchaypo, just checked the bastion i'm logged into, yeah needs update01:35
*** tchaypo has quit IRC01:35
*** tchaypo has joined #tripleo01:36
*** sseago has quit IRC01:36
lifelesssowe need to update the bastion and the seed and the uc and the oc01:38
lifelessoc - should build fresh images to pull it in01:38
tchaypookay, that's better01:39
tchaypoupgrading bastion.hp2 now01:40
*** marun has joined #tripleo01:44
*** nosnos has joined #tripleo01:45
*** ramishra has joined #tripleo01:46
* tchaypo reads the kolla thread01:47
tchaypoit'd be really nice if the undercloud could resolve names01:53
lifelesstchaypo: there is a bug open on that01:56
tchaypo... nova list is taking a while..02:01
tchaypomorganfainberg: take a look on window 1 - theres a bunch that have now failed and are being deleted and re-built.. and one that's ERROR|deleting just for fun02:10
morganfainbergtchaypo, cool thanks02:10
*** ramishra has quit IRC02:11
lifelessok, hp1 bastion, seed, uc, oc1 and oc2 controllers upgraded02:11
tchaypoin unrelated news, <1sec between lightning and thunder.02:11
tchaypohp2 bastion, seed, uc all upgraded.02:11
morganfainbergtchaypo, huh, ok so keystone doens't seem to be saying invalid tokens.02:11
morganfainbergaccording to the db02:11
tchayponot seeing WARNINGs from ironic-conductor nor neutron-server either.02:13
morganfainbergi wonder...02:15
tchayponeutron-server does mention "no more dhcp-agents" once or twice but that doesn't seem to be fatal02:16
morganfainbergdid we stop everything?02:16
morganfainbergor are somethings still running02:16
morganfainbergcause the keystone log stopped02:16
tchaypoI haven't tried to stop anything02:16
morganfainbergi'm *thinking* i know what is going on here.02:17
tchaypo... I'm suddenly worried about what the apt-get upgrade did02:17
morganfainbergnot too worried aobut that02:17
*** marun has quit IRC02:17
morganfainbergat least i don't *think* that is related here02:17
tchaypoWhat are you thinking?02:17
lifelessdum dum dum daaaa02:18
morganfainbergi think this is an issue with token delete / invalidation is causing SQL issues02:18
morganfainbergSep 25 02:13:02 undercloud-undercloud-4ijtx4oh5yh7 keystone: 2014-09-25 02:13:02.349 10114 DEBUG keystone.notifications [-] Invoking callback _delete_user_tokens_callback for event identity user deleted for{'resource_info': u'a37337710c814ab1a69c80123c62d5c6'} notify_event_callbacks /opt/stack/venvs/keystone/local/lib/python2.7/site-packages/keystone/
morganfainberglast thing in the log02:18
morganfainbergdoing a select to see how many tokens that user has02:18
morganfainbergit's *really* slow select02:18
tchaypolots of them?02:18
tchaypoor lack of index?02:18
morganfainbergeither that or the token table is locked02:18
morganfainbergand we're waiting on a flush02:18
morganfainbergzero tokens for that user02:19
tchaypogetting back to lifeless question about memory for mysql earlier - i can't see any overall memory setting in the logs (lots of settings for memory for specific things)02:19
tchayposo I'm not sure what he was referring to02:19
lifelesswe have a parameter02:19
lifelessin the heat template02:20
lifelessthat configures the mysql slab cache02:20
lifelessstack-show should  let you see its setting02:20
morganfainbergoh dear.02:21
morganfainbergsome of these keystone queries are consistently in the slow log02:21
tchaypoin undercloud-source.yaml?02:22
lifelessmorganfainberg: see, as a keystone dev you should totally be running a production keystone :_02:22
lifelesstchaypo: heat stack-show undercloud ...02:22
morganfainbergcrap. why are where claused on this user_id bit02:22
morganfainbergwhy are we where claused on the user_id....02:23
tchaypoheat stack-show undercloud | grep mysql shows nothing02:23
StevenKtchaypo: grep -i ?02:23
lifelessMysqlInnodbBufferPoolSize is the parameter02:24
morganfainbergas much as i'd hate to do it, i bet if we used the memcache token backend it would "just work™" (no we shouldn't really do this)02:24
morganfainbergoh yep there it is.02:25
morganfainbergwe're deleting users.02:25
morganfainbergyep, that is the culprit it looks like it makes token table unhappy02:27
lifelessmissing FK ?02:27
lifelessor cascade delete?02:27
morganfainberglifeless, missing index it looks like02:28
morganfainbergit's a patch i was pushing back on this cycle because it's an alter to the token table.02:28
morganfainbergand no one could point out why the search on uder_id was happening as much as it was02:28
morganfainbergwell.. looks like we delete users.02:28
morganfainberglifeless, it also *feels* like a mysql tuning issue as well02:29
morganfainbergcombine both and you get ick.02:29
lifelessheat creates temporary users02:29
tchaypo "MysqlInnodbBufferPoolSize": "100",02:29
morganfainbergdoing a profile of the query02:29
morganfainbergit's really ugly atm02:29
lifelesstchaypo: that would be a little small02:29
morganfainberg100? haahah02:29
morganfainbergoh yeah02:29
StevenKIsn't that the default?02:30
StevenKI think02:30
morganfainbergStevenK, mysql's defaults are insane for any load beyond "toy" level02:30
morganfainberglifeless, tchaypo,
StevenKmorganfainberg: I'd bang the Postgres drum a little here, but people don't like me doing that. :-P02:31
morganfainbergwith the small bufferpoolsize we're causing backups on that type of query with even moderate roles.02:31
morganfainbergerm s/roles/tokens02:31
morganfainbergack s/roles/rows02:31
lifelessStevenK: its a dev default for vms02:32
lifelessStevenK: and a case where I don't like having CI/prod be the ones to make it different02:32
morganfainberglifeless, the sad part is... that query returns 0 rows.02:32
lifelessmorganfainberg: much work02:32
lifelessmorganfainberg: nothing to show for it02:32
lifelessmorganfainberg: wow02:32
morganfainbergso query02:32
tchaypo100 is not the default02:32
tchaypo100 is hard-coded in devtest_undercloiud.sh02:33
*** dshulyak_ has joined #tripleo02:33
StevenKHah. Nice.02:33
morganfainberghm, let me see if i can bump that up inside mysql02:33
morganfainbergi'd like to see if it affects things *a lot*02:33
StevenKIt's also hard-coded inside devtest_overcloud.sh02:34
tchaypobut we can override that with $HEAT_ENV02:34
*** jml has quit IRC02:34
lifelesstchaypo: it is meant to be overridden via the env file for the uc02:34
tchaypoactually looking at I'm not sure if the HEAT_ENV we read in on line 196 gets respected or tampled02:35
lifelesstchaypo: suggest you check its there and if not add it (thats in tie)02:35
lifelesstchaypo: there's a heat env for the uc and oc for hp1 and hp2 in tie02:35
lifelesstchaypo: you may have not used it, which would cause issues ;)02:36
*** jml has joined #tripleo02:36
tchaypo"MysqlInnodbBufferPoolSize": 4096,02:36
*** sseago has joined #tripleo02:37
* StevenK glares at his mirror pulse including new releases of firefox for each supported release.02:37
morganfainbergugh, ok can't set the pool size live02:38
tchaypoapparently I did not use it.02:38
morganfainbergalso we might want more than 1 buffer pool.02:38
StevenKtchaypo: Huh. We have sun here now02:39
morganfainbergtchaypo, lifeless, mind if i index that column live?02:39
morganfainbergtchaypo, lifeless, see how it affects things.02:39
tchayporain is easing off for me, no more thunder02:39
tchaypomorganfainberg: go for it02:39
tchaypoi'm going to have to rebuild the undercloud soon but let's do what we can to troubleshoot first02:40
lifelesstchaypo: why rebuild?02:40
StevenKmorganfainberg: Using CONCURRENTLY ?02:40
lifelessmorganfainberg: lets fix the buffer pool size first :)02:40
morganfainberglifeless, ah just tried adding the index.02:41
morganfainbergcan drop the index at any point anyway02:41
morganfainbergeasy enough02:41
lifelessit was more that its going to disrupt while it scans02:41
lifelessso we may as well suck up the disruption and fix the pool :)02:42
morganfainbergyeah sorry :P02:42
morganfainberggive me a minute or so to finish the indexing :( sorry already started.02:42
tchaypoIn that case, I may as well kick off the undercloud build now?02:42
lifelesstchaypo: what build?02:42
morganfainbergwait please don't rebuild yet!02:42
morganfainbergwe can test to see if this will have an impact without needing to rebuild.02:43
StevenKmorganfainberg: Huh, I don't like mysql's EXPLAIN output. But I've been using postgres' EXPLAIN for years02:43
morganfainbergStevenK, it's a bit wonky02:43
morganfainbergStevenK, have a POSTGRES example? haven't seen it in a while02:43
morganfainbergok so now the question is how big of an index do we need for this.. but wow. yeah this is going to be much better with an index02:44
tchaypolifeless: of the undercloud. But actually that's probably overkill for now02:44
morganfainbergi think we could get away with an index(6) and be overkill02:44
StevenKHuh. Clearly cleaning up my Launchpad developer enivronment has removed my postgres clients02:44
lifelesstchaypo: so when you rebuild02:44
tchaypowe can tweak mysql.cnf by hand for now and make sure that the next build uses the heat env02:44
lifelesstchaypo: its a great way to *test* that what you've fixed is fixed02:44
lifelesstchaypo: but its a terrible way to debug and fix issues02:45
lifelesstchaypo: doing a stack-update is entirely the right way to apply configuration changes02:45
morganfainberg that is with index(10)02:45
lifelesstchaypo: but that != rebuild02:45
morganfainbergyeah ok i'm convinced we need the index.02:45
morganfainbergok lets apply the buffer pool change and i'll check the stats as well02:45
StevenKmorganfainberg: has some small examples02:46
tchaypoto do a stack update I'd have to rebuild the heat template, then heat stack-update from the new template?02:46
lifelesstchaypo: huh02:46
morganfainberglifeless, tchaypo, without losing the ksytone state (db) how do we update the my.cnf values/innodb buffer pool sizes?02:46
StevenKmorganfainberg: I can hit up a friend to get more complex EXPLAIN output if you wish02:46
lifelesstchaypo: update the env file, then stack-update -e envfile undercloud02:46
morganfainbergStevenK, nah, that should be fine.02:46
morganfainbergStevenK, just curious02:46
morganfainbergStevenK, more like
morganfainbergstill i can see why you like the pgsql one02:47
StevenKmorganfainberg: I found it invaluable for working out if an index is being used, how much the postgres stats were used, and the column width of what postgres is doing02:47
tchaypostack-update shouldn't lose the keystone state, should it?02:47
morganfainbergso i think we also need to index trust_id in the token table.02:48
morganfainberggrr, this makes for the migration no fun for real production loads.02:48
morganfainbergunless they do flush-tokens02:48
*** rhallisey has quit IRC02:50
lifelessmorganfainberg: if mysql supports partial indices02:51
morganfainberglifeless, it does.02:51
lifelessmorganfainberg: you could index where the token timestamp is > time-migration-is-run02:51
morganfainberglifeless, oh wait uhm.02:51
lifelessmorganfainberg: colour me evil :)02:51
morganfainberglifeless, hmm... it'd be easier to token_flush02:51
morganfainbergactually i want (if we're using sql as a backend for tokens still) to move to automatic partition tables02:52
morganfainbergwith rotation02:52
morganfainbergbut that is going to require fancy code in oslo.db i think02:52
*** jml has quit IRC02:54
*** jml has joined #tripleo02:56
lifelessmorganfainberg: not to mention deploymnet02:57
lifelessmorganfainberg: or a distributed lock manager :)02:57
morganfainberglifeless, yeah.02:57
morganfainberglifeless, well... new enough mysql has store proceedures (don't shoot me)02:58
lifelessmorganfainberg: and if three clients call the stored procedure at the same time on 3 different galera nodes?02:58
morganfainbergi'm trying to think if I can fix this without needing to migrate the token table.02:58
morganfainberglow impact change.02:58
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
uvirtbotLaunchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress]03:00
uvirtbotLaunchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New]03:00
uvirtbotLaunchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New]03:00
uvirtbotLaunchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New]03:00
*** untriaged-bot has quit IRC03:00
morganfainberglifeless, hey don't poke too many holes in my hopes and dreams :P03:00
tchaypois it okay for me to go ahead with the stack update?03:02
morganfainbergtchaypo, i have no issues with it03:04
* tchaypo crosses fingers03:05
morganfainberglifeless, this also comes down to the size of the token, 42k rows should not be that hard to scan.... unless each one contains ~10k+ of data in it :(03:05
morganfainbergor 20k03:07
lifelesspostgresql pushes such data out to separate storage automatically03:07
lifelessso table scans that don't actually filter on token content would be fast03:08
morganfainberglifeless, the page sizes internal to mysql get some wonkyness.03:08
lifelesseach row just has a uuid to the TOAST entry03:08
morganfainberglifeless, the issue isn't the scan in this case, it's also the blob loading in the horrible extra field03:08
morganfainbergat 15k rows to scan, we're loading (bad math) ~150000000 bytes03:09
morganfainbergsince we are selecting the data already.03:09
morganfainbergso we're hitting the 100M buffer pool easily03:09
lifelessone thing you can do in this sort of case is manually do what pg does03:10
lifelessthat is, have a k:v table for bulk data03:10
lifelessand a FK reference in the main table03:10
morganfainbergthe whole "extra" blob in keystone is awful.03:10
lifelessand the main table only has the fields that you select on03:10
morganfainbergwe use it in too many places03:10
morganfainbergthis issue screams of three things: bad schema design (we've been slowing improving it), known token issues, and then small buffers/caches03:12
* morganfainberg sulks.03:12
morganfainbergthis might be another RC issue (better to find it now than need to backport it though)03:12
*** Chao_Yan has quit IRC03:12
*** pcrews has quit IRC03:22
*** otoolee- has joined #tripleo03:23
tchaypoapparently AWS have informed people to expect their nodes to be randomly rebooted over the next day or so03:25
tchaypobut neglected to use the appropriate soundtrack03:25
tchaypoin other news, this stack still seems to be UPDATE_IN_PROGRESS... 17 minutes so far03:25
tchaypobut my.cnf has been updated03:26
tchaypoit looks like mysqld hasn't been restarted though so I'm not sure if it will have read the new config03:27
*** Chao_Yan has joined #tripleo03:28
lifelesstchaypo: os-collect-config.log is your friend03:30
tchaypooh. I think I know what it's going to say03:31
tchaypoNope, I was wrong.03:32
*** bvivek has joined #tripleo03:35
tchaypooh goody.03:42
tchaypoheat client installed is 0.2.1003:43
tchaypo0.2.11 adds stack-cancel-update, which would be useful right now03:43
tchaypoIs there some other way to make the update go away?03:43
stevebakerI think the server side of that only just landed post j-3, you may not have it yet03:44
tchaypoaction-suspend sounds like what I want, until I read the docs and realise that it's something completely different03:44
tchaypoit suspends the stack, not an action03:45
tchaypostevebaker: do you have a hilight, or just super-powers?03:46
*** Chao_Yan has quit IRC03:46
*** Chao_Yan has joined #tripleo03:48
stevebakerI've been keeping an eye on this all day, its been most interesting03:50
tchayporight now I think the only thing I can do is wait for the stack-update to time out and then try it again?03:52
tchaypoos-collect-config is barfing in an error in the template so it will never apply..03:52
stevebakerwithout cancel-update, that is your option03:53
tchayponext time I03:53
tchaypo'm using a shorter timeout03:54
lifelesstchaypo: whats theerror ?03:54
tchaypo.... the timeout last time was 360 minutes. that's going to take a while.03:55
lifelesstchaypo: (you've summarise but not actuall explained)03:55
lifelesstchaypo: or pastebinned03:55
tchaypoI tried to merge hp2_uc_env.json with the existing undercloud_env.json and ended up with conflicting neutron parameters03:56
tchaypoos-collect-config must do something to tell heat the update completed.03:56
*** ramishra has joined #tripleo03:57
lifelessso to work around that you can just signal complete by hand from within the machine03:57
lifelesswhich is the last script in the post .d dir03:58
morganfainberglifeless, tchaypo: ok so restored a patch/bug for RC blocker on this03:59
morganfainbergshould help a lot to have the columns indexed03:59
morganfainbergthanks for being patient with me, i feel bad for blocking the change earlier in the cycle but couldn't figure out what was causing the spam to that where clause in the select03:59
tchaypomorganfainberg: we're happy to play on your guilt04:00
tchaypoI mean, no problem, it seems like a reasonable thing to do if you can't figure out where the spam came from :)04:00
lifelessmorganfainberg: well, whoever put the patch forward could presumablyhave gotten you d=soem debug traces04:01
tchaypolast line of os-collect-config log that I can see is04:03
tchaypoSep 25 04:01:49 undercloud-undercloud-4ijtx4oh5yh7 os-collect-config: + ifdown eth204:03
lifelesstchaypo: you may need the serial console04:04
tchaypoexcept that we don't have serial consoles04:04
* tchaypo sets up vnc port-forward04:04
lifelesstchaypo: we don't? iLO ?04:04
tchaypooh yeah04:05
tchayporeal hardware!04:05
morganfainberglifeless, tchaypo:
*** lynxman has quit IRC04:10
*** lynxman has joined #tripleo04:10
openstackgerritOpenStack Proposal Bot proposed a change to openstack/python-tuskarclient: Updated from global requirements
*** ChuckC_ has joined #tripleo04:11
tchaypoand now, for the first time, I need to know the root password of a node04:13
tchaypoor heat-admin would suffice04:13
*** bvivek has quit IRC04:15
*** Azher has joined #tripleo04:16
lifelesstchaypo: stack:stack04:22
tchaypoof course04:22
tchaypogood old stackuser element04:22
tchaypobr_ctlplane now has an IP in the public net04:23
lifelessmessed up :)04:24
tchaypoho hum. stoppped responding on serial console now04:31
tchaypoi smell a power-cycle coming on04:32
*** nikunj2512 has joined #tripleo04:34
nikunj2512Please review this patch -
*** killer_prince is now known as lazy_prince04:37
*** morgan_remote_ has joined #tripleo04:40
*** marun has joined #tripleo04:46
tchaypoIt's getting let for morganfainberg again, and I need to eat before my head gets so light it floats off, and this server isn't talking on serial any more.04:50
tchaypoI'm going to give up and rebuild the undercloud witht he working heat env, then give it a tiny little update that just updates mysql instead of the other things it just tried to change at the same time, and then we can take another crack tomorrow04:51
tchaypoI'll try another overcloud deploy - with debugging turned on, but with retries set to 3 instead of 50, i think04:51
morgan_remote_Works for me. Hope to have the index merge tomorrow (pending gate slowness of course)04:53
*** lsmola has quit IRC04:57
*** dshulyak_ has quit IRC04:58
*** lsmola has joined #tripleo05:11
*** pradeep has joined #tripleo05:18
*** tzumainn has quit IRC05:20
*** ramishra has quit IRC05:25
*** bvivek has joined #tripleo05:34
*** Ng has quit IRC05:47
*** Ng has joined #tripleo05:48
*** ChanServ sets mode: +v Ng05:48
*** victor_lowther__ has quit IRC05:48
*** victor_lowther__ has joined #tripleo05:49
*** pensu has joined #tripleo06:07
*** vdreamarkitex has quit IRC06:08
*** vdreamarkitex has joined #tripleo06:09
*** mrunge has joined #tripleo06:10
*** ramishra has joined #tripleo06:12
*** ramishra has quit IRC06:17
*** pradeep has quit IRC06:22
*** andreaf has quit IRC06:23
*** pradeep has joined #tripleo06:26
*** ramishra has joined #tripleo06:28
*** k4n0 has joined #tripleo06:37
*** ramishra has quit IRC06:38
*** jprovazn has joined #tripleo06:43
*** jtomasek has joined #tripleo06:44
*** rdopieralski has joined #tripleo06:46
*** dtantsur|afk is now known as dtantsur06:56
*** marun has quit IRC06:57
openstackgerritA change was merged to openstack-infra/tripleo-ci: Add TRIPLEO_DEBUG var for enabling debug logging
*** pensu1 has joined #tripleo07:07
*** pensu has quit IRC07:09
*** pensu1 has quit IRC07:11
shadowerlsmola: ping07:12
*** ifarkas has joined #tripleo07:12
*** jcoufal has joined #tripleo07:12
*** Chao_Yan has quit IRC07:15
openstackgerritA change was merged to openstack/os-cloud-config: Updated from global requirements
*** pensu has joined #tripleo07:27
openstackgerritA change was merged to openstack/tuskar-ui: Set pki_setup False
openstackgerritA change was merged to openstack/tuskar-ui: Stop using intersphinx
*** gfidente has joined #tripleo07:33
*** saurabhs has left #tripleo07:35
*** pradeep has quit IRC07:36
*** pradeep has joined #tripleo07:44
*** jcoufal has quit IRC07:46
*** jcoufal has joined #tripleo07:46
lsmolashadower: pong07:47
shadowerlsmola: hey, remember how you sent peter the list of elements necessary to build the swift images?07:48
shadowerlsmola: do you remember the link? I can't find it07:48
shadowertesting the block storage templates now and want to do the object store after07:49
pensuHi all, I have been trying to work with Ironic in RDO instack tripleo is the error I am getting....07:49
pensu$ ironic node-list07:49
pensupublicURL endpoint for baremetal service not found07:49
pensuAny idea which endpoint I need to fix?07:49
*** pblaho has joined #tripleo07:50
shadowerdtantsur: ^ any ideas?07:50
StevenKpensu: If nova baremetal-node-list works, then it's using nova-baremetal and not ironic07:51
shadoweromg am I seriously that daft?07:51
shadowerlsmola: I am! Thanks!07:52
dtantsurpensu, did you set endpoint in keystone?07:55
StevenKdevtest will do that for you if USE_IRONIC is not 007:57
StevenKAnd its default is 107:57
dtantsurright, I didn't notice it's instack07:58
dtantsurI'd still double-check keystone endpoints07:59
*** rushiagr_away is now known as rushiagr08:00
*** jp_at_hp has joined #tripleo08:01
d0ugalThe nodes.csv file, is that how the Tuskar UI fines the baremetal vms?08:05
*** dkehn has quit IRC08:05
* d0ugal is watching tzumainn's demo08:06
*** dkehn has joined #tripleo08:07
*** yuanying has quit IRC08:08
*** jistr has joined #tripleo08:09
*** morgan_remote_ has quit IRC08:10
pensuStevenK: yeah, looks like it's using baremetal only. Do we support ironic?08:10
tchaypowho is we?08:11
*** athomas has joined #tripleo08:11
*** derekh has joined #tripleo08:13
pensutchaypo: sorry, I meant does tripleo support ironic?08:13
tchaypohrm. 4 hours later and my undercloud is still CREATE_IN_PROGRESS08:13
tchaypothat seems suboptimal08:13
tchaypopensu: use - as StevenK said, just set USE_IRONIC to non-008:14
*** lucasagomes has joined #tripleo08:15
derekhtchaypo: your undercloud thats still CREATE_IN_PROGRESS was the image built in the last day or so ?08:15
tchaypoSep 25 06:19 undercloud.qcow208:15
tchaypoyep, 3 hours ago08:16
derekhtchaypo: could this be your problem
uvirtbotLaunchpad bug 1373430 in horizon "Error while compressing files" [Critical,In progress]08:16
tchaypoderekh: i love you08:16
tchaypoI always feel guilty when people hand me answers like this08:16
tchaypoI feel like I chated and mised an opportunity to learn something, or something like that.08:16
tchaypoI think it's protestant-work-ethic guilt08:17
derekhShould I just let you find the horizon patch yourself so ?08:17
tchaypooh no08:20
tchaypoa bit of guilt is a wonderfulbit of relaxation08:21
tchaypo123779 is the patch, right?08:21
derekhtchaypo: yup08:23
* tchaypo pulls patch08:24
*** pradeep has quit IRC08:25
* shadower makes sure not to rebuild his images08:25
tchaypowell, while that's happening, I'm just going to put the laptop down ad watch people playing squash08:26
*** bvivek has quit IRC08:29
*** pradeep has joined #tripleo08:34
*** andreaf_ is now known as andreaf08:39
*** bvivek has joined #tripleo08:44
*** openstackgerrit has quit IRC08:48
*** rushiagr is now known as rushiagr_away08:52
*** untriaged-bot has joined #tripleo09:00
untriaged-botUntriaged bugs so far:09:00
uvirtbotLaunchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress]09:00
uvirtbotLaunchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New]09:00
uvirtbotLaunchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New]09:00
uvirtbotLaunchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New]09:00
*** untriaged-bot has quit IRC09:00
*** lazy_prince has quit IRC09:08
*** Dafna has joined #tripleo09:19
*** pradeep has quit IRC09:22
*** pensu has quit IRC09:22
*** pradeep has joined #tripleo09:23
*** pensu has joined #tripleo09:23
*** dsneddon has quit IRC09:24
*** dsneddon has joined #tripleo09:25
*** dtantsur is now known as dtantsur|brb09:25
*** killer_prince has joined #tripleo09:26
*** killer_prince is now known as lazy_prince09:27
*** lazy_prince has quit IRC09:48
*** lazy_prince has joined #tripleo09:48
*** pradeep1 has joined #tripleo09:55
*** pradeep has quit IRC09:56
*** rushiagr_away is now known as rushiagr09:56
*** loki184 has joined #tripleo10:07
*** mrunge has quit IRC10:16
*** yamahata_ has quit IRC10:20
*** akrivoka has joined #tripleo10:23
*** Azher has quit IRC10:38
*** bvivek has quit IRC10:39
*** pradeep1 has quit IRC10:42
*** pradeep has joined #tripleo10:42
*** pradeep has quit IRC10:45
*** dtantsur|brb is now known as dtantsur10:48
*** ramishra has joined #tripleo10:49
*** pensu1 has joined #tripleo10:49
*** pensu has quit IRC10:50
*** pradeep has joined #tripleo10:52
*** pradipta_away has quit IRC10:57
*** bvivek has joined #tripleo10:57
*** pensu1 has quit IRC11:13
*** pradeep has quit IRC11:17
*** pradeep1 has joined #tripleo11:17
*** lucasagomes is now known as lucas-hungry11:20
*** loki184 has quit IRC11:35
*** jang has quit IRC11:35
*** jang has joined #tripleo11:39
*** pradeep1 has quit IRC11:41
*** regebro has quit IRC11:44
*** nikunj2512 has quit IRC11:44
*** pradeep has joined #tripleo11:44
*** denis_makogon has joined #tripleo11:45
*** bvivek has quit IRC11:51
*** denis_makogon has left #tripleo11:51
*** bvivek has joined #tripleo11:52
*** julim has joined #tripleo11:55
*** jang has quit IRC11:57
*** jang has joined #tripleo11:58
*** jang1 has joined #tripleo12:01
*** regebro has joined #tripleo12:04
*** zz_naotok is now known as naotok12:07
*** pradeep has quit IRC12:09
*** morazi has joined #tripleo12:12
*** tzumainn has joined #tripleo12:14
*** rhallisey has joined #tripleo12:21
*** jdob has joined #tripleo12:21
*** rushiagr is now known as rushiagr_away12:26
*** dprince has joined #tripleo12:26
*** rushiagr_away is now known as rushiagr12:28
*** sdake has quit IRC12:30
*** sdake has joined #tripleo12:31
*** lucas-hungry is now known as lucasagomes12:37
*** akrivoka has quit IRC12:38
*** akrivoka has joined #tripleo12:47
*** naotok is now known as zz_naotok12:48
*** regebro has quit IRC12:51
*** pensu has joined #tripleo12:51
*** regebro has joined #tripleo12:54
*** andreaf is now known as andreaf_12:59
*** noslzzp has joined #tripleo12:59
*** regebro has quit IRC13:08
*** regebro has joined #tripleo13:09
*** pblaho has quit IRC13:18
*** bvivek has quit IRC13:33
*** ramishra has quit IRC13:46
*** ramishra has joined #tripleo13:47
*** sballe_ has joined #tripleo13:54
*** pcrews has joined #tripleo13:56
*** regebro has quit IRC13:57
*** morgan_remote_ has joined #tripleo13:59
*** nosnos has quit IRC14:00
*** nosnos has joined #tripleo14:00
*** nosnos has quit IRC14:04
*** jprovazn has quit IRC14:05
*** ramishra has quit IRC14:11
*** pensu has quit IRC14:12
*** mestery has quit IRC14:14
*** mestery has joined #tripleo14:14
*** openstackgerrit has joined #tripleo14:15
*** ChuckC has joined #tripleo14:19
*** ChuckC_ has quit IRC14:20
*** ramishra has joined #tripleo14:20
*** bvivek has joined #tripleo14:22
*** ChuckC has quit IRC14:25
*** pensu has joined #tripleo14:26
*** sballe_ has quit IRC14:30
*** ramishra has quit IRC14:30
*** ramishra has joined #tripleo14:31
*** openstackgerrit has quit IRC14:40
*** openstackgerrit has joined #tripleo14:41
openstackgerritLucas Alvares Gomes proposed a change to openstack-infra/tripleo-ci: Update virsh ssh commands to support setting the locale
*** ramishra has quit IRC14:46
*** pensu has quit IRC14:51
*** andreaf_ is now known as andreaf14:52
*** ramishra has joined #tripleo14:53
*** untriaged-bot has joined #tripleo15:00
untriaged-botUntriaged bugs so far:15:00
uvirtbotLaunchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress]15:00
*** dtantsur is now known as dtantsur|brb15:00
uvirtbotLaunchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New]15:00
uvirtbotLaunchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New]15:00
uvirtbotLaunchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New]15:00
*** untriaged-bot has quit IRC15:00
*** andreaf is now known as andreaf_15:01
*** akrivoka has quit IRC15:02
*** bcrochet has quit IRC15:04
*** beekneemech is now known as bnemec15:05
*** pensu has joined #tripleo15:05
*** bcrochet has joined #tripleo15:09
*** marun has joined #tripleo15:10
*** ramishra has quit IRC15:10
*** jang1 has quit IRC15:11
*** jang has quit IRC15:11
*** ChuckC has joined #tripleo15:12
*** jang has joined #tripleo15:14
*** dshulyak_ has joined #tripleo15:20
*** bvivek has quit IRC15:22
*** rdopieralski has quit IRC15:22
*** yamahata_ has joined #tripleo15:23
*** athomas has quit IRC15:24
*** athomas has joined #tripleo15:27
*** andreaf has joined #tripleo15:30
*** rwsu has quit IRC15:33
*** andreaf has quit IRC15:37
*** andreaf has joined #tripleo15:38
*** k4n0 has quit IRC15:43
*** pensu has quit IRC15:45
*** rwsu has joined #tripleo15:45
*** morazi has quit IRC15:45
*** morazi has joined #tripleo15:48
*** slagle_ has joined #tripleo15:52
*** slagle_ has quit IRC15:52
*** slagle has quit IRC15:53
*** slagle has joined #tripleo15:54
*** jistr has quit IRC15:54
*** cwolferh_ has joined #tripleo15:57
openstackgerritRyan Brady proposed a change to openstack/diskimage-builder: Enable RHEL Registration
*** ChuckC has quit IRC16:06
openstackgerritRyan Brady proposed a change to openstack/diskimage-builder: Enable RHEL Registration
*** dtantsur|brb is now known as dtantsur16:08
openstackgerritRyan Brady proposed a change to openstack/diskimage-builder: Enable RHEL Registration
*** rhallisey has quit IRC16:10
*** morgan_remote_ has quit IRC16:10
*** rushiagr is now known as rushiagr_away16:11
*** lsmola has quit IRC16:12
*** Dafna has quit IRC16:14
*** jp_at_hp has quit IRC16:19
*** jml has quit IRC16:22
*** dshulyak_ has quit IRC16:23
*** jml has joined #tripleo16:26
*** Azher has joined #tripleo16:28
*** pensu has joined #tripleo16:28
*** jml has quit IRC16:31
*** jml has joined #tripleo16:33
*** openstackgerrit has quit IRC16:33
*** cody-somerville has quit IRC16:33
*** dshulyak_ has joined #tripleo16:34
*** cody-somerville has joined #tripleo16:36
*** cody-somerville has joined #tripleo16:36
*** athomas has quit IRC16:38
*** dshulyak_ has quit IRC16:39
*** dshulyak_ has joined #tripleo16:41
*** pradeep has joined #tripleo16:42
*** jml has quit IRC16:43
*** dshulyak_ has quit IRC16:45
*** derekh has quit IRC16:48
*** lucasagomes has quit IRC16:48
*** rushiagr_away is now known as rushiagr16:48
*** jml has joined #tripleo16:49
*** openstackgerrit has joined #tripleo16:53
*** rhallisey has joined #tripleo16:54
*** rhallisey has quit IRC16:56
*** rhallisey has joined #tripleo16:56
*** dshulyak_ has joined #tripleo16:59
*** jdob has quit IRC17:00
*** jdob has joined #tripleo17:00
*** pradeep has quit IRC17:01
*** dshulyak_ has quit IRC17:02
*** dshulyak_ has joined #tripleo17:06
*** michchap has quit IRC17:09
*** pensu has quit IRC17:10
*** marun has quit IRC17:10
*** tzumainn has quit IRC17:16
*** tzumainn has joined #tripleo17:23
*** grauwulf has quit IRC17:23
*** pensu has joined #tripleo17:25
openstackgerritNicholas Randon proposed a change to openstack/tripleo-image-elements: Fix RabbitMQ element clustering start and stop
*** jdob has quit IRC17:29
*** jdob has joined #tripleo17:30
*** ChuckC has joined #tripleo17:30
*** pensu has quit IRC17:30
*** grauwulf has joined #tripleo17:31
*** openstackgerrit has quit IRC17:33
*** sballe_ has joined #tripleo17:36
*** pradeep has joined #tripleo17:38
*** jtomasek has quit IRC17:38
*** tzumainn has quit IRC17:39
*** openstackgerrit has joined #tripleo17:39
*** morgan_remote_ has joined #tripleo17:40
*** pcrews has quit IRC17:41
*** tzumainn has joined #tripleo17:42
*** pcrews has joined #tripleo17:42
*** ifarkas has quit IRC17:43
*** pensu has joined #tripleo17:45
*** gfidente has quit IRC17:47
*** jehb has joined #tripleo17:48
*** penick has joined #tripleo17:57
*** Daviey has quit IRC18:07
*** pradeep has quit IRC18:08
*** jang1 has joined #tripleo18:09
*** Daviey has joined #tripleo18:19
*** rushiagr is now known as rushiagr_away18:20
*** dtantsur is now known as dtantsur|afk18:22
*** dprince has quit IRC18:24
*** lazy_prince is now known as killer_prince18:25
openstackgerritErik Colnick proposed a change to openstack/tripleo-specs: TripleO DVR enablement
openstackgerritErik Colnick proposed a change to openstack/tripleo-specs: TripleO DVR enablement
*** pcrews has quit IRC18:32
*** sballe_ has quit IRC18:36
*** sballe_ has joined #tripleo18:37
*** jcoufal has quit IRC18:40
*** dshulyak_ has quit IRC18:45
*** cody-somerville has quit IRC18:52
*** cody-somerville has joined #tripleo18:54
*** andreaf has quit IRC18:59
*** andreaf has joined #tripleo18:59
*** tzumainn has quit IRC19:08
*** tzumainn has joined #tripleo19:10
*** pcrews has joined #tripleo19:12
*** Azher has quit IRC19:16
*** cwolferh_ has quit IRC19:25
*** jehb has quit IRC19:26
*** morgan_remote_ has quit IRC19:50
openstackgerritA change was merged to openstack/python-tuskarclient: Stop using intersphinx
openstackgerritA change was merged to openstack/tuskar: Stop using intersphinx
*** jml has quit IRC20:02
*** yamahata_ has quit IRC20:02
*** radez has joined #tripleo20:05
*** jml has joined #tripleo20:06
*** jml has joined #tripleo20:06
*** radez is now known as radez_g0n320:08
*** radez_g0n3 is now known as radez20:10
*** radez is now known as radez_g0n320:11
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Nothing to see here
*** andreaf has quit IRC20:23
*** andreaf has joined #tripleo20:23
*** cwolferh has joined #tripleo20:27
*** pcrews has quit IRC20:34
openstackgerritErik Colnick proposed a change to openstack/tripleo-specs: TripleO DVR enablement
*** untriaged-bot has joined #tripleo21:00
untriaged-botUntriaged bugs so far:21:00
uvirtbotLaunchpad bug 1373063 in tripleo "port not added to bridge after a reboot" [Undecided,In progress]21:00
uvirtbotLaunchpad bug 1369256 in tripleo "ubuntu: hash sum mismatch when installing packages" [Undecided,New]21:00
uvirtbotLaunchpad bug 1366164 in tripleo "Add domain roles for admin user" [Undecided,New]21:00
uvirtbotLaunchpad bug 1367742 in tripleo "keepalived brings the VIP up in multiple places on a controller reboot" [Undecided,New]21:00
*** untriaged-bot has quit IRC21:00
*** jdob has quit IRC21:02
*** pensu has quit IRC21:04
*** sdake_ has joined #tripleo21:05
*** pcrews has joined #tripleo21:05
*** jml has quit IRC21:19
*** jml has joined #tripleo21:20
jogolifeless: I am happy to +2 if it gets a few +1s from tripleo cores21:23
SpamapSjogo: I +1'd  ;)21:29
jogoSpamapS lifeless: thanks21:30
*** jang1 has quit IRC21:40
*** sdake_ has quit IRC21:41
*** ccrouch has quit IRC21:44
tchaypomorganfainberg: lifeless: I need to take the cats to the vet this morning, probably not properly around for another 2.5 hours yet. It looks like the second undercloud build I kicked off last night also got bitten by the horizon bug, so I'll need to fix that when I come back21:55
tchaypoin short I don't expect much progress today21:55
*** ccrouch has joined #tripleo22:04
*** morgan_remote_ has joined #tripleo22:12
*** penick has quit IRC22:18
SpamapShaha.. I can't tell if this is real or not
lifelessSpamapS: tis real22:20
lifelesstchaypo: ok22:21
SpamapSlifeless: there's so much snark.. even the logo.. the kitchen sink.. it's brilliant22:23
lifelessSpamapS: I know right :022:27
bkeroIt has roughly a useless amount of snark compared to systemd.22:30
*** penick has joined #tripleo22:31
morganfainbergtchaypo, lifeless, sounds good. still battleing the gate for the index update, though you should be able to apply it directly22:34
*** andreaf has quit IRC22:38
*** andreaf has joined #tripleo22:38
*** jml has quit IRC22:52
*** penick has quit IRC22:53
*** jml has joined #tripleo22:55
*** sdake_ has joined #tripleo23:00
*** yuanying has joined #tripleo23:10
*** penick has joined #tripleo23:11
*** sdake has quit IRC23:12
*** ccrouch has quit IRC23:14
*** ChuckC has quit IRC23:14
*** ccrouch has joined #tripleo23:34
*** sdake_ has quit IRC23:34
*** sdake has joined #tripleo23:36
*** sballe_ has quit IRC23:39
*** boris-42 has quit IRC23:47
*** pradeep has joined #tripleo23:48
*** boris-42 has joined #tripleo23:48

Generated by 2.14.0 by Marius Gedminas - find it at!