Sunday, 2014-08-10

*** noslzzp has joined #tripleo01:14
*** pcrews has quit IRC01:43
*** lblanchard has quit IRC02:24
*** noslzzp has quit IRC02:24
*** adam_g has quit IRC02:43
*** adam_g has joined #tripleo02:44
*** adam_g has joined #tripleo02:44
*** ddieterl_ has quit IRC02:54
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/135126303:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/135450903:00
uvirtbotLaunchpad bug 1351263 in tripleo "Seed init failed, nova list returned 503" [Undecided,New]03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784803:00
uvirtbotLaunchpad bug 1354509 in tripleo "Glance requirements.txt for osprofiler too loose" [Undecided,Confirmed]03:00
uvirtbotLaunchpad bug 1347848 in tripleo "/etc/hosts information about cloud hsots lost on reboot" [Undecided,In progress]03:00
*** untriaged-bot has quit IRC03:00
*** eghobo has joined #tripleo03:17
*** eghobo has quit IRC04:03
*** eghobo has joined #tripleo04:23
*** cwolfe__ has joined #tripleo04:33
*** cwolferh_ has quit IRC04:37
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Add a second ssh key for tchaypo  https://review.openstack.org/11311005:15
*** dshulyak_ has joined #tripleo05:17
*** eghobo has quit IRC05:29
lifelessadam_g: you're welcome :)05:32
openstackgerritA change was merged to openstack/tripleo-heat-templates: Fix overcloud controller scaling  https://review.openstack.org/11253805:45
tchaypolifeless: so I've been poking at the bastions05:50
tchaypoIf I haven't missed anything, it looks like the things we might need to update aren't in that repo - we just need to get that repo up-to-date on hp2. It looks like all the config it's going to use lives one level higher05:51
lifelessthats the intent yes05:51
lifelessbut the repo refs file is probably still in the repo on hp205:51
lifelessso checking for git diff output would be a good starting point05:51
tchaypoit's still in the repo on hp1 too *thinks more* wait, you mean, look for what's changed on hp2 between the git revision and what's live05:53
tchaypohp2 has a longer list of patches, which look roughly like the list of things we'd need to get the vlan stuff working05:55
tchayposo I'm guessing we want to commit then; then merge with what's on hp1?05:55
*** vinsh has quit IRC06:04
*** cwolferh has joined #tripleo06:06
*** cwolfe__ has quit IRC06:10
*** eghobo has joined #tripleo06:19
*** akuznetsov has joined #tripleo07:03
*** eghobo has quit IRC07:45
*** tchaypo has quit IRC07:54
*** tchaypo has joined #tripleo07:54
*** akuznetsov has quit IRC08:05
*** akuznetsov has joined #tripleo08:20
*** akuznetsov has quit IRC08:25
*** akuznetsov has joined #tripleo08:27
*** akuznetsov has quit IRC08:31
*** jmoffitt has quit IRC08:47
*** jmoffitt has joined #tripleo08:48
*** untriaged-bot has joined #tripleo09:00
untriaged-botUntriaged bugs so far:09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/135126309:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/135450909:00
uvirtbotLaunchpad bug 1351263 in tripleo "Seed init failed, nova list returned 503" [Undecided,New]09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784809:00
uvirtbotLaunchpad bug 1354509 in tripleo "Glance requirements.txt for osprofiler too loose" [Undecided,Confirmed]09:00
uvirtbotLaunchpad bug 1347848 in tripleo "/etc/hosts information about cloud hsots lost on reboot" [Undecided,In progress]09:00
*** untriaged-bot has quit IRC09:00
*** jang1 has quit IRC10:25
*** jang has quit IRC10:25
*** jang has joined #tripleo10:25
*** jang has quit IRC10:28
*** rlandy has joined #tripleo10:30
*** jang has joined #tripleo10:31
*** jang has quit IRC10:32
*** jang has joined #tripleo10:33
*** jang has quit IRC10:36
*** jang has joined #tripleo10:39
*** jang has quit IRC10:40
*** jang has joined #tripleo10:41
*** akuznetsov has joined #tripleo10:55
*** jang1 has joined #tripleo12:12
*** dshulyak_ has quit IRC12:33
*** boris-42 has quit IRC14:02
*** dshulyak_ has joined #tripleo14:04
*** noslzzp has joined #tripleo14:57
*** untriaged-bot has joined #tripleo15:00
untriaged-botUntriaged bugs so far:15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/135126315:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/135450915:00
uvirtbotLaunchpad bug 1351263 in tripleo "Seed init failed, nova list returned 503" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784815:00
*** untriaged-bot has quit IRC15:00
uvirtbotLaunchpad bug 1354509 in tripleo "Glance requirements.txt for osprofiler too loose" [Undecided,Confirmed]15:00
uvirtbotLaunchpad bug 1347848 in tripleo "/etc/hosts information about cloud hsots lost on reboot" [Undecided,In progress]15:00
*** dshulyak_ has quit IRC15:03
*** noslzzp has quit IRC15:10
*** Leslie_ has joined #tripleo15:14
Leslie_Hello Everyone15:14
Leslie_I'm studying tripleo for now15:14
Leslie_I got one question:15:15
Leslie_when building seed VM, /etc/os-collect-config.conf only has default command, but after VM boot, seems like some applications append collector=heat_local, then seems like cfn-init-data are got triggered15:16
Leslie_so does anyone know the detail work flow15:16
Leslie_I'm curious which applications read /var/lib/heat-cfn-tools/cfn-init-data at when15:18
*** rlandy has quit IRC16:35
*** eghobo has joined #tripleo16:37
*** eghobo has quit IRC16:48
*** rwsu has quit IRC16:50
*** dshulyak_ has joined #tripleo17:37
*** dshulyak_ has quit IRC17:53
*** noslzzp has joined #tripleo17:55
*** noslzzp has quit IRC18:07
*** noslzzp has joined #tripleo18:16
*** noslzzp has quit IRC18:28
*** pensu has joined #tripleo19:08
*** penick has joined #tripleo19:15
*** jml has quit IRC19:18
*** jml has joined #tripleo19:19
*** dshulyak_ has joined #tripleo19:26
*** dshulyak_ has quit IRC19:33
*** dshulyak_ has joined #tripleo19:34
*** pensu has quit IRC19:34
*** pensu has joined #tripleo19:35
*** akuznetsov has quit IRC19:37
*** eghobo has joined #tripleo19:41
*** eghobo has quit IRC19:41
*** eghobo has joined #tripleo19:42
*** pensu has quit IRC19:50
*** dshulyak_ has quit IRC19:54
*** dshulyak_ has joined #tripleo19:59
*** dshulyak_ has quit IRC19:59
*** openstack has joined #tripleo20:17
*** eguz has joined #tripleo20:36
*** eghobo has quit IRC20:38
*** penick has quit IRC20:40
*** robsparker has quit IRC20:42
*** robsparker has joined #tripleo20:57
*** untriaged-bot has joined #tripleo21:00
untriaged-botUntriaged bugs so far:21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/135126321:00
uvirtbotLaunchpad bug 1351263 in tripleo "Seed init failed, nova list returned 503" [Undecided,New]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/135450921:00
uvirtbotLaunchpad bug 1354509 in tripleo "Glance requirements.txt for osprofiler too loose" [Undecided,Confirmed]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784821:00
uvirtbotLaunchpad bug 1347848 in tripleo "/etc/hosts information about cloud hsots lost on reboot" [Undecided,In progress]21:00
*** untriaged-bot has quit IRC21:00
*** boris-42 has joined #tripleo21:15
tchaypomorning lifeless21:19
openstackgerritSteve Baker proposed a change to openstack/tripleo-heat-templates: Switch to heat_template_version: 2013-05-23  https://review.openstack.org/11314821:21
lifelesstchaypo: morning!21:25
lifelesstchaypo: lets deploy a cloud21:25
lifelesstchaypo: commit what?21:30
tchayporepo_refs.yaml - it looks like we need to merge between current repo state on hp2, current working state on hp2, and current repo state on hp121:34
tchaypowell, lots of stuff on hp1 needs to be updated on hp2, but it looks like we need to also merge in the currently-live repo_refs.yaml, which looks to me as though it's been set up for the vlan work we've been doing21:35
tchaypo*thinks* this would imply that we want to have consistent repo_refs across all regions. I'm pretty sure that's accurate - I think regional differences are going to be dealt with as config elsewhere, not as which patches are being applied?21:36
*** jang1 has quit IRC21:48
lifelesstchaypo: config is patches is config21:49
lifelesstchaypo: repo_refs.yaml in the tripleo-end-to-end repo in hp1 is no longer relevant21:49
lifelesstchaypo: in the same place on hp2 its relevant, because thats how the old tripleo-end-to-end works21:50
tchaypothe hp2 version looks like what we want21:50
lifelesstchaypo: we need to migrate it out of there into the same structure as hp1 uses21:50
lifelesstchaypo: and then reconcile the two sets of patches21:50
tchaypospecifically - the format for repos: has changed, the gerrit_refs structure looks the same21:51
lifelessalso where we store the refs21:52
lifelesslike the file path on disk21:52
lifelesstotally different21:52
tchayposo: I'll commit what's on hp2 now, then do another commit with the same content but the new format for repos, then add a new upstream pointing at the same place as the hp1 upstream, then merge..21:52
lifelessnooo21:52
lifelessplease21:52
lifelesslets get consensus on next step first21:53
lifelesswhats the file path to repo_refs.yaml on hp121:53
lifelesswhats the file path to repo_refs.yaml on hp221:53
tchaypoboth have /home/shared/tripleo-end-to-end/repo_refs.yaml; hp1 also has /home/shared/repo_refs.yaml21:54
tchaypoagree about getting consensus first. above was me trying to start that by proposing a series of steps. maybe i made it sound like I'd already reached a decision?21:54
lifelessyah21:55
lifelessok so /home/shared/tripleo-end-to-end/repo_refs.yaml is irrelevant in hp121:55
lifelesssee the README - its an example file21:55
tchaypothe ../repo_refs.yaml on hp1 is newer by about 90 minutes21:55
tchaypoah right. So where I was speculating about whether we *want* to have the same repo_refs in all regions, the answer is "no"21:56
tchaypowhich I think is what you meant by "config is patches is config"21:56
lifelesssome of our config21:56
lifelessis stored in tripleo-image-elements21:56
tchaypohence upating the patches to use for that repo == updating the config to use for this region21:57
lifelessyes21:57
tchaypoI'd like to git init /home/shared; give it a .gitignore of .*, then manually add in config files like repo_refs.yaml so that I can track and revert my changes. any objection?22:01
lifelessuhm22:02
lifelessyeah22:02
lifelessif we want to version track meta config like this22:02
lifelesswe shouldn't make it a containing dir of other things22:02
lifelessas thats an awesome way to confuse people22:03
tchaypobetter to make a subdir called regional_configs (or something), track that, and make the files at the top level symlinks into the repo?22:03
lifelessI've no objection to having local history of this stuff (though by definition, since its the thing tracking what we haven't committed, it can't ever be CId etc)22:03
lifelesstchaypo: we *have* regional configs.22:04
lifelesstchaypo: in tie22:04
tchaypoyou're talking about elements/tripleo-cd/configs? the files with "hp1" in the name?22:05
lifelessyes22:05
lifelessand the ones with hp2 etc22:05
tchaypohp1 hasn't been updated to have the commit that added those yet22:06
lifelessthey haven't landed yet22:06
lifelessand hp1 doesn't need that config22:06
tchaypoI think that if we tracked repo_refs.yaml in there it would lead to what I'm going to call "fun" but which a person more interested in a sane working system might call "not working"22:07
lifelesstchaypo: so, if you want to vc these files - and like I say, I've no objection, I would be ok with a new subdir 'repo_refs' and put the file in there22:07
lifelessright22:08
lifelessrepo_refs can't be sensibly versioned in the product trees22:08
lifelesssince its the 'and this is why our product doesn't work right now' set of patches22:08
* tchaypo notes the rollup_1_2_3_4 branch names22:08
* tchaypo approves of how easy this makes it to compare state between regions22:08
lifelessso with your new understanding, whats your proposed action plan?22:10
tchaypobtw I took the liberty of adding some aliases to ~/.gitconfig on both bastions22:10
tchaypocreate /home/shared/repo_refs, copy repo_refs.yaml in, commit. Modify to new format based on reading hp1's sample, commit, ask for review. Add new remote to hp2, fetch changes in the new remote, do a diff and make sure it looks as I expect (ie, should be identical to " git diff -r f5fc0e7b -r fbdbef8" on hp1, except for the changes to repo_refs.yaml on disk)22:12
tchaypono wait - once I've copied repo_refs off to the new location I can revert the one in tripleo-end-to-end to match the repo, which will simplify the diff.22:13
lifelessby simplify you mean 'should be no diff' ?22:13
tchaypoAt that point I'm going to pause and re-read the readme and take a look at the config files under /home/shared22:13
tchaypoyes, should be no diff :)22:13
lifelessso that sounds fine to me; please change hp1 in the same way of course22:23
tchaypowhat, we want consistency? *gasp*22:27
tchaypoI know we have some puppet configs, which I *think* touch on these machines, but I haven't looked.22:29
lifelessnope22:31
lifelesswe aren't operating these with  puppet at all22:31
tchayponope, looking at site.pp in the config repo it doesn't seem to touch these machines22:31
tchaypoand there's no /etc/puppet22:31
*** eguz has quit IRC22:32
lifelesstchaypo: so while you mull on that22:55
lifelessI'm going to revert the HOT patches22:55
lifelessah stevebaker has proposed it. Yay22:56
stevebakerlifeless: yes, take a look22:57
stevebakercheck looks happy22:57
lifelesstchaypo: I'm taking a lock on hp123:00
tchaypoack23:01
lifelesstchaypo: the state there seems inconsistent - there is a repo_refs subdir now, but still a repo_refs.yaml above it23:01
lifelesstchaypo: where were you up to ?23:01
tchaypoyes, I got interrupted partway through making that repo23:01
lifelesstchaypo: ok23:01
lifelesstchaypo: I will finish up, since I am trying to recover this :)23:02
lifelesstchaypo: when you say interrupted23:02
tchayporepo_refs now has git history23:02
lifelesstchaypo: what specifically do you mean ?23:02
tchaypoif you make changes in the parent I can copy them in, make a new commit, and then pivot to a symlink23:03
tchaypoor would you prefer to pivot first?23:03
lifelesstchaypo: what symlink?23:03
tchaypointerrupted by cats walking past23:03
tchayporepo_refs.yaml -> repo_refs/repo_refs.yaml23:03
lifelesstchaypo: I had $ git add repo_refs.yaml23:03
lifelessfatal: Unable to create '/home/shared/repo_refs/.git/index.lock': File exists.23:03
lifelesstchaypo: which makes me think that you didn't down tools after I took the lock :)23:03
tchaypoOh. I thought you were going to be working in the parent directory and the repo_refs subdir would be out of your way :(23:04
lifelessno23:04
lifelessif we're moving stuff to a subdir23:04
* tchaypo backs right off23:04
lifelesswe're moving stuff to a subdir23:04
lifeless /home/shared/repo_refs.yaml{,.variables} deleted23:05
lifelessupdating overcloud now23:07
lifelesstchaypo: you have the lock on hp2 though23:07
tchaypowhat generates repo_refs.yaml.variables? I thought it must be in the loop at the bootom of seed.hp2.tripleo.org.rc but I can't see it writing anything23:08
tchaypoand seed.hp1.tripleo.org.rc doesn't seem to have that loop anyway23:08
lifelesstchaypo: you need to read the tripleo-end-to-end README more closely23:08
tchaypothe README (on hp1) says to source the file but doesn't mention it being created.23:09
lifelesstchaypo: thats true23:10
tchaypoFrom the placement I'm guessing that prep_source_repos must create it23:10
lifelesstchaypo: :)23:10
lifelesstchaypo: yes, perhaps that should be added to the README23:10
lifelessoh yay23:11
lifeless2014-08-10 23:11:28.647 1946 WARNING os_collect_config.cfn [-] 400 Client Error: InvalidParameterValue23:11
lifelessstevebaker: WAT23:11
stevebaker?23:12
lifelessnvm, I think I'm confused23:12
lifelessseparately23:12
lifelessI just had keepalived spit the dummy at me23:13
lifelessand I'm thoroughly confused as to why23:13
lifeless+ echo 'Configuration for keepalived is not yet known. Refusing to proceed.'23:13
lifelessstevebaker: actually no - I think this cfn thing matters23:13
lifelessstevebaker: the collector is failing23:13
lifelessstevebaker: on this live cloud I just stack-updated23:14
stevebakerwhat did the update include?23:14
lifelessstevebaker: well, the HOT changes23:14
lifelessstevebaker: and I added a Debug parameter23:15
lifelessso I can get debug output from the 3 control plane nodes23:15
lifelesset voila23:16
lifeless/lib/python2.7/site-packages/heat/engine/parameters.py\", line 480, in _validate_user_parameters\n    raise exception.UnknownUserParameter(key=param)\n\nUnknownUserParameter: The Parameter (Debug) was not defined in template.\n", "type": "UnknownUserParameter"}, "title": "Bad Request"} to_json /opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/common/serializers.py:4323:16
lifelessthough I'm stumped why that would be throwing an error on cfn metadata pollin23:16
lifeless# heat stack-list23:16
lifelessERROR: The Parameter (Debug) was not defined in template.23:16
lifelessstevebaker: So.... I think I have wedged this heat :)23:17
stevebakerproblem solved!23:17
stevebaker(bad taste joke)23:17
lifelessthe parameter is in the template I used to stack-update23:17
lifeless  Debug:23:17
lifeless    default: ''23:17
lifeless    description: Set to truthy to enable debugging on all services.23:17
lifeless    type: string23:17
lifelessI'm going to guess that there is a bug in stack-updates that add a parameter23:18
stevebakerlifeless: so the update is complete but metadata polling isn't working?23:18
stevebakerlifeless: or is the stack in a failed state23:19
lifelessstevebaker: I can't tell23:19
lifelessstevebaker: stack-list is broken23:19
lifelessstevebaker: see above23:19
stevebakerugh23:20
stevebakerI wish someone would properly deal to our validate flows23:21
lifelessstevebaker: https://bugs.launchpad.net/heat/+bug/135496223:22
uvirtbotLaunchpad bug 1354962 in heat "stack-update adding a parameter leads to heat stack-list breaking" [Undecided,New]23:22
lifelessstevebaker: I'm not sure that its a validate problem23:22
lifelessstevebaker: I'm going to guess that because the update hasn't completed23:22
lifelessstevebaker: but the parameters have been updated23:22
lifelessstevebaker: its trying to validate the old template with the new environment23:22
stevebakerlifeless: stack-list should always work though23:22
lifelessstevebaker: oh, I totally agree.23:23
lifelessok, so its now at update-failed23:24
lifelessI've disabled that check :)23:24
*** ci-overcloud has joined #tripleo23:28
ci-overcloud************** ci-overcloud complete status=1 ************23:28
*** ci-overcloud has quit IRC23:28
*** ci-overcloud has joined #tripleo23:29
ci-overcloud************** ci-overcloud complete status=1 ************23:29
*** ci-overcloud has quit IRC23:29
stevebakerlifeless: once we have functional tests in-tree and https://review.openstack.org/#/c/112938/ lands then we can fairly quickly write tests which verify all the combinations of template changes when stack updates23:30
lifelessnow I'm getting StackValidationFailed: Property error : NovaCompute0:23:30
lifelesskey_name Value must be a string23:31
stevebakerI would be very suprised if the transition to HOT would work with a stack-update. It didn't even remotely occur to me that it was a requirement23:33
lifelessshould we rollback the whole thing?23:34
stevebakerlifeless: is a rebuild an option?23:40
lifelessstevebaker: tear down folks production clouds on upgrade?23:41
lifelessstevebaker: [checking that that is what you mean]23:41
stevebakerok, that would be a no23:41
openstackgerritA change was merged to openstack/tripleo-specs: Also bind public services to a dedicated VIP  https://review.openstack.org/10015123:42
openstackgerritlifeless proposed a change to openstack/tripleo-specs: Tweak HAProxy spec to deal with mixed services.  https://review.openstack.org/10795623:42
*** xuhaiwei has joined #tripleo23:46
stevebakerlifeless: I don't see this working until the undercloud heat is upgradable23:47
lifelessstevebaker: it is upgradeable, using the seed heat.23:48
stevebakerlifeless: ...and then testing for and fixing this particular update bug23:49
openstackgerritA change was merged to openstack/tripleo-specs: Update the Tuskar storage spec to reference Heat  https://review.openstack.org/10768423:50
lifelessstevebaker: right23:50
lifelessstevebaker: so I think we need to rollback to pre the hot patches23:50
stevebaker;( yes23:50
lifelessre-propose all the pending work that was open when they landed23:50
lifelessok, so I'm going to take a breather, then come back to help23:51
*** noslzzp has joined #tripleo23:53
stevebakerlifeless: lets poke at it a little more. Can you paste your overcloud template?23:54
lifelessstevebaker: http://paste.ubuntu.com/8012166/23:54
tchaypolifeless: looking at the testenv.jsons - the details of the public_vlan doesn't seem to have been added to the spreadsheet23:56
lifelesstchaypo: its spread on two tabs23:56
lifelesstchaypo: internal and public23:56
tchaypooh23:57
tchaypofound it. hp2 is listed as tag 25 in the sheet but 1598 in the json23:57
tchaypo*updates sheet on assumption that json is correct*23:58
stevebakerlifeless: back in a bit23:58
tchaypothe json on hp1 has an "undercloud" key that's missing from hp223:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!