Monday, 2014-03-17

*** xuhaiwei has joined #tripleo00:04
* StevenK stabs the nvidia kernel module00:07
*** matsuhashi has joined #tripleo00:12
openstackgerritA change was merged to openstack/tripleo-incubator: Remove $(dirname $0) from docs.
tchaypoStevenK: do tell00:14
StevenKtchaypo: Display lockups with NVRM: Xid (0000:01:00): 8, Channel 0000000100:14
tchaypowhere are you seeing that?00:23
StevenKdmesg, before I rebooted00:23
StevenKWhich is the only way to get the card and the kernel module back into sync00:23
*** matsuhashi has quit IRC00:31
lifelessStevenK: what driver?00:35
lifelesswendar: on the offchance you're still up - you have mail00:38
StevenKlifeless: nvidia 304.8800:41
lifelessStevenK: hmm, I have radeon on this laptop. should I change?00:47
StevenKlifeless: Sorry, I was complaining about my desktop, not my laptop00:47
*** matsuhashi has joined #tripleo00:52
lifelessah, kk00:54
*** fandi has quit IRC00:58
tchayposlagle: I don't suppose you're around?01:02
xuhaiweilifeless: I am confused about this in the tripleo-incubator README, ' os-apply-config reads a JSON metadata file and generates templates', what are the 'templates' here?01:06
lifelessxuhaiwei: moustache templates01:07
xuhaiweiand ' moustache templates' are?01:07
lifelessxuhaiwei: moustache templates!01:07
*** CaptTofu has joined #tripleo01:12
*** CaptTofu has quit IRC01:17
xuhaiweilifeless: I think I have understood mustache templates, but what are these templates for? used by heat to do the configuration?01:25
lifelessxuhaiwei: by oac to capture the metadata json into config files01:26
*** CaptTofu has joined #tripleo01:26
xuhaiweioac read a JSON metadata file and generates templates, so the templates are the config files, right?01:27
tchaypoxuhaiwei: welcome back01:27
xuhaiweitchaypo: good morning, it's not morning in your place maybe01:29
StevenKNope, it's been the afternoon for 30 minutes01:29
xuhaiweihere is UTC+901:30
*** nosnos has joined #tripleo01:30
tchaypolifeless: I'm fairly certain that is just caused by using a version of pip <1.5 - is there anything we should do about this? I'm inclined to close it as not being our problem, but I'm wondering if we specify a version of pip somewhere..01:31
lifelessmordred: does pip 1.5+ work with openstack stuff ?01:32
mordredlifeless: it should01:32
mordredwe're not using it in the gate yet because of tox fail01:32
tchaypomordred: so I'm interpreting that as also meaning that you don't "pip install diskimage-builder" in the gate?01:33
lifelesstchaypo: thats correct01:34
xuhaiweilifeless: it's really vague here, after hearing your explanation, my understanding is oac uses JSON metadate to generate mustache templates, and the mustache templates are used by heat to do the configuration for openstack, am I right?01:34
tchaypoI'm going to go ahead and close01:35
tchaypohrm. Won01:35
lifelessxuhaiwei: no, heat doesn't do any configuration. its a web service. configuration happens locally.01:35
tchaypo"won't fix" or "invalid"?01:35
lifelessxuhaiwei: heat exports the metadata for the machine. oac turns that into files on disk01:36
tchaypoI think this is invalid - as far as i can tell, it's not a bug with DIB nor with tripleo01:36
xuhaiweilifeless: and the 'files' are used by openstack?01:36
*** shakamunyi has joined #tripleo01:38
tchaypoIf I understand oac correctly, the files it's generating will generally be the config file for the thing being configured01:38
tchaypoif you look at the examples in the README ( has a handy web version), you'll see that it generates /etc/mysql/mysql.conf01:39
xuhaiweitchaypo: for example, like this? connection = mysql://{{keystone.database.user}}:{{keystone.database.password}@{{}}/keystone01:42
tchaypoYep, that seems to be a mustache template. The README notes that there's an example tree at
tchaypo is a template, which looks like it should end up at /etc/keystone/keystone.conf01:47
xuhaiweiI found it, you mean the keystone.conf is generated by os-apploy-config? sounds reasonable01:51
xuhaiweibut why is there only configuration file for keystone?01:53
tchaypoThat particular example seems to be for the "keystone" element01:53
*** CaptTofu has quit IRC01:53
*** CaptTofu has joined #tripleo01:54
xuhaiweiok, that makes sense for me, thank you01:54
tchaypo"element" in this case being a thing diskimage-builder uses when building an disk image01:54
tchaypoIf you jump up a few levels you'll see that other elements have their own trees01:55
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Teach testenv about numbers and bridges
tchaypoeg for the debian-mirror element01:55
tchaypoI assume something must munge all these trees together to produce a single unified tree for o-a-c to run over01:55
* StevenK makes a note to destroy his wireless AP01:57
StevenKI can't scp over IPv6 when connected to it01:57
StevenKBut IPv4 works fine01:57
tchaypoDo you speak dutch, perchance?01:58
*** CaptTofu has quit IRC01:58
tchaypoline 4 of has "VM's"01:59
tchaypousing 's to form a plural is correct/common in dutch01:59
StevenKAdded in 578f9037 by lifeless02:00
lifelessI'm ' special02:01
tchaypoin other news, every time I see getopt (and the accompanying TEMP= and while true; do) I have a little cry.02:01
StevenKtchaypo: What would you prefer?02:01
tchaypoThe problem is, there's nothing else that's as portable, so I can't point to anything I'd prefer to use02:02
xuhaiweitchaypo: your explanation of oac makes sense to me, thank you.02:03
StevenKBleh, I think I'm missing : for n and p02:04
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Teach testenv about numbers and bridges
tchaypohere's my manyth newbie question for the day02:08
tchaypowhat's toci?02:08
greghaynesI always assumed triple o CI02:08
tchaypooh. We're either talking about a mesoamerican deity, or TripleO CI02:08
lifelesstchaypo: TripleO-CI02:09
tchaypoit makes me sad that googling for this things always finds the github repo and never the git.openstack repo02:09
* tchaypo -> loonch02:21
*** weshay has quit IRC02:35
*** CaptTofu has joined #tripleo02:42
*** matsuhashi has quit IRC02:48
*** tserong has quit IRC02:52
*** matsuhas_ has joined #tripleo02:52
*** tserong has joined #tripleo02:53
*** tserong has joined #tripleo02:53
*** killer_prince is now known as lazy_prince03:02
*** dkehn has joined #tripleo03:04
*** dkehn_ has quit IRC03:06
lifelessI hate nbd03:17
lifeless 2911 root      20   0 59788 1768  888 R  98.5  0.1 416:30.47 qemu-nbd03:17
*** matsuhas_ has quit IRC03:22
*** dkehn is now known as dkehn_03:23
*** dkehn__ is now known as dkehn03:23
lifelessI really want to give bcache a spin03:27
lifelessbut you have to reformat your block devices03:27
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Switch to using jq to write out initial JSON
StevenKlifeless: Thoughts on ^ ?03:30
StevenKlifeless: Not looking for an actual review, just a high-level looks good or 'wtf are you smoking'03:30
*** xuhaiwei has quit IRC03:31
lifelessStevenK: one quibble03:32
StevenKlifeless: It turns up in the JSON file as "ssh-key": "-----BEGIN RSA PRIVATE KEY-----\nMIIE...03:34
lifelessStevenK: does jq / python read the resulting json ok ?03:37
StevenK% jq ".[\"ssh-key\"]" < testenv.json03:41
StevenK"-----BEGIN RSA PRIVATE KEY-----\nMIIE...03:41
StevenK>>> f['ssh-key']03:42
StevenKu'-----BEGIN RSA PRIVATE KEY-----\nMIIE...03:42
*** CaptTofu has quit IRC03:48
StevenK{"ssh-key": "-----BEGIN RSA PRIVATE KEY-----\nMIIE... is what we get from the current python call, so it's perfect03:49
StevenKlifeless: Feel like approving so I can follow the same pattern in toci?03:52
StevenKWithout having to do a horrible rebase later03:53
*** xuhaiwei has joined #tripleo04:08
*** matsuhashi has joined #tripleo04:43
*** lazy_prince has quit IRC04:46
*** akuznetsov has joined #tripleo04:47
*** lazy_prince has joined #tripleo04:57
*** lazy_prince is now known as killer_prince04:58
tchaypoStevenK: I'll verify this and add a comment on the review, but I believe you want <<< rather than <<05:05
tchaypoafaik the difference is that << doesn't do any parameter substitution05:06
StevenKtchaypo: <<< is different again05:13
*** killer_prince2 has joined #tripleo05:13
*** killer_prince2 is now known as lazy_prince05:13
tchaypomy testing shows me I'm wrong, the code as written does the expected.05:13
tchaypooh. I see.05:16
tchaypoThe mange doesn't say that no expansion is performed on the here document, it says no expansion is performed on word (or in the case of <<<, it is performed)05:17
tchaypodon't mind me.05:17
tchaypoSo my next thing is - afaict using simple "cat <<EOF > JSON_FILE" should suffice, why invoke jq?05:17
tchaypoMy assumption is that you're expecting that jq will barf if the input isn't valid json, and it's better to break at this point than to break later on when something is reading the file and the poor user will be stuck trying to figure out where this came from?05:18
StevenKtchaypo: Because it will get extended later05:20
StevenKThis is a first shot spilt it out first as a seperate branch05:20
tchaypodefine "it"05:20
StevenKtchaypo: It being the work I'm doing on testenv05:22
*** morganfainberg_Z is now known as morganfainberg05:34
lifelessStevenK: have we landed the branch to consume that now?05:34
*** akuznetsov has quit IRC05:44
StevenKlifeless: That consumes nodes? The current HEAD of incubator will deal with either05:47
*** CaptTofu has joined #tripleo05:49
*** nosnos has quit IRC05:54
*** nosnos has joined #tripleo05:54
*** CaptTofu has quit IRC05:54
*** matsuhashi has quit IRC05:56
*** matsuhashi has joined #tripleo05:57
*** matsuhashi has quit IRC05:57
lazy_princeHi all, in case if you get time, pls review it has passed ci and is waiting for review.05:58
*** matsuhashi has joined #tripleo06:00
openstackgerritSteve Kowalik proposed a change to openstack-infra/tripleo-ci: Change how the JSON is updated for readability
*** rpodolyaka has joined #tripleo06:02
*** akuznetsov has joined #tripleo06:02
*** matsuhashi has quit IRC06:09
*** matsuhashi has joined #tripleo06:10
*** matsuhas_ has joined #tripleo06:12
*** matsuhashi has quit IRC06:14
*** rpodolyaka has quit IRC06:16
*** rpodolyaka has joined #tripleo06:16
*** rpodolyaka has quit IRC06:19
*** nosnos has quit IRC06:24
*** nosnos has joined #tripleo06:25
*** matsuhas_ has quit IRC06:25
*** matsuhashi has joined #tripleo06:25
tchaypoho hum. In my change to rid devtest.html of telling the user to use $(dirname $0) I've introduced a chicken/egg06:27
tchayposure, source devtest_variables will set $TRIPLEO_ROOT and prepend $TRIPLE_ROOT/triple-incubator/scripts to the path06:28
tchaypoand that's all well and good *if you've run at least once before*06:28
tchaypobut the very first time, $TRIPLEO_ROOT is empty06:28
lifelessStevenK: and you successfully tested the testenv patch ?06:37
lifelesstchaypo: how did it pass CI then ?06:38
*** rpodolyaka has joined #tripleo06:38
*** matsuhashi has quit IRC06:39
*** matsuhashi has joined #tripleo06:40
tchaypowhich is mentioned at
tchaypoit looks like takes a shortcut and does a git clone into /opt/stack/new/tripleo-incubator06:45
tchayposo by the time we source $TRIPLEO_ROOT/tripleo-incubator/scripts has the scripts in06:46
tchaypobut tha won't be true for someone setting up their environment for the first time06:46
lifelesstchaypo: are we broken right now?06:52
lifelesstchaypo: if so, we should revert the change asap and then you can mull it over at leisure06:52
tchaypowe're broken right now for anyone who clones onto a new machine which doesn't have $TRIPLEO_ROOT already existing06:53
tchaypodoes git review have a magic for reverting, or do I just do a standard git revert; git review?06:53
tchaypo*reads docs*06:53
StevenKtchaypo: You'll have to git revert ; git ci --amend ; git review06:53
StevenKRevert doesn't add the Change-id that gerrit wants06:53
tchayponeed to amend to get the.. yep.06:54
StevenKlifeless: derekh tested patch set 2 for sure, not certain about 3, and I never managed to successfully test any of them06:55
tchayposhould I revert my change, or the change that merged mmy change into master?06:55
StevenKYour change06:55
tchaypoI'm thinking that it would be clearer if I do the former06:55
StevenKYou can't revert a merge commit06:55
StevenKI think git will yell06:55
*** rpodolyaka has quit IRC06:56
*** rpodolyaka has joined #tripleo06:58
*** rpodolyaka has quit IRC07:00
xuhaiweiI don't see blueprint for tripleo-image-elements, does it mean almost all the functions are already done?07:01
StevenKFeature work doesn't require blueprints, we use trello to track that07:02
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Revert "Remove $(dirname $0) from docs."
*** rpodolyaka has joined #tripleo07:03
* tchaypo looks at trello07:05
xuhaiweiok, I looked at trello once on the last Friday, it's not easy for me to understand it07:06
StevenKtchaypo: Re your review, I'm tempted to switch it to cat <file> in a later patch, but I need to talk to derek about it07:06
*** rpodolyaka has quit IRC07:07
tchaypoI think I'd prefer jq, on balance. It's a smidge more heavyweight, and in the normal case we get no benefit07:07
tchaypobut in the case where we've somehow managed to screw up the json I think it's probably better to find out about it early07:07
StevenKtchaypo: I'm talking about the python one-liner that's embedded07:07
tchaypoStevenK: are you able to add me to the trello org?07:07
tchaypooh, that one07:07
tchaypomeh. out-of-scope for that change.07:08
StevenKIf we want to continue to replace \n with \\n, we could use sed -e ':a;N;$!ba;s/\n/\\n/g' /root/.ssh/test_env_rsa , but I think that's worse07:08
lifelessxuhaiwei: look under tripleo, not tripleo-image-elements07:08
StevenKtchaypo: I bugged lifeless to add me07:09
lifelessxuhaiwei: but most of our work is too fine grained to make blueprints, we do have blueprints for big things like HA design07:09
tchaypolifeless: *bug**bug*07:09
lifelesstchaypo: for what?07:09
tchaypolifeless: add me to the org in trello07:09
lifelesswhats your trello uid ?07:09
xuhaiweilifeless: look the bp under tripleo?07:10
StevenKxuhaiwei: As in
tchaypolifeless: it's now tchaypo07:10
lifelesstchaypo: done07:13
tchaypowoo. Should I add a card for (and the reimplementation that's going to follow) or is that too small a piece of work?07:14
tchaypoI'm thinking that since I've discovered a way in which the ci environment doesn't mirror what we tell users to do, this probably warrants a bug and a card07:14
lifelessit certainly warrants a bug07:16
lifelesstchaypo: but is it something to worry about right now, vs e.g. HA ?07:16
lifelesstchaypo: or reuse-on-baremetal ?07:16
lifelesstchaypo: or provisioning mirrors to make builds more reliable?07:16
lifelessNg: / GheRivero:
lifelessok, I'm awl for a bit07:18
tchaypoit's ~4 lines of code to fix; to my mind that's so trivial I may as well do it now so I can stop thinking about it07:18
tchaypovs coming back in N weeks and having to recreate mental state07:18
*** jprovazn has joined #tripleo07:32
*** e0ne has joined #tripleo07:46
*** CaptTofu has joined #tripleo07:50
lifelesstchaypo: 4 lines is not really worth a card then :)07:51
tchaypourr. obviously that's not a few lines to make the CI env more like a naive user env. It's a few lines to make the naive user env work, and then I think we can drop this and move on07:51
lifelesstchaypo: I mean you can, but its not really a team coordination issue07:51
tchaypothat's spooky. I go away for 30 minutes and then just as I'm typing a response to myself you preempt me and make it look like I'm responding to you07:51
tchaypoyou're going to be one very annoying parent if you can pull off that trick often enough :)07:52
lifelessjust won a CS:GO game 16:1, which is unheard of :)07:52
*** CaptTofu has quit IRC07:55
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Remove $(dirname $0) from docs.
*** e0ne has quit IRC07:56
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Remove $(dirname $0) from docs.
*** rpodolyaka has joined #tripleo07:57
rpodolyakamorning tripleo07:57
*** giulivo has joined #tripleo07:59
* tchaypo -> dinner08:00
*** nijaba has joined #tripleo08:02
*** nijaba has quit IRC08:02
*** nijaba has joined #tripleo08:02
*** rcarrillocruz has joined #tripleo08:02
*** rpodolyaka1 has joined #tripleo08:03
rcarrillocruzhmm, guys, some change apparently broke devtest to be run unattended?08:03
rcarrillocruzvagrant@precise64:~/tripleo-incubator/scripts$ ./devtest.sh08:03
rcarrillocruzNot running - this script is destructive and requires --trash-my-machine to run.08:03
rcarrillocruzvagrant@precise64:~/tripleo-incubator/scripts$ ./ --trash-my-machine08:03
rcarrillocruz./ line 144: command not found08:03
*** gcha has joined #tripleo08:05
*** rpodolyaka1 has quit IRC08:07
GheRiveromorning all08:10
lifelessrcarrillocruz: yes, fixed by which GheRivero should be landing soon :)08:12
openstackgerritA change was merged to openstack/tripleo-incubator: Revert "Remove $(dirname $0) from docs."
*** rcarrillocruz1 has joined #tripleo08:13
*** nosnos has quit IRC08:13
*** rcarrillocruz has quit IRC08:13
*** nosnos has joined #tripleo08:14
*** matsuhashi has quit IRC08:17
*** matsuhashi has joined #tripleo08:17
*** rdopieralski has joined #tripleo08:18
*** Lotus907efi1 has quit IRC08:21
*** rcarrillocruz1 has quit IRC08:21
*** openstack has quit IRC08:21
*** openstack has joined #tripleo08:30
*** openstackstatus has joined #tripleo08:30
*** andreaf has joined #tripleo08:31
*** e0ne has joined #tripleo08:32
*** nosnos has quit IRC08:34
*** jcoufal has joined #tripleo08:36
*** fandi has joined #tripleo08:37
*** jistr has joined #tripleo08:40
*** e0ne_ has joined #tripleo08:44
*** morganfainberg is now known as morganfainberg_Z08:45
*** e0ne has quit IRC08:48
*** dividehex has quit IRC08:50
*** fandi has quit IRC08:54
*** vkozhukalov_ has joined #tripleo08:55
*** petertoft has joined #tripleo08:56
*** matsuhas_ has quit IRC08:56
*** dividehex has joined #tripleo08:58
*** martyntaylor has joined #tripleo08:59
*** lucasagomes has joined #tripleo09:02
*** rpodolyaka1 has joined #tripleo09:03
*** andreaf2 has joined #tripleo09:04
*** lucasagomes has quit IRC09:05
*** lucasagomes has joined #tripleo09:05
*** mrunge has joined #tripleo09:05
*** matsuhashi has joined #tripleo09:06
*** andreaf has quit IRC09:07
*** rpodolyaka1 has quit IRC09:08
*** martyntaylor has left #tripleo09:09
*** lucasagomes has quit IRC09:09
*** rcarrillocruz1 has joined #tripleo09:11
*** rcarrillocruz has quit IRC09:12
*** nosnos has joined #tripleo09:14
*** rcarrillocruz has joined #tripleo09:18
*** rcarrillocruz1 has quit IRC09:20
*** jcoufal has quit IRC09:20
*** xuhaiwei has quit IRC09:34
*** martyntaylor has joined #tripleo09:39
*** jtomasek has joined #tripleo09:42
*** fandi has joined #tripleo09:48
*** CaptTofu has joined #tripleo09:51
*** vkozhukalov_ has quit IRC09:51
*** nosnos_ has joined #tripleo09:55
*** CaptTofu has quit IRC09:56
*** nosnos has quit IRC09:58
*** derekh has joined #tripleo10:02
*** martyntaylor has left #tripleo10:02
derekhI'm not here right now so caon't look into it but FYI no overcloud jobs have passed since friday evening
derekhsetting firedrill, so it gains attention10:03
*** derekh changes topic to "FIREDRILL overcloud ci job not passing | TripleO | tripleo-cd running preserve-ephemeral WIP patches and | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt"10:04
derekhthat page with the jobs listed updates every 20 minutes,10:04
derekhgotta go10:04
*** derekh has quit IRC10:04
*** nosnos_ has quit IRC10:10
*** martyntaylor has joined #tripleo10:10
*** e0ne_ has quit IRC10:12
*** nosnos has joined #tripleo10:12
*** martyntaylor has quit IRC10:13
*** e0ne has joined #tripleo10:14
*** tteggel has quit IRC10:18
*** tteggel has joined #tripleo10:18
*** akrivoka has joined #tripleo10:19
openstackgerritA change was merged to openstack/tripleo-image-elements: Correct permissions on /tftpdboot/pxelinux.cfg directory
openstackgerritA change was merged to openstack/tripleo-heat-templates: Change nova compute driver location
*** pbelanyi has joined #tripleo10:27
*** matsuhashi has quit IRC10:38
*** rcarrillocruz1 has joined #tripleo10:42
*** rcarrillocruz has quit IRC10:45
*** nosnos has quit IRC10:46
*** nosnos has joined #tripleo10:47
*** matsuhashi has joined #tripleo10:54
*** vkozhukalov_ has joined #tripleo10:57
*** rpodolyaka1 has joined #tripleo11:01
openstackgerritA change was merged to openstack/tripleo-image-elements: Add link for Ironic rootwrap
openstackgerritA change was merged to openstack/tripleo-image-elements: Change nova scheduler driver location
*** xuhaiwei has joined #tripleo11:02
*** rpodolyaka1 has quit IRC11:05
*** matsuhashi has quit IRC11:05
openstackgerritjan grant proposed a change to openstack/tripleo-incubator: Use a pypi mirror for the bootstrap process.
*** jcoufal has joined #tripleo11:09
*** openstackgerrit has quit IRC11:10
*** openstackgerrit has joined #tripleo11:11
*** matsuhashi has joined #tripleo11:13
rcarrillocruz1guys, is there any reason why when we run we clone tripleo-incubator into the .cache folder? I think it would be good to have an option to simply copy the already cloned tripleo-incubator files from where we run devtest in first instance. That way any modification we do to devtest scripts will be tested, as is now no matter what we modify in tripleo-incubator scripts will be taken into account, as we clone tripleo-incub11:18
rcarrillocruz1unless my workflow is broken, any thoughts?11:18
jangI've a change to do something like this.11:18
jangI use a directory of checked-out git repos (including tripleo-incubator) and have a small sourced file that sets DIB_REPOLOCATION and DIB_REPOREF for each of them. Then if I run devtest from inside that directory, an equivalent checkout (based on that repo) happens to my ~/.cache11:20
*** andreaf2 has quit IRC11:21
rcarrillocruz1nice, i wondered if it was just me that found this a bit annoying to circumvent11:22
rcarrillocruz1thanks jang11:22
jangyou're welcome.11:29
jang(and also welcome to give that a +1 if you find it useful, heh :-) )11:30
*** matsuhashi has quit IRC11:31
jangrcarrillocruz1: I use it with this: as a part of the script that sets up my environment before running devtest.sh11:32
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Node profile suggestions
*** rcarrillocruz1 has quit IRC11:35
*** fandi has quit IRC11:38
*** lazy_prince2 has quit IRC11:45
*** e0ne has quit IRC11:45
*** weshay has joined #tripleo11:46
*** e0ne has joined #tripleo11:49
*** sdake has quit IRC11:49
*** CaptTofu has joined #tripleo11:52
*** lazy_prince has joined #tripleo11:54
*** lazy_prince is now known as killer_prince11:54
*** martyntaylor has joined #tripleo11:55
*** CaptTofu has quit IRC11:56
*** xuhaiwei has quit IRC11:59
*** CaptTofu has joined #tripleo12:02
*** dprince has joined #tripleo12:03
*** rdopiera1ski is now known as rdopieralski12:04
*** lynxman has quit IRC12:04
*** lynxman has joined #tripleo12:05
*** e0ne_ has joined #tripleo12:08
*** gonery has joined #tripleo12:11
*** e0ne has quit IRC12:12
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Fix spelling in a comment in test data
*** rdopieralski has quit IRC12:14
*** rdopieralski has joined #tripleo12:16
*** matsuhashi has joined #tripleo12:27
*** CaptTofu has quit IRC12:31
*** martyntaylor1 has joined #tripleo12:32
*** e0ne has joined #tripleo12:32
*** martyntaylor has quit IRC12:35
*** e0ne_ has quit IRC12:36
*** matsuhashi has quit IRC12:37
*** nosnos_ has joined #tripleo12:37
*** nosnos has quit IRC12:37
*** matsuhashi has joined #tripleo12:37
*** matsuhashi has quit IRC12:42
*** e0ne has quit IRC12:45
*** e0ne has joined #tripleo12:45
*** pblaho has quit IRC12:48
*** Guest26078 is now known as ohadlevy12:50
*** ohadlevy has quit IRC12:50
*** ohadlevy has joined #tripleo12:50
*** CaptTofu has joined #tripleo12:50
*** weshay has quit IRC12:54
*** sseago_ has joined #tripleo12:54
*** CaptTofu has quit IRC12:54
*** kiall_ is now known as Kiall12:59
*** rbrady1 is now known as rbrady13:00
*** matsuhashi has joined #tripleo13:03
*** lblanchard has joined #tripleo13:04
*** matsuhashi has quit IRC13:05
*** matsuhashi has joined #tripleo13:05
openstackgerritA change was merged to openstack/diskimage-builder: Move install type enablement into base element
*** matsuhashi has quit IRC13:10
*** nosnos_ has quit IRC13:12
*** dkehn__ is now known as dkehn13:14
*** jtomasek has quit IRC13:17
*** hashar has joined #tripleo13:19
*** julim has joined #tripleo13:22
*** jtomasek has joined #tripleo13:32
*** blamar has joined #tripleo13:39
*** weshay has joined #tripleo13:43
*** jpeeler has joined #tripleo13:47
*** jpeeler has joined #tripleo13:47
*** slagle has quit IRC13:58
*** lazy_prince has joined #tripleo13:59
*** killer_prince has quit IRC14:00
*** lazy_prince is now known as killer_prince14:00
*** slagle has joined #tripleo14:00
*** jcoufal has quit IRC14:03
*** jcoufal has joined #tripleo14:04
*** rwsu has joined #tripleo14:07
*** killer_prince is now known as lazy_prince14:10
openstackgerritA change was merged to openstack/tuskar-ui: Node profile suggestions
openstackgerritA change was merged to openstack/tuskar-ui: Fix spelling in a comment in test data
*** victor_lowther__ has quit IRC14:11
*** victor_lowther__ has joined #tripleo14:12
*** pblaho has joined #tripleo14:21
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Return template attributes
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Allow set-source-vars to consume a git manifest
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Allow set-source-vars to consume a git manifest
*** julim_ has joined #tripleo14:33
*** julim has quit IRC14:36
*** lazy_prince has quit IRC14:40
*** rcarrillocruz has joined #tripleo14:54
goneryHi, I'm a bit lost with the ${foo}-package-install and ${foo}-source-install difference. Do you have any point that explain how it works?15:01
jangI forget where the details are - the source-repositories element?15:07
*** newell has joined #tripleo15:07
SpamapSgonery: It is actually about to change.15:07
*** tburnes has joined #tripleo15:07
goneryI finally found that:
gonerySpamapS: how, I thought it was an ongoing changes15:09
*** killer_prince has joined #tripleo15:10
SpamapSgonery: ohh! that just landed :)15:10
jangthat's much neater.15:12
goneryah! awesome. I was getting mad =)15:13
*** vkozhukalov_ has quit IRC15:14
*** andreaf has joined #tripleo15:14
SpamapSYeah that is only 2 hours old :)15:15
*** slagle has quit IRC15:16
goneryI'm discovering TripleO and I did some (minor) changes on the doc like this one
*** athomas has joined #tripleo15:17
*** slagle has joined #tripleo15:17
goneryI've the feeling this documentation is a bit outdated. Does it make sense to push them?15:17
slaglegonery: which docs?. what's in the diskimage-builder README should be accurate15:20
slaglelooks like some markdown syntax errors slipped through though15:20
slagleif you look at the source it might be clearer15:20
slaglei'll push a fix for the md errors15:21
goneryok so, I continue my changes on tripleo-incubator/doc/source/deploying.rst15:22
*** shakamunyi has quit IRC15:31
openstackgerritA change was merged to openstack/tripleo-incubator: doc: correctly index the disk-image-create calls
jeblairthere are 9 tripleo-test-cloud hosts in nodepool that have been in the delete state for a long time (38-212 hours)15:33
jeblairlifeless, davidlenwell, Ng: ^15:34
*** akuznetsov has quit IRC15:36
*** sballe has joined #tripleo15:39
*** CaptTofu has joined #tripleo15:41
*** andreaf has quit IRC15:42
SpamapSslagle: oh markdown errors, that reminds me that we need to land my markdown validator in dib :)15:43
SpamapSgonery: and yes, deploying.rst has not seen much revision lately15:43
SpamapSjeblair: davidlenwell is not associated ;)15:43
SpamapSjeblair: <-- the source of truth15:44
SpamapSjeblair: and I'm looking into it now15:44
*** lblanchard has quit IRC15:45
*** lblanchard has joined #tripleo15:46
SpamapSjeblair: the ones stuck in deleted are known neutron races that require nova-compute to be restarted.. they should start disappearing soon15:48
*** jprovazn has quit IRC15:49
jeblairSpamapS: davidlenwell has a voice15:49
SpamapSjeblair: oh we haven't been managing voice15:51
SpamapSgood idea though :)15:52
SpamapSwe should tie tripleo-cd-admins into a bot or something :-P15:52
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Correct markdown errors
*** marun has joined #tripleo15:53
jeblairSpamapS, lifeless: feel free to self-manage the channel access list (we have a script that makes sure that the infra bots have what they need now)15:54
*** andreaf has joined #tripleo15:54
jeblairSpamapS, lifeless: i'm personally not a fan of voicing, but regardless, it's probably better to be consistent one way or the other.15:55
jeblairSpamapS: anyway, cool, thanks for looking into it.15:55
*** pbelanyi has quit IRC15:59
SpamapSjeblair: all gone16:06
*** jtomasek has quit IRC16:06
*** akrivoka has quit IRC16:10
*** jrist has joined #tripleo16:12
*** rdopieralski has quit IRC16:13
*** CaptTofu has quit IRC16:16
*** CaptTofu has joined #tripleo16:17
NgSpamapS: we also need a playbook to start being worked up, atm I wouldn't know what to do with stuck nodepool stuff16:17
NgI'm not even sure I know what nodepool is :)16:17
*** Hefeweizen has quit IRC16:18
SpamapSNg: it is stuck nova stuff16:18
SpamapSNg: and agreed, the play for things stuck with neutron errors that won't delete is16:19
SpamapS$ . stackrc ; for i in $(nova list | grep ci-overcloud-NovaCompute|awk -F\| '{split($7,x,"=") ; print x[2]}') ; do ssh heat-admin@$i sudo service nova-compute restart ; done16:19
SpamapSNg: though I suspect we can eliminate those by updating nova16:20
SpamapSwe should probably get _actual_ CD going for our cloud :-P16:20
* SpamapS looks at Heat and curses16:20
SpamapSno retry.. no graceful shutdown.. I want to replace Heat with a moderately long bash script16:20
*** jtomasek has joined #tripleo16:20
*** akuznetsov has joined #tripleo16:21
*** CaptTofu has quit IRC16:22
*** jcoufal has quit IRC16:24
*** cwolferh_ is now known as cwolferh16:25
*** e0ne has quit IRC16:25
*** e0ne has joined #tripleo16:25
* SpamapS quits whining and attacks graceful shutdown :-P16:25
*** jrist has quit IRC16:27
openstackgerritChris Krelle proposed a change to openstack/tripleo-image-elements: Install Ironic and IronicClient in to nova's venv
openstackgerritGonéri Le Bouder proposed a change to openstack/tripleo-incubator: doc: deploying.rst improvement
openstackgerritGonéri Le Bouder proposed a change to openstack/tripleo-incubator: Rest syntax fixes
SpamapSswift logs to syslog by default?16:35
SpamapSbut everybody else goes to stdout?16:35
*** hashar has quit IRC16:36
SpamapS filed.. perhaps actually a swift bug16:39
clarkbSpamapS: it is inconsistent :)16:40
clarkband somewhat intentionally so I guess. I gave up on making them be like everyone else a while back16:41
SpamapSjogo: <-- an example of nova failing to spin up instances because neutron-openvswitch-agent wasn't quite ready.. I think.16:41
clarkbswift is more interested in consistency across their releases than consistency with openstack (which isn't a bad thing, just different priorities)16:41
SpamapSjogo: or it is because neutron-server wasn't ready. Having a hard time following the execution.16:42
SpamapSI want a tool that takes all of the logs in a dir like that and just sorts them by timestamp into one big log.16:42
SpamapSclarkb: sucks for operators though16:42
clarkbSpamapS: yup16:42
SpamapSclarkb: and it hurts their adoption IMO16:42
SpamapSclarkb: so many openstack operators go "oh.. swift? Yeah, we might do that someday"16:42
clarkbSpamapS: I completely agree as the "operator" in this case needing to deal with swift logs16:43
SpamapSeven though having object storage is super important for writing cloud apps16:43
*** martyntaylor has joined #tripleo16:44
davidlenwelljeblair: thank you for standing up for me.. but yeah .. I don't care about that thing .. ;)16:45
SpamapSdavidlenwell: nice latency ;)16:46
davidlenwellSpamapS: I was moving to sf over the weekend .. have not touched irc in a while16:46
*** martyntaylor1 has quit IRC16:47
*** rcarrillocruz1 has joined #tripleo16:48
*** martyntaylor has quit IRC16:49
SpamapSdavidlenwell: I reject your excuse, and I fire you from #tripleo. ;)16:49
*** rcarrillocruz has quit IRC16:50
*** akuznetsov has quit IRC16:54
davidlenwellSpamapS: was i hired by tripleo?16:56
*** bauzas has quit IRC16:56
SpamapSdavidlenwell: yes, covertly. Just before I fired you.16:57
SpamapSdavidlenwell: so congratulations! and condolences. ;)16:58
*** sseago_ has quit IRC16:58
davidlenwell:) :(16:58
SpamapSCrap.. CI may be broken17:00
SpamapSSeen two instances where we fail to reach the booted instance17:00
*** jistr has quit IRC17:03
SpamapSlooks like no overcloud jobs have passed since Friday17:03
* SpamapS reads /topic17:04
slaglesee topic :)17:04
slaglei've been trying to run through devtest locally17:04
* SpamapS should probably _start_ his day with email, rather than open it 2 hours in17:05
*** e0ne has quit IRC17:13
*** morganfainberg_Z is now known as morganfainberg17:14
*** bauzas has joined #tripleo17:15
*** CaptTofu has joined #tripleo17:21
*** akuznetsov has joined #tripleo17:26
*** rcarrillocruz has joined #tripleo17:26
*** rcarrillocruz1 has quit IRC17:27
*** akuznetsov has quit IRC17:30
*** mrunge has quit IRC17:33
*** jtomasek has quit IRC17:35
*** akuznetsov has joined #tripleo17:38
*** CaptTofu has quit IRC17:42
*** vkozhukalov_ has joined #tripleo17:45
*** pblaho has quit IRC17:50
*** john-n-seattle has joined #tripleo17:52
*** petertoft has quit IRC17:54
*** john-n-seattle has left #tripleo17:55
lifelessdprince: hi; I've been keeping an eye out for the post-ci-sprint cleanup patches but (probably me being blind) haven't spotted them...17:55
dprincelifeless: well. I'm still in the fix DHCP hole.17:56
SpamapSslagle: any progress on devtest fails?17:56
dprincelifeless: so I haven't sent much up yet... although I plan to tackle it this week17:56
slagleSpamapS: my overcloud images are building now17:57
Lotus907efihow is devtest failing for slagle? I am curious if it is similar to the devtest fails I am seeing17:57
slaglethen we'll see17:57
slagleLotus907efi: in the CI environment, you can't ping instances on the overcloud17:57
slagleLotus907efi: was just trying to see if it repro's locally or not17:58
Lotus907efimy builds are not getting that far17:58
lifelessdprince: ah cool; do you need any help ?17:58
lifelessI bet its qemu-nbd sucking17:59
dprincelifeless: maybe. I'm trying to get access to our Force 10 switches ATM. Once I have that I should be able to help myself more easily.17:59
Lotus907efiI have a problem with a devtest system behind a firewall and easy_install of a dependent module called pbr failing17:59
* slagle notes that he hasn't run on fedora in a little while, and it's an epic fail currently17:59
lifelessI've seen it hit 99% CPU routinely trying to boot the guest vm the last day or so17:59
openstackgerritThomas Goirand proposed a change to openstack/tuskar-ui: Fix deprecated use of django.conf.urls.defaults
dprincelifeless: but yes. another set of eyes (possibly better than mine) would be nice once I'm in the drivers seat...17:59
dprincelifeless: do you guys have full access to your HP switches?18:00
* Ng dinners18:00
SpamapSdprince: no18:02
dprinceSpamapS: okay. good to know18:03
* dprince wants the gun18:03
*** rcarrillocruz1 has joined #tripleo18:05
*** tburnes has quit IRC18:06
*** akuznetsov has quit IRC18:06
*** akuznetsov has joined #tripleo18:06
*** rcarrillocruz has quit IRC18:07
Lotus907efilifeless: this problem with the run failing on a system behind a proxy firewall is pretty serious for us. Think I should file a bug against something?18:10
*** cwolferh_ has joined #tripleo18:11
*** rcarrillocruz has joined #tripleo18:12
*** rcarrillocruz1 has quit IRC18:12
*** cwolferh has quit IRC18:14
lifelessdprince: no access to the switches at all18:15
lifelessLotus907efi: is this the mirror issue or pbr issue?18:15
*** rcarrillocruz1 has joined #tripleo18:15
Lotus907efino, I already filed a bug for the run-mirror issue18:15
Lotus907efithis is the pbr module issue18:16
*** rcarrillocruz has quit IRC18:16
Lotus907efithe bug for the run-mirror issue with failing to build httpretty is openstack-ci/+bug/129371718:17
Lotus907efiis there someway I can turn on some sort of extended debugging to see if easy_install is even trying to find or access a .pydistutils.cfg file18:18
Lotus907efiI would hate to have to run the whole thing inside of strace to see if a file open takes place18:19
Lotus907efithe output would be enormous18:19
lifelessLotus907efi: so you've got a ~/.pydistutils.cfg containing [easy_install]\nindex_url = <local mirror> ?18:28
lifelessand what does the failure look like ?18:29
Lotus907efiI have a pastebin of the complete run output if that would help18:29
*** Hefeweizen has joined #tripleo18:30
Lotus907efiit looks the same as without the .pydistutils.cfg file18:30
*** blamar has quit IRC18:30
lifelessis the .pydistutils.cfg file readable by the user? e.g. if you're running as 'fred', it should be in fred's homedir and have at least mode u+r18:30
lifelessanyhow, point me at the pastebin please :)18:31
lifelessclarkb: SpamapS: I wonder if a new qemu landed in saucy or something to trigger this breakage. Perhaps we should configure up guestfish. How hard is that ?18:31
Lotus907efiI am running as root on this system and the file is in root's home dir18:31
lifelessthats not failing in your user area18:32
lifelessthats inside the image build18:32
lifelesswhich is a chroot18:32
lifelessBuilding elements: ramdisk base  ubuntu deploy stackuser apt-sources use-ephemeral pypi use-ephemeral18:34
lifelessthats good18:34
lifelessit should set a index url18:34
*** sdake_ is now known as sdake18:34
lifelessdi you have PYPI_MIRROR_URL set ?18:34
lifelesslooks like it18:35
lifeless+ /opt/stack/venvs/os-refresh-config/bin/pip install -U os-refresh-config18:35
lifelessDownloading/unpacking os-refresh-config18:35
lifeless uses an insecure transport scheme (http). Consider using https if has it available18:35
lifeless  Downloading os-refresh-config-0.1.1.tar.gz18:35
Lotus907efieverything BEFORE the easy_install of pbr gets pulled from the pypi mirror I have setup18:35
Lotus907efiso if it is a chroot does the .pydistutils.cfg need to be placed inside the chroot environment somewhere?18:37
slagleSpamapS: from a failed ci overcloud job log:
slagleand i'm seeing something similar locally as well18:37
slaglecould be this is related:
lifelessLotus907efi: is pbr in your local mirror ?18:39
slagleyea, exact same traceback in my local environment which is using all Fedora18:39
Lotus907efiyes, I checked for that18:39
slagleso something "upstream" is hosing us i suspect18:39
Lotus907efiI did a find and it is there18:39
lifelessslagle: SpamapS: interesting. Ok I have a suggestion.18:40
lifelessslagle: SpamapS: finish migrating to ml2. We're running an unsupported combo at the moment.18:40
lifelessslagle: SpamapS: specifically we're running the ml2 plugin on the network node, but the ovs plugin on the hypervisor.18:40
lifelesswe should be running ml2 in both places18:40
*** blamar has joined #tripleo18:41
lifelessI'm speculating that the mismatch we have is affecting vif plug callbacks18:42
slaglelifeless: will see if i can switch over my running devtest18:42
slagleand see if that helps18:42
lifelessLotus907efi: can you make a virtualenv and then pip install pbr within it successfully ?18:42
slagleit's definitely that change i linked that is causing the "failure"18:42
lifelessreading that change18:43
lifelessso also check with markmcclain perhaps about whether the neutron side is all landed ?18:43
greghaynesI was getting pbr fails earlier when using run_mirror + the binary mirror dir, havent dug much into it yet but using the regular mirror dir seemed to work for now18:45
lifelessLotus907efi: if you export break=before-install and run devtest again, it will drop you into a shell after doing the pypi configuration18:46
Lotus907efiwhat do you mean by "binary mirror dir" verses "regular mirror dir" ???18:46
openstackgerritChris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic bits to _undercloud
greghaynestheres a /Ubuntu-13.10 dir that run-mirror makes18:46
lifelessoh ! I see18:46
lifelessLotus907efi: so when you did 'find'18:46
lifelessLotus907efi: can you pastebin the pbr files you found ?18:46
lifelessif its all wheels, its not in the mirror as far as easy_install is concerned (AFAIK)18:47
*** e0ne has joined #tripleo18:48
Lotus907efithere is a pbr tar.gz file under a Ubuntu-13.10 directory18:48
Lotus907efihmm, maybe not18:49
greghaynesThere was for me, but still fail18:49
Lotus907efithere is a pbr-0.6-py2.py3-none-any.whl file under Ubuntu-13.10/pbr18:49
Lotus907efibut under the mirror dir there is a pbr/pbr-0.6.tar.gz file18:50
greghaynesYes, I imagine lifeless has something in mind with the easy_install not knowing how to handle wheels18:50
lifelesswell, you're pointing at mirror, not mirror/Ubuntu-13.10 right? so that should be ok18:50
Lotus907efiyes, I am pointing at mirror18:50
Lotus907efifrom my file: export PYPI_MIRROR_URL=
*** tburnes has joined #tripleo18:51
*** rcarrillocruz has joined #tripleo18:52
Lotus907efiand the .pydistutils.cfg has index_url =
lifelessLotus907efi: the one inside the image? at this point in the process the one in your root user home dir is irrelevant18:53
lifelessLotus907efi: diskimage-builder chroots18:53
*** rcarrillocruz1 has quit IRC18:54
Lotus907efiso as I asked before does the .pydistutils.cfg file have to be moved under the chroot somehow?18:54
lifelessno, the pypi element creates it18:54
lifelessbut we can debug whats going on18:54
Lotus907efihow can I check if that is actually happening?18:54
lifelessif you export break=before-install18:54
lifelessthen run devtest18:54
lifelessit will drop into bash18:54
lifelessbetween the pre-install.d and install.d phases18:55
Lotus907efialright and the bash shell will be inside the chroot?18:55
lifelessyou can then make a virtualenv in there18:55
lifelessand try to pip install pbr18:55
Lotus907efiok, then I just check for the existence of the file18:55
lifelessor easy_install pbr18:55
Lotus907efihow do I make a virtualenv?  I have not done that before18:55
lifelessand strace or whatnot in a very granular fashion18:55
lifelessget into the shell18:55
lifelessI'll lookup some details for you18:56
Lotus907efialright, let me give this a try18:56
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Use xtradb-cluster in place of mysql
lifelessonce you're in the shell18:57
Lotus907efiso just in the shell environment before I run --trash-my-machine I do  "export break=before-install" then run --trash-my-machine18:57
lifelessonce you're in the shell18:57
lifelesscd to /tmp/in_target.d/install.d/18:57
lifelesswe'll run a few helper scripts that should succeed18:57
Lotus907efiok, I am going to let this go while I make a quick run to the post office18:57
Lotus907efiI have to drop something off for my stepson18:57
lifeless00-baseline-environment and 00-ccache and 01-install-pip18:58
lifelessafter they succeed18:58
Lotus907efiand I will check in as soon as I get back18:58
lifelesswe can step through the behaviour of 01-os-refresh-config line by line18:58
*** rpodolyaka1 has joined #tripleo19:02
SpamapSlifeless: sure that makes sense. I really haven't ever looked closely.19:03
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Use xtradb-cluster in place of mysql
greghaynes-> noms19:14
*** weshay has quit IRC19:19
*** petertoft has joined #tripleo19:24
*** vkozhukalov_ has quit IRC19:25
SpamapSslagle: over here now.. so we have a lot of settings in ovs_neutron_plugin.ini only...19:29
SpamapSslagle: feels like it would be helpful to have an actual neutron expert review that19:30
*** fandi has joined #tripleo19:30
*** jang1 has joined #tripleo19:31
*** hashar has joined #tripleo19:48
openstackgerritA change was merged to openstack/diskimage-builder: Support adding DHCP interfaces one at a time.
*** fandi has quit IRC19:54
openstackgerritA change was merged to openstack/tripleo-image-elements: Install Ironic and IronicClient in to nova's venv
*** weshay has joined #tripleo19:59
openstackgerritRichard Su proposed a change to openstack/tripleo-image-elements: Update selinux policies for mysqld
*** dprince has quit IRC20:01
lifelessNg: aiee why are you landig CI-failed patches.20:08
Nglifeless: I realised I was being stupid after those two and stopped :/20:09
* SpamapS gives Ng a blanket and walks him over to the fire.. there there.. it's Monday... have some soup20:09
* SpamapS motions to the executioner to start sharpening the axe behind Ng's back20:10
greghaynesI wouldnt trust that soup, either20:10
NgI was intending on spending most of the evening getting things landed, and got a little ahead of myself20:11
lifelessNg: it's usual to read the jenkins comments for tripleo-ci :)20:11
SpamapSgreghaynes: it's just kava.. we don't want him to fight... ;)20:12
Nglifeless: I did see them, but I thought "oh right, CI is broken" and then two patches later I was mid-review when it hit me20:13
Ngapparently being married makes me stupider20:13
*** akuznetsov has quit IRC20:14
SpamapSNg: well we could have told you that...20:15
lifelessmarriage is proof of stupidity? Not sure I agree...20:15
SpamapSNo it is the beginning of stupidity.20:16
SpamapS"fat, dumb, and happy.."20:16
lifelessSpamapS: ah, thats sugar :)20:17
* SpamapS goes for a quick walk 'round the block to counter the first one20:17
SpamapSlifeless: you're like a dog on a bone20:17
Nga low sugar bone20:17
lifelessSpamapS: since I just finished breakfast - 2 rashers of bacon and three eggs... I sure am20:17
*** rcarrillocruz1 has joined #tripleo20:17
lifelessSpamapS: I've lost another 2Kg since the sprint20:17
NgI still have no idea how you make that work, I can barely find time to get a single piece of jam toast in my face in the entire kids-breakfasting process20:18
SpamapSI eat before the children wake.20:18
lifelessNg: cynthia had an egg, 1/2 rasher of bacon and is topping up with a yoghurt20:18
*** rcarrillocruz has quit IRC20:18
NgSpamapS: good grief, I'd be getting up yesterday20:18
lifelessNg: of course, I let Cynthia eat herself... :)20:19
NgI'd pass myself on the way to bed20:19
SpamapSUsually 1 egg and either a yogurt or chunk of buttered toast.20:19
SpamapSAh, my children get up at 0645 - 070020:19
SpamapSI try to wake around 063020:19
*** rcarrillocruz has joined #tripleo20:19
Ng0645. LUXURY!20:19
SpamapSand.. letting them stay up until 2000, which is a bit unorthodox.20:20
Ngand cheating!20:20
SpamapSbut seriously, my 2 year old is the laziest baby ever. He has slept 10 - 12 hours straight through the night since he was 12 months old.20:20
SpamapSand 8+ hours in a row since he was 6 months20:21
SpamapSso also: genes.20:21
SpamapSok.. walk.. must get back before devtest finally finishes20:21
*** rcarrillocruz1 has quit IRC20:22
openstackgerritGonéri Le Bouder proposed a change to openstack/tripleo-incubator: Rest syntax fixes
openstackgerritGonéri Le Bouder proposed a change to openstack/tripleo-incubator: doc: deploying.rst improvement
slagleugh, how are we supposed to get the admin tenant id into neutron.conf?20:37
*** julim_ has quit IRC20:37
slaglethat's not known until init-keystone has run20:39
Lotus907efilifeless: back from post office20:39
Lotus907efibut I do not have a shell prompt20:39
Lotus907efiit says: --------------------- END PROFILING ---------------------20:39
Lotus907efiStarting debug shell. Exit to resume building.20:39
Lotus907efiAt stage before-install20:39
Lotus907efibut when I hit enter/return I do not get any sort of shell prompt20:40
Lotus907efiam I supposed to have a shell prompt here?20:42
*** slagle changes topic to "FIREDRILL overcloud ci job not passing ( | TripleO | tripleo-cd running preserve-ephemeral WIP patches and | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt"20:43
lifelessLotus907efi: you're in a shell :)20:45
lifelessLotus907efi: type 'ls' and enter20:45
Lotus907efihmm, ok - weird but ok20:46
lifelessLotus907efi: one reason you might not have a prompt is if stdout is a pipe, and I have a terrible suspicion devtest has done that to you20:46
lifelessLotus907efi: please file a bug about that20:46
*** victor_lowther__ is now known as victor_lowther20:46
Lotus907efiumm, I redirected stdout to a tee command so I could capture a log20:47
Lotus907efiwhoops, my bad20:47
Lotus907efiok, in root's home dir in this chroot env there is indeed a .pydistutils.cfg file20:51
Lotus907efiand it seems to have the right stuff in it20:51
*** rpodolyaka1 has quit IRC20:52
*** rpodolyaka1 has joined #tripleo20:53
Lotus907efiso I have done "cd /tmp/in_target.d/install.d/" and I see lots of stuff there. You mentioned running 00-baseline-environment and 00-ccache and 01-install-pip20:53
Lotus907efishould I do that now?20:53
Lotus907efialready ran 00-baseline-environment and 00-ccache20:56
Lotus907efinow going to run 01-install-pip20:56
*** rpodolyaka1 has quit IRC20:57
Lotus907eficrap, got an error when it tried to get setuptools20:59
Lotus907efiCould not find any downloads that satisfy the requirement setuptools20:59
Lotus907efi  Some externally hosted files were ignored (use --allow-external setuptools to allow).20:59
Lotus907efiCleaning up...20:59
Lotus907efiNo distributions at all found for setuptools20:59
Lotus907efithat worked before just fine20:59
lifelesshow very odd21:00
Lotus907efiI know why21:00
Lotus907efiI have proxy env varis set21:00
Lotus907efiand it did not pick up the right no_proxy one with the mirror system listed21:01
Lotus907efilet me try something here21:01
Lotus907efiok, I fixed it21:03
Lotus907efiok, 01-install-pip finished successfully21:03
Lotus907efiit said "Successfully installed virtualenv"21:04
Lotus907efiso what now?21:04
lifelesslets try the 01-os-refresh-config script21:06
lifelessin case that thing you fixed fixes it too21:06
*** e0ne has quit IRC21:06
*** e0ne has joined #tripleo21:07
*** dguerri_ is now known as dguerri21:08
*** rcarrillocruz1 has joined #tripleo21:09
*** lblanchard has quit IRC21:09
Lotus907efihmmm, so I stopped the whole process and re-ran --trash-my-machine without my redirection piped to a tee command21:09
Lotus907efiand I still do not get a shell prompt21:10
lifelessLotus907efi: right, devtest does it's own tees21:10
lifelessLotus907efi: thats why I asked you to file a bug :)21:10
Lotus907efibut anyway I cd'ed to /tmp/in_target.d/install.d/ and then ran 00-baseline-environment21:10
*** e0ne has quit IRC21:10
Lotus907efithen I ran 00-ccache21:11
Lotus907efibut it has not returned yet21:11
Lotus907efiand I cannot enter any more commands21:11
*** hashar is now known as hasharMeeting21:11
lifelessodd ;)21:11
Lotus907efiok, I will file a bug on this no shell prompt21:12
*** rcarrillocruz has quit IRC21:12
Lotus907efiok, it just took quite a while21:12
Lotus907efino I can do "ls" again21:12
Lotus907efinow going to run 01-install-pip21:12
Lotus907efiok, that ran ok21:13
Lotus907efinow for 01-os-refresh-config21:13
Lotus907efiCommand python egg_info failed with error code 1 in /opt/stack/venvs/os-refresh-config/build/os-refresh-config21:14
Lotus907efisocket.timeout: timed out21:14
*** e0ne has joined #tripleo21:15
Lotus907efiin the apache access log on my mirror system the last two lines in the log say:21:15
Lotus907efi10.23.16.147 - - [17/Mar/2014:15:13:38 -0600] "GET /pypi/mirror/os-refresh-config/ HTTP/1.1" 200 448 "-" "pip/1.5.4 CPython/2.7.5+ Linux/3.11.0-12-generic"21:15
Lotus907efi10.23.16.147 - - [17/Mar/2014:15:13:38 -0600] "GET /pypi/mirror/os-refresh-config/os-refresh-config-0.1.1.tar.gz HTTP/1.1" 200 8722 "-" "pip/1.5.4 CPython/2.7.5+ Linux/3.11.0-12-generic"21:15
*** rwsu has quit IRC21:16
Lotus907efiso it fetched pypi/mirror/os-refresh-config/os-refresh-config-0.1.1.tar.gz21:16
Lotus907efibut nothing after that21:16
lifelessso thats the problem reproduced21:17
lifelessif you look in /opt/stack/venvs/21:17
lifelessand delete os-refresh-config21:17
lifelesswe're ready to debug step by step21:17
Lotus907efiit did catch a pip.log file with the failure in it21:17
lifelessso the script we're going to be stepping through is elements/os-refresh-config/install.d/01-os-refresh-config in tripleo-image-elements (but I'm going to feed you commands one at a time)21:18
lifelessonce you've deleted /opt/stack/venvs/os-refresh-config21:18
lifelessvirtualenv --setuptools /opt/stack/venvs/os-refresh-config21:18
lifeless/opt/stack/venvs/os-refresh-config/bin/pip install -U pip21:18
lifeless/opt/stack/venvs/os-refresh-config/bin/pip install -U 'setuptools>=1.0'21:18
Lotus907efiok, let me do this one step at a time21:19
lifeless/opt/stack/venvs/os-refresh-config/bin/pip install -U pbr21:19
lifeless/opt/stack/venvs/os-refresh-config/bin/pip install -U os-refresh-config21:19
Lotus907efiso /opt/stack/venvs/os-refresh-config is a directory21:19
Lotus907efiwith stuff under it21:20
Lotus907efiyou want me to delete it all?21:20
Lotus907efithe directory and all of it's contents?21:21
Lotus907efiok, removing whole directory21:22
*** rpodolyaka1 has joined #tripleo21:23
Lotus907efiran "virtualenv --setuptools /opt/stack/venvs/os-refresh-config" and it said it said: "Installing setuptools, pip...done."21:23
Lotus907efiso onto the next command21:23
Lotus907efiand "/opt/stack/venvs/os-refresh-config/bin/pip install -U pip" said:  Requirement already up-to-date: pip in /opt/stack/venvs/os-refresh-config/lib/python2.7/site-packages21:24
Lotus907efiSuccessfully installed setuptools21:24
Lotus907efiSuccessfully installed pbr21:25
Lotus907efiSuccessfully installed os-refresh-config21:25
Lotus907efinow what?21:25
lifelessok so we have a workaround21:26
lifelessbut lets see if we can get more data on the failure21:26
lifelessdelete the directory again21:26
lifelessthen do21:26
lifelessvirtualenv --setuptools /opt/stack/venvs/os-refresh-config21:26
lifeless/opt/stack/venvs/os-refresh-config/bin/pip install -U pip21:26
lifeless/opt/stack/venvs/os-refresh-config/bin/pip install -U 'setuptools>=1.0'21:26
lifeless/opt/stack/venvs/os-refresh-config/bin/easy_install pbr21:27
*** rpodolyaka1 has quit IRC21:27
Lotus907efiwhoops, seem to have a problem with the install pbr21:29
Lotus907efierror: None21:29
Lotus907efiit worked before21:29
openstackgerritBen Nemec proposed a change to openstack/tripleo-image-elements: Add mysql-mariadb-compat element
openstackgerritBen Nemec proposed a change to openstack/tripleo-image-elements: Add cinder-lio element
openstackgerritBen Nemec proposed a change to openstack/tripleo-image-elements: Don't install percona-toolkit if distro is RHEL7
openstackgerritBen Nemec proposed a change to openstack/tripleo-image-elements: Factor out tgt-specific parts of cinder element
openstackgerritBen Nemec proposed a change to openstack/tripleo-image-elements: Allow os-svc-daemon to specify dir name with -d option
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Add RHEL 7 image element
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Add redhat-common element
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Use redhat-common in fedora element
Lotus907efieasy_install pbr failed21:30
lifelessLotus907efi: great21:30
*** rwsu has joined #tripleo21:31
Lotus907efithat's great?21:31
lifelessstrace -o /tmp/pbr-fail.log -s 512 /opt/stack/venvs/os-refresh-config/bin/easy_install pbr21:31
lifelessLotus907efi: means we have a small codepath to analyse21:31
Lotus907efiok, so run that strace?21:33
Lotus907efiwant me to try and redirect the output to a file?21:34
Lotus907efioh, nevermind21:34
Lotus907efiI see the output file now21:34
*** beekneemech is now known as bnemec21:36
Lotus907efithat file is 800+KB21:36
Lotus907efiyou want it put somewhere?21:36
Lotus907efihmm, looking at the bottom of that file I see where there was a stat attempted of /root/.pypirc which did not exist but I do not see a mention of a .pydistutils.cfg so far21:40
Lotus907efihmmm, way way earlier in the output I see .pydistutils.cfg being opened21:42
Lotus907efiand something gets the index_url = setting21:42
lifelessLotus907efi: please do put it somewhere21:47
lifelessStevenK: C is home sick today21:47
lifelessgreghaynes: ^21:48
lifelessI'm looking for a volunteer to insert21:48
lifeless/opt/stack/venvs/$thing/bin/pip install -U pbr21:48
Lotus907efiis 800+KB too big to pastebin?  it seems like the last 100 - 200 lines is where the interesting stuff takes place anyway21:48
lifelessinto all our venv creation stuff right after the setuptools21:48
* greghaynes reads context21:48
lifelessLotus907efi: if you compress it and attach it to the bug21:48
lifelessLotus907efi: that would be great21:48
lifelessLotus907efi: if there isn't a bug, please file one on
*** petertoft has quit IRC21:49
Lotus907efiok, I can do that  .... except I don't actually have a bug open on this particular problem as of yet21:49
Lotus907efibut I can open one21:49
Lotus907efiok, filling bug21:49
lifelessmy proposal here is that we simultaneously workaround the problem by pip installing pbr and gather data to improve tooling or whatever to avoid the problem if someone triggers easy_install some other way21:49
*** rcarrillocruz has joined #tripleo21:54
greghaynesWhere do we currently do the easy_install pbr from?21:56
*** rcarrillocruz1 has quit IRC21:56
greghaynesI notice you wre asking Lotus907efi to to it just before the o-r-c install... its a dep from that?21:57
* greghaynes rtfs's o-r-c21:58
*** CaptTofu has joined #tripleo21:58
greghaynesok, I dont get why its easy_installing rather than pip installing if we are pip installing o-r-c in the first place22:00
*** bauzas has quit IRC22:00
greghaynesbut yes for fast fix we can jsut stick that as a pip install beforehand...22:00
*** e0ne has quit IRC22:02
greghaynesLotus907efi: Got a bug #?22:05
Lotus907efiworking on it now22:06
Lotus907efidescribing the conditions is complex22:06
greghaynesty. Yep :)22:06
Lotus907efisince this only seems to occur when the system is behind a firewall and requires proxy settings22:06
SpamapSLotus907efi: specifically there is some weirdness with the HP corporate firewall and the way pip/easy_install download via https22:07
SpamapSLotus907efi: I'd be curious to see if you had the same problems bouncing through the SOCKS proxy.22:07
SpamapSwhich is effectively identical when you're doing https anyway22:08
* SpamapS grumbles something about allowing SSH out but not HTTPS ...22:08
Lotus907efiI don't think I have tried a socks proxy22:08
greghaynesThe require https proxy baffles me22:09
SpamapSLotus907efi: there's some instructions in the internal wiki's.. I'll dig them out if you don't have them at hand.22:09
Lotus907efifile bug:
jang1Lotus907efi: I've something that works if you have a pypi mirror to point at: brief summary notes on the wiki under my personal space (jan grant)22:10
Lotus907efiI do have a pypi mirror22:11
jang1by "that works" I mean the mob in Bristol have been going through a similar degree of pain, but it's working now.22:11
greghayneshaha - my browser happily tries to open the bz2 as a text file22:11
greghayneson the bug report22:11
greghaynesMethinks the lp server doesnt mimetype that correctly22:11
Lotus907efihmm, not my fault if your browser has broken-ness :)22:12
Lotus907efijang1: where do I find these notes?22:12
jang1hang on, let me see if I can see it from here...22:13
Lotus907efiyou have to put on your glasses to see it from there?  :D22:13
Lotus907efior you have to use binoculars?22:13
jang1I'm not on any vpns at the mo.22:14
Lotus907efiok, I was just kidding22:14
Lotus907efijust punning on the "can see it from here"22:14
jang1something I discovered today was that if you have "extra-index-url" set in your pip conf it seems to leak into, and bugger up, the chroot environments22:15
jang1so, ditch that too.22:15
Lotus907efiI don't think I have that22:15
Lotus907efiI have index_url in .pydistutils.cfg22:16
Lotus907efibut not extra-index-url22:16
SpamapSjang1: that seems like a bug.. I don't see anywhere in any extra-data.d scripts that would pull in the local pip.conf22:17
jang1no, it surprised me too - I thought those were pristinely generated.22:17
jang1at the time I just said "... don't do that, rely on PYPI_MIRROR_URL" and left it at that.22:17
jang1I didn't dig into where ti came from, but if I've time tomorrow I'll have an experiment.22:18
jang1since most of tomorrow will be spent waiting on debootstrap, I should have plenty of time :-)22:18
Lotus907efiok, am I still filling a bug on the fact that does not actually provide a shell prompt when "export break=before-install" is set?22:18
SpamapSjang1: could you report the bug now? It's ok if it isn't supported by much evidence.. but if others who subscribe to the bugs have seen it it may remind them to report22:18
jang1will do.22:18
SpamapSjang1: I'd report against tripleo .. but it may be diskimage-builder too22:19
jang1and now, it's a school night, and I'm contractually pobliged to retire.22:22
jang1night all.22:22
SpamapSargh.. my overcloud failed with NoValidHost22:22
*** jang1 has quit IRC22:23
*** rpodolyaka1 has joined #tripleo22:24
SpamapSwait_for 60 1 [ "\$(nova hypervisor-stats | awk '\$2==\"count\" { print \$4}')" != "0" ]22:24
SpamapSstill racey22:24
SpamapSit was likely 2, instead of 322:24
Lotus907efifilled bug on the no shell prompt from when break=before-install is set:
*** rcarrillocruz1 has joined #tripleo22:28
*** rpodolyaka1 has quit IRC22:28
*** rcarrillocruz has quit IRC22:29
*** e0ne has joined #tripleo22:30
* tchaypo finally has suspend/resume on his laptop22:33
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Explicitly pip install pbr before its installed as dependency
greghaynesLotus907efi: can you see if that ^ fixes it for you22:33
greghaynesFixed my issue when trying to use the binary pip mirror as well22:34
SpamapStchaypo: bragger22:34
greghaynessuspend/resume 'just worked' on ubuntu for me22:35
greghayneswow, devtest is silly fast with that fix and binary pip mirror22:36
SpamapSgreghaynes: yeah, binary pip mirror is huge, I want to get us using that in CI22:37
tchaypoIt just worked as soon as i enabled the nvidia driver instead of using noveau22:38
greghaynesOh, youre on a different laptop than the rest of us?22:38
*** e0ne has quit IRC22:38
tchaypodon't know what you have so I'm not sure if that's different22:39
greghaynesIs it the huge one adam_g has also?22:39
SpamapStchaypo: oh lord.. noveau is terrible22:39
SpamapSI'm sad that those new ones are Nvidia22:39
SpamapSI rather like Intel.22:39
tchaypogreghaynes: oh no, that's a bigger number, I'm jealous now22:39
*** openstackgerrit has quit IRC22:39
SpamapS9470m are the EliteBook's22:40
*** openstackgerrit has joined #tripleo22:40
tchaypoyou mean utlrabook?22:40
Lotus907efihmmm, let me look at it and see22:40
* greghaynes jsut got slice battery in mail today22:40
adam_gyeah nvidia/nouveau has been giving me headaches on my zbook22:40
tchaypooh, i see, elitebook, is the HP custom name for an ultrabook?22:40
greghaynestchaypo: not really - its more like a mac clone22:40
adam_gin fact, its currently sitting in its initramfs until i can fix up modprobe.conf22:41
tchaypogreghaynes, adam_g - I've been adding notes to - can you guys spot anything obvious that I'm missing?22:42
*** hasharMeeting has quit IRC22:44
greghaynesooo, this devtest run might clock in at under 20mins22:47
* greghaynes crosses fingers22:47
*** andreaf has quit IRC22:50
*** michchap has joined #tripleo22:50
adam_gtchaypo, lgtm22:50
cody-somervilleWhat kind of problems are you guys having with the zbooks?22:53
*** michchap_ has quit IRC22:53
tchaypocody-somerville: can't get them in apj yet, so my problem is that I've got an 8570w instead (which seems perfectly functional, if a bit overweight). lifeless seems pretty happy with his22:55
*** newell has quit IRC22:56
cody-somervilletchaypo: Who told you they weren't available in APJ?22:57
adam_gcody-somerville, no issues with the laptop itself,  just general nvidia/nouveau/linux driver PITA22:58
cody-somervilleadam_g: Do you have the zBook 14?22:59
tchaypocody-somerville: extrapolation from lifeless mentioning he wasn't able to get one here, and from chris ordering me a zbook but the 8570w getting delivered instead22:59
adam_gcody-somerville, 1522:59
Shrewscody-somerville: loving mine23:01
cody-somervilleadam_g: Part of the problem might be that the zbook 15 has hybrid graphics23:02
cody-somervilleadam_g: Have you not had any luck telling it to just use the integrated Intel HD Graphics 4600?23:02
cody-somervilleShrews: You have the 14 though, right?23:02
Shrewscody-somerville: correct23:03
adam_gcody-somerville, have not tried that. ive just been using the proprietary nvidia drivers and they work okay, i just need to remember to get all the post-upgrade DKMS stuff in place before i upgrade my kernel next time23:03
cody-somervilleSo there's like five different laptops being used by our team now. I don't think that was intended.23:04
SpamapSwe had the 9470m's and then the zbook was a trial23:04
cody-somervilleNg: ping23:05
*** blamar has quit IRC23:07
openstackgerritClint "SpamapS" Byrum proposed a change to openstack/tripleo-incubator: Wait for enough hosts to be available in nova
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Explicitly pip install pbr before its installed as dependency
Lotus907efigreghaynes: I think you need to put the temp fix into os-collect-config as well23:12
greghaynesYep, just ran into that - should be fixed in that update ^?23:13
*** blamar has joined #tripleo23:13
cody-somervilleadam_g: tchaypo: Hey. Can you both send me the SKU Number of your laptops?23:15
adam_gcody-somerville, sure23:15
*** sseago_ has joined #tripleo23:23
*** rpodolyaka1 has joined #tripleo23:24
*** arosen has joined #tripleo23:25
arosenHi, I'm looking at fixing this bug: but i had a quick question23:28
arosenit looks like tripleo-image-elements doesn't actually create keystone endpoints in that repo23:28
*** rpodolyaka1 has quit IRC23:29
tchaypogreghaynes: could the pbr issue you're looking at be caused by out-of-order dependencies?23:34
tchaypoI'm looking at
lifelesscody-somerville: zbook14 APJ - we had mail from the fulfillment person saying they weren't available23:36
lifelessgreghaynes: yes, pip install $foo for setup_requires invokes easy_install on missing deps23:36
lifelessgreghaynes: so manually installing pbr is a workaround23:37
lifelessgreghaynes: and the strace should let us figure out why easy_install is bypassing the selected mirror23:37
*** xuhaiwei has joined #tripleo23:37
lifelessgreghaynes: that patch needs to be much more extensive23:38
cody-somervillelifeless: the e-mail you got was that the particular model I ordered for you in APJ was not available. They did later provide me with a model number for zbook 14 that is available in New Zealand.23:38
lifelessgreghaynes: heat-cfntools23:38
lifelesscody-somerville: ah! wow we make our life hard don't we :(23:38
tchaypocody-somerville: in my case, they confirmed to chris they were shipping a zbook and then confirmed that they *had* shipped the 8570w, no explanation23:39
lifelessgreghaynes: basically grep for bin\/pip install and plaster it everywhere23:39
greghayneshaha, ok23:40
*** CaptTofu has quit IRC23:41
*** CaptTofu has joined #tripleo23:41
Lotus907efiI clean out my sources and started completely from scratch with a fresh checkout of HEAD23:43
greghaynestchaypo: maybe but that seems unlikely23:43
Lotus907efiand now python-cinderclient is also failing due to easy_install of pbr23:43
greghaynesYep, I think thats what lifeless was getting at with plaster it everywhere23:44
Lotus907efiyeah, crud23:44
Lotus907efiI thought we were getting close23:44
greghayneseh, I think he already did the hard part in finding the workaround23:45
*** CaptTofu has quit IRC23:46
* tchaypo reads dib-seed.log23:49
* tchaypo realises just how many random env vars it relies on23:49
openstackgerritAaron Rosen proposed a change to openstack/tripleo-image-elements: Add configuration required for neutron-nova events
greghaynesLotus907efi: with the latest revision I dont think python-cinderclient should fail23:51
greghaynesthere are other things that should though, so maybe a couple more...23:51
Lotus907efiso where is the python-cinderclient code at anyway?23:53
*** blamar has quit IRC23:53
Lotus907efiI am not seeing any cinderclient code changes mentioned in
greghaynesLotus907efi: I hope the install-openstack-client change takes care of the cinderclient install23:59
greghaynesdid on my box...23:59

Generated by 2.14.0 by Marius Gedminas - find it at!