Thursday, 2014-02-27

*** e0ne has quit IRC00:02
*** CaptTofu has joined #tripleo00:04
derekhtrying to avoid the java console, how does one go about using the text console ?00:04
*** kevinbenton is now known as kevin00:05
*** kevin is now known as kevinbenton00:05
SpamapSderekh: ssh to it00:06
SpamapSderekh: that is the same auth details as we use for ipmi00:06
derekhSpamapS: ahh ok, got it thanks00:10
derekhhmmm, same problem as last time, SYN/ACK not making it back to connecting host00:13
derekhlifeless: I see you rebooted last time, assuming I can't figure anything out in the next few minutes will do the same00:18
greghaynesderekh: IIRC http://lists.openstack.org/pipermail/openstack-dev/2014-February/028009.html was why the reboot happened00:23
openstackgerritjames Polley proposed a change to openstack/tripleo-incubator: Add a missing line of config to the sample squid config  https://review.openstack.org/7671500:23
*** matsuhashi has joined #tripleo00:23
greghaynesIn case you hadnt already ready that00:24
derekhgreghaynes: yup, thats what I'm seeing at the moment00:24
derekhgreghaynes: thanks00:24
derekhrebooting ....00:27
*** saurabhs1 has quit IRC00:29
*** rbrady has joined #tripleo00:31
*** saurabhs has joined #tripleo00:32
*** UtahDave has joined #tripleo00:32
*** saurabhs has quit IRC00:35
rbradySpamapS: ping00:36
*** cd-undercloud has joined #tripleo00:40
cd-undercloud************** overcloud complete status=1 ************00:40
*** cd-undercloud has quit IRC00:40
*** shakayumi has joined #tripleo00:41
*** shakayumi has quit IRC00:41
*** shakayumi has joined #tripleo00:42
fungilooks like derekh is on the ball00:43
fungii was about to give you a heads up, but you're clearly hammering on the problem already for a while. best of luck!00:44
*** bauzas has quit IRC00:44
derekhfungi: same issue as last time, server power cycled the host when it comes back will see what I have to deal with00:44
derekhfungi: thanks, I should have done the same for you, sorry00:45
fungisounds like the sort of fun i'm glad i no longer have to deal with myself00:45
fungino worries. just glad you know--it's not really adversely impacting us00:45
*** dshurtle has joined #tripleo00:46
rbradyderekh: do you have a second for a heat template question?00:47
derekhrbrady: ask away,00:47
*** saurabhs has joined #tripleo00:49
rbradylifeless wanted me to use the value from neutron.ovs.local_ip in a conf file template, but use that value to get/make the subnet.  e.g. value is 192.0.2.7 and I need to change it to 192.0.2.0/2400:49
*** jp_at_hp has joined #tripleo00:50
rbradyderekh: I haven't figured out how that is possible in a mustache template00:51
rbradyderekh: any guidance to point me in the right direction?00:51
derekhrbrady: I don't think it is possible in a mustache template00:51
*** weshay has joined #tripleo00:52
derekhrbrady: maybe it is possible to use a function available in the heat template or just to pass the value you need into the heat template00:52
*** saurabhs has left #tripleo00:53
*** jp_at_hp has quit IRC00:53
derekhrbrady: or an executable os-apply-config template but I've never done one of those00:54
rbradyderekh: executable os-apply-config template?00:54
derekhrbrady: https://github.com/openstack/os-apply-config/blob/master/os_apply_config/apply_config.py#L99-L10000:55
*** e0ne has joined #tripleo00:57
*** dshurtle has quit IRC00:58
*** dshurtle has joined #tripleo00:58
*** dshurtle has quit IRC00:58
*** dshurtle has joined #tripleo00:58
*** dshurtle has quit IRC00:58
*** dshurtle has joined #tripleo00:59
*** e0ne has quit IRC01:02
*** dshurtle has quit IRC01:02
*** dshurtle has joined #tripleo01:03
*** dshurtle has quit IRC01:08
*** dshurtle has joined #tripleo01:09
adam_ghmm.. devtest_undercloud.sh failing for me with http://paste.ubuntu.com/7002765/01:12
adam_gis that a tmpfs + lack of RAM issue i've seen mentioned?01:12
rbradyderekh: thanks for the insight, going to give the executable template a try01:20
*** UtahDave has quit IRC01:21
derekhrbrady: np, I've never used it just seen it in the code but worth a try, I'm not sure if we are using it anywhere else yet01:21
rbradyadam_g: looks like you've hit this bug: https://bugs.launchpad.net/tripleo/+bug/128526901:22
adam_grbrady, ah, could be01:23
adam_gthanks01:23
rbradyadam_g: np01:24
*** rbrady has quit IRC01:33
*** rbrady has joined #tripleo01:40
*** nosnos has joined #tripleo01:46
*** cd-undercloud has joined #tripleo01:48
cd-undercloud************** overcloud complete status=1 ************01:48
*** cd-undercloud has quit IRC01:48
openstackgerritDustin J. Mitchell proposed a change to openstack/diskimage-builder: Update README formatting and content  https://review.openstack.org/7548001:55
*** newell has quit IRC01:56
*** e0ne has joined #tripleo01:57
*** e0ne has quit IRC02:02
*** dkehn__ is now known as dkehn02:15
*** dshurtle has quit IRC02:26
*** jcooley_ has quit IRC02:37
derekheverything seems back up on ci-overcloud but vm's not getting ip's02:41
derekhany ideas out there02:42
*** julim has joined #tripleo02:44
*** cody-somerville has joined #tripleo02:47
derekhlifeless, devananda, Ng, SpamapS_, jog0_, GheRivero, derekh, dprince, slagle I got all services back up, vm's  still don't seem to be getting IP's so I've mist something, anybody want to take over ?02:49
derekhfungi: I gotta go, VM's not getting net access, will try and sort it out in a few hours if nobody else has picked up on it before then02:56
*** e0ne has joined #tripleo02:57
*** derekh is now known as derekh_afk02:57
*** e0ne has quit IRC03:01
*** CaptTofu has quit IRC03:04
*** lblanchard has joined #tripleo03:14
*** cd-undercloud has joined #tripleo03:26
cd-undercloud************** overcloud complete status=1 ************03:26
*** cd-undercloud has quit IRC03:26
*** cd-undercloud has joined #tripleo03:28
cd-undercloud************** overcloud complete status=128 ************03:28
*** cd-undercloud has quit IRC03:28
*** ramishra has joined #tripleo03:34
*** cd-undercloud has joined #tripleo03:41
cd-undercloud************** overcloud complete status=128 ************03:41
*** cd-undercloud has quit IRC03:41
*** coolsvap has quit IRC03:42
*** weshay has quit IRC03:56
*** e0ne has joined #tripleo03:57
*** e0ne has quit IRC04:01
*** jcooley_ has joined #tripleo04:17
*** cd-undercloud has joined #tripleo04:28
cd-undercloud************** overcloud complete status=1 ************04:28
*** cd-undercloud has quit IRC04:28
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Call virsh managedsave-remove  https://review.openstack.org/7572904:30
*** saurabhs has joined #tripleo04:35
*** jcooley_ has quit IRC04:46
*** lblanchard has quit IRC04:47
*** jcooley_ has joined #tripleo04:52
openstackgerritRichard Su proposed a change to openstack/diskimage-builder: Additional swift storage service mappings  https://review.openstack.org/7675304:55
*** e0ne has joined #tripleo04:57
*** jcooley_ has quit IRC04:59
*** matsuhashi has quit IRC05:00
*** cd-undercloud has joined #tripleo05:01
cd-undercloud************** overcloud complete status=128 ************05:01
*** cd-undercloud has quit IRC05:01
*** e0ne has quit IRC05:02
*** coolsvap has joined #tripleo05:04
*** CaptTofu has joined #tripleo05:05
openstackgerritRichard Su proposed a change to openstack/tripleo-image-elements: rsync configurations to support multi-node swift  https://review.openstack.org/7610505:06
*** CaptTofu has quit IRC05:09
*** coolsvap has quit IRC05:16
*** coolsvap has joined #tripleo05:17
*** sdake_ has joined #tripleo05:19
*** vkozhukalov has joined #tripleo05:36
*** cd-undercloud has joined #tripleo05:46
cd-undercloud************** overcloud complete status=1 ************05:46
*** cd-undercloud has quit IRC05:46
*** cd-undercloud has joined #tripleo05:48
cd-undercloud************** overcloud complete status=6 ************05:48
*** cd-undercloud has quit IRC05:48
*** killer_prince has quit IRC05:54
*** sballe has joined #tripleo05:55
*** e0ne has joined #tripleo05:57
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Use xtradb-cluster in place of mysql  https://review.openstack.org/7554005:57
*** sballe has quit IRC05:59
*** e0ne has quit IRC06:01
*** killer_prince has joined #tripleo06:03
*** rpodolyaka1 has joined #tripleo06:07
*** lazy_prince has joined #tripleo06:11
*** sdake_ has quit IRC06:13
*** vkozhukalov has quit IRC06:18
openstackgerritSteve Baker proposed a change to openstack/tripleo-heat-templates: Design fiction porting undercloud-bm to software-config  https://review.openstack.org/7253306:21
*** andreaf has joined #tripleo06:23
openstackgerritSteve Baker proposed a change to openstack/tripleo-heat-templates: Design fiction porting undercloud-bm to software-config  https://review.openstack.org/7253306:26
*** cd-undercloud has joined #tripleo06:38
cd-undercloud************** overcloud complete status=1 ************06:38
*** cd-undercloud has quit IRC06:38
*** saju_m has joined #tripleo06:48
*** pblaho has joined #tripleo06:56
*** e0ne has joined #tripleo06:57
killer_princehi all, i am trying to create a RHEL 6.5 golden image using diskimage-builder with a custom element which has dependency on os-refresh-config. but it keeps failing. here are the logs http://paste.openstack.org/show/70123/. Can someone please help me..?06:58
*** e0ne has quit IRC07:02
*** CaptTofu has joined #tripleo07:05
*** CaptTofu has quit IRC07:11
*** jtomasek has joined #tripleo07:15
*** cd-undercloud has joined #tripleo07:17
cd-undercloud************** overcloud complete status=1 ************07:17
*** cd-undercloud has quit IRC07:17
*** cd-undercloud has joined #tripleo07:18
cd-undercloud************** overcloud complete status=6 ************07:18
*** cd-undercloud has quit IRC07:18
lifelessderekh_afk: looking07:30
lifelessderekh_afk: neutron has flip-flopped which agent ID it's using, again.07:30
lifelessderekh_afk: should be up now for l3, moving onto DHCP07:31
*** sirushti is now known as shortstop07:32
*** rpodolyaka1 has left #tripleo07:32
*** jprovazn has joined #tripleo07:32
lifelessderekh_afk: probably wants a rolling restart on compute07:40
lifelessci-overcloud back up07:42
lifelessStevenK: did you get setup-baremetal beaten into shape?07:42
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Create stack in right order  https://review.openstack.org/7626707:45
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Better error handling of Overcloud actions  https://review.openstack.org/7620707:45
*** mrunge has joined #tripleo07:51
*** vkozhukalov has joined #tripleo07:53
*** jcooley_ has joined #tripleo07:56
*** e0ne has joined #tripleo07:57
*** e0ne has quit IRC08:02
*** cd-undercloud has joined #tripleo08:05
cd-undercloud************** overcloud complete status=1 ************08:05
*** cd-undercloud has quit IRC08:05
*** saju_m has quit IRC08:10
*** saju_m has joined #tripleo08:10
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Create stack in right order  https://review.openstack.org/7626708:14
openstackgerritA change was merged to openstack/tripleo-heat-templates: Different Flavors for Tuskar roles  https://review.openstack.org/7589508:15
mariosping - re https://review.openstack.org/#/c/76267/ <-- is it WIP?08:20
marioslsmola: ^^^08:20
lsmolamarios: yes, but feel free to review :-)08:21
marios:)08:21
marioslsmola: thx, wasn't sure08:22
lsmolamarios: I am not figting with tox, it wants to regenerate config.sample for some reason08:22
marioslsmola: lets talk about this if we have call today. do you think its worth pulling out the stack_id fix so we can land that at least?08:22
*** ramishra has quit IRC08:23
*** ramishra has joined #tripleo08:23
lsmolamarios: hm, I guess it can go together08:23
lsmolamarios: i kind of depends on the different order of calling08:24
lsmolamarios: otherwise we would have to call Overcloud update on the end08:24
*** saju_m has quit IRC08:25
lsmolamarios: do you have also issue with generate.sample, when you run tox -ept27 ?08:25
*** saju_m has joined #tripleo08:26
marioslsmola: haven't seen that one yet08:26
lsmolamarios: ImportError: No module named stevedore.named08:26
lsmolamarios: hm probably my bad :-)08:29
lsmolamarios: I need to update test requirements08:29
*** al has quit IRC08:29
lsmolamarios: seems tox can't handle that08:30
*** al has joined #tripleo08:30
*** bauzas has joined #tripleo08:30
*** jistr has joined #tripleo08:30
*** rdopieralski has joined #tripleo08:30
*** rlandy has joined #tripleo08:36
*** jcooley_ has quit IRC08:39
*** jrist has quit IRC08:41
*** saurabhs has quit IRC08:41
rpodolyakamorning all08:42
lsmolamorning08:43
lsmolamarios: btw. do we need to still clone the templates and put the patch in config? when we have the templates in requirements?08:43
*** cd-undercloud has joined #tripleo08:46
cd-undercloud************** overcloud complete status=1 ************08:46
*** cd-undercloud has quit IRC08:46
marioslsmola: yeah because the templates themselves aren't packaged as part of the tripleo-heat-templates08:46
lsmolamarios: ha, ok08:46
*** shardy_afk is now known as shardy08:46
lsmolamarios: I thought it is packaging the whole repo08:47
marioslsmola: i also want to revisit the 'pulling from git master' - we didn't publish to pypi because 'it will go away'08:47
marioslsmola: but for now is fine08:47
*** e0ne has joined #tripleo08:47
marioslsmola: you can see in tripleo-heat-templates setup.cfg this is where the package is described08:47
lsmolamarios: ok08:47
openstackgerritA change was merged to openstack/tuskar-ui: Add overcloud_role method to Node  https://review.openstack.org/7634808:48
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Make the node details page prettier  https://review.openstack.org/7586008:48
openstackgerritA change was merged to openstack/tuskar-ui: Update overcloud role page according to wireframes  https://review.openstack.org/7581208:51
*** saju_m has quit IRC08:52
openstackgerritA change was merged to openstack/tuskar-ui: Actually get node profiles from the API  https://review.openstack.org/7476208:53
openstackgerritYongli He proposed a change to openstack/tuskar-ui: Exception message should not be localize  https://review.openstack.org/7679708:54
*** jrist has joined #tripleo08:54
openstackgerritYongli He proposed a change to openstack/tuskar: Exception message should not be localize  https://review.openstack.org/7679908:55
*** hashar has joined #tripleo08:55
lifelessmarios: I don't think t-h-t is going anywhere fast08:57
lifelessmarios: changing as the python logic moves to heat, yes but the package + template, noooo08:57
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Create stack in right order  https://review.openstack.org/7626708:57
openstackgerritA change was merged to openstack/tuskar-ui: Make the node details page prettier  https://review.openstack.org/7586008:58
marioslifeless: right, and agree08:59
*** ifarkas has joined #tripleo09:01
*** martyntaylor has joined #tripleo09:04
*** CaptTofu has joined #tripleo09:06
*** martyntaylor has quit IRC09:09
*** martyntaylor has joined #tripleo09:10
*** CaptTofu has quit IRC09:11
*** hashar_ has joined #tripleo09:12
lsmolalifeless: marios so it make sense to turn it into proper egg, right? templates included09:12
derekh_afklifeless: thanks, when the controller came back the neutron services came up with .novalocal in hostname09:15
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Create stack in right order  https://review.openstack.org/7626709:15
*** hashar has quit IRC09:15
derekh_afklifeless: I changed hostname back and restarted those services and they started reporting in withou .nocalocal09:15
derekh_afklifeless: what did I miss?09:15
*** cd-undercloud has joined #tripleo09:27
cd-undercloud************** overcloud complete status=1 ************09:27
*** cd-undercloud has quit IRC09:27
*** ramishra has quit IRC09:29
*** cd-undercloud has joined #tripleo09:29
cd-undercloud************** overcloud complete status=7 ************09:29
*** cd-undercloud has quit IRC09:29
openstackgerritA change was merged to openstack/tripleo-image-elements: Revert "Workaround broken Neutron requirements."  https://review.openstack.org/7647409:32
lifelessderekh_afk: In my report on the first incident I noted they'd move to .novalocal09:33
lifelessderekh_afk: and filed a bug09:33
lifelessderekh_afk: and moved the network etc to the new agent instance09:33
lifelessderekh_afk: so when you switched agent back around, you disabled the working agent and enabled the one without the network configured09:33
*** e0ne has quit IRC09:34
*** andreaf has quit IRC09:34
derekh_afklifeless: ahh crap, I missread your report (I did read it)09:34
*** e0ne has joined #tripleo09:34
lifelessderekh_afk: neutron net-list-on-dhcp-agent + neutron router-list-on-l3-agent to diagnose this09:34
*** derekh_afk is now known as derekh09:34
derekhlifeless: ok, thanks, will do better next time09:35
lifelessderekh: heh, I think you did fine09:36
lifelessderekh: we need more ops docs09:37
derekhlifeless: yup09:37
lifelessderekh: you communicated clearly; you didn't delay, but did look closely. all good stuff.09:37
lifelessderekh: were we down 14 hours? no. :)09:37
lifelessderekh: I haven't done a rolling restart of nova-compute, I think that will be needed to clear some of the error status VMs, but... I think we should try to gather some details to add into the bug this time09:38
StevenKlifeless: I *think* I have. It now works for new mode setups, tomorrow mornings plan is to take an axe to creates-nodes and testenv to create a legacy mode JSON and see if setup-baremetal still loves me.09:38
lifelesssince its not so overwhelming as last time09:38
StevenKlifeless: Bah, sprinkle in correct spellings to taste.09:38
*** lucasagomes has joined #tripleo09:38
lifelessStevenK: git checkout origin/master; devtest_testenv.sh ?09:39
derekhlifeless: I would have said 8 hours, when I sent my first message it was only down a few minutes09:39
StevenKlifeless: It isn't quite that simple, but close.09:39
lifelessderekh: well, still < 14 :)09:39
derekhlifeless: as I was using it at the time and lost connectivity to my vm09:39
lifelessderekh: I was getting drunk with dad at the time ;)09:39
StevenKlifeless: It is probably going to be git co -- <testenv/create-nodes> in the cache and then throw devtest.sh at the wall09:40
* derekh needs a few drink too :-)09:40
lifelessStevenK: th[ats not a combo that we use though09:40
lifelessStevenK: I think you want --use-existing + a testenv, not mixed scripts09:41
StevenKI would have preferred to have setup-baremetal run against the undercloud for the new setup, but it didn't come up09:41
StevenKlifeless: Why, testenv didn't change much, and I want a properly populated JSON with node-macs.09:42
lifelesssimpler to reproduce what CI does09:42
lifelesswhich is latest code, but reuse envirnomnet09:42
openstackgerritA change was merged to openstack/tuskar: Better error handling of Overcloud actions  https://review.openstack.org/7620709:43
lifelessStevenK: look at what toci_gate actually runs09:43
lifelessStevenK: I mean, fundamentally - up to you, but I'd clean-env,d evtest_testenv in trunk, swich to my feature and run with --existing-environment09:44
*** ramishra has joined #tripleo09:45
*** jp_at_hp has joined #tripleo09:47
lifelessjp_at_hp: could you try https://review.openstack.org/#/c/76636/09:52
*** d0ugal has quit IRC09:54
*** mrunge is now known as mrunge_away09:57
mordredNg, lifeless: the rabbit cluster work is started by someone? or tbd?10:08
lifelessnearly done10:09
lifelessjprovazn <---10:09
*** pbelanyi has joined #tripleo10:09
*** jeremydei has joined #tripleo10:09
*** athomas has quit IRC10:09
jprovaznmordred: lifeless: yep. reviews welcome: https://review.openstack.org/#/c/70238/  and then https://review.openstack.org/#/c/68392/10:10
mordredjprovazn: cool! thanks10:11
jprovaznnp10:11
*** Hefeweizen has quit IRC10:13
*** cd-undercloud has joined #tripleo10:17
cd-undercloud************** overcloud complete status=1 ************10:17
*** cd-undercloud has quit IRC10:17
*** akrivoka has joined #tripleo10:28
*** d0ugal has joined #tripleo10:31
killer_princei created a new element and then created a os-refresh-config/configure.d and placed a script under this directory. I also defined dependency on os-refresh-config for this new element. when i am building the image using this element, i am not able to find this script on the image under /opt/stack/os-config-refresh/configure.d10:32
killer_princeany idea what i missed in the whole step ?10:32
*** athomas has joined #tripleo10:33
lifelesshave a look in /opt/stack/os-refresh-config/configure.d :)10:35
*** saju_m has joined #tripleo10:35
killer_princelifeless10:36
lifelesskiller_prince:10:36
killer_princethere is no such directory when image is booted10:37
lifelessinteresting10:37
lifelessuhm10:37
lifelessthe build log should give enough info to debug this10:37
lifelessor you can drop into a shell in the build10:37
killer_princelifeless: I am checking the build log now..10:37
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Proof of concept of error handling in the API  https://review.openstack.org/7681910:40
killer_prince ln -s /opt/stack/venvs/os-refresh-config/bin/os-refresh-config /usr/local/bin/os-refresh-config10:43
killer_princefor d in pre-configure.d configure.d migration.d post-configure.d10:43
killer_princeinstall -m 0755 -o root -g root -d /opt/stack/os-config-refresh/pre-configure.d10:43
killer_prince for d in pre-configure.d configure.d migration.d post-configure.d10:43
killer_princeinstall -m 0755 -o root -g root -d /opt/stack/os-config-refresh/configure.d10:44
lifelesssorery yeah I meant os-config-refresh. my figures btrayed me10:44
killer_princeall these are there but the script that i placed in my custom element is not there..10:44
killer_princelet me pass the log contents...10:45
lifelesskiller_prince: its idnight here10:46
lifelesskiller_prince: I hope someone will look at it, but I won't) too late!10:47
*** jprovazn has quit IRC10:47
*** CaptTofu has joined #tripleo10:48
killer_princelifeless: np..10:48
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Proof of concept of error handling in the API  https://review.openstack.org/7681910:51
*** CaptTofu has quit IRC10:52
killer_princehere are the logs http://paste.openstack.org/show/70190/10:54
*** CaptTofu has joined #tripleo10:56
*** cd-undercloud has joined #tripleo10:57
cd-undercloud************** overcloud complete status=1 ************10:57
*** cd-undercloud has quit IRC10:57
*** jcooley_ has joined #tripleo11:01
openstackgerritlifeless proposed a change to openstack/diskimage-builder: Better apt-sources docs.  https://review.openstack.org/7303511:02
openstackgerritlifeless proposed a change to openstack/diskimage-builder: Improve local-config proxy handling.  https://review.openstack.org/7663611:02
*** jcooley_ has quit IRC11:11
jangwrt. sunnyvale hackathon next week: does anyone know if there's transport lined up twixt the hotel and the venue?11:14
*** jprovazn has joined #tripleo11:14
*** nosnos has quit IRC11:14
*** ramishra has quit IRC11:28
*** ramishra has joined #tripleo11:28
openstackgerritDmitry Tantsur proposed a change to openstack/tuskar-ui: Manage image ID's for node profile  https://review.openstack.org/7582311:30
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Using flavors from Roles  https://review.openstack.org/7682611:31
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Refactor the overcloud views tests  https://review.openstack.org/7616111:33
*** cd-undercloud has joined #tripleo11:35
cd-undercloud************** overcloud complete status=1 ************11:35
*** cd-undercloud has quit IRC11:35
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Using flavors from Roles  https://review.openstack.org/7682611:36
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Proof of concept of error handling in the API  https://review.openstack.org/7681911:38
killer_princei created a new element and then created a os-refresh-config/configure.d and placed a script under this directory. I also defined dependency on os-refresh-config for this new element. when i am building the image using this element, i am not able to find this script on the image under /opt/stack/os-config-refresh/configure.d11:47
killer_princeany idea what i missed in the whole step ?11:47
killer_princehere are the logs http://paste.openstack.org/show/70190/11:47
*** jcooley_ has joined #tripleo11:53
killer_princeokay. i found that there is a configure.d at /11:53
killer_princeand my scripts do exist there..11:54
killer_princeis this expected or some kind of bug in diskimage-builder11:54
*** jcooley_ has quit IRC11:59
*** martyntaylor has left #tripleo12:01
slaglekiller_prince: what's the name of the new element you added? (looking through that log)12:03
*** cd-undercloud has joined #tripleo12:04
cd-undercloud************** overcloud complete status=128 ************12:04
*** cd-undercloud has quit IRC12:04
killer_princeslagle: localboot12:08
slaglekiller_prince: according to that log, the image build failed for a different reason12:08
slaglemissing gcc in the image12:08
slaglewhat image type are you trying to build? rhel?12:08
killer_princeyes..12:09
*** hashar_ has quit IRC12:09
killer_princeslagle: sorry, i pasted the wrong logs.. let me paste it again..12:10
slaglederekh: are we merging stuff? or still blocked on CI?12:11
slaglei saw a couple of commits flow through w/o successful runs...12:11
killer_princeslagle: http://paste.openstack.org/show/70214/12:11
derekhslagle: Stuff should be back working again, zuul seems to be running jobs but a lot are queued,12:15
derekhslagle: it doesn't seem to be running as many jobs as it could, there is a lot of vm;s in an error state since the outage12:15
derekhslagle: so maybe it has hit its quota, gonna restart nova-compute on each compute node to see if that help get the nodes out of the error start12:16
slaglekiller_prince: line 1350 is the error12:16
slaglekiller_prince: missing arparse for some reason, so the os-refresh-config scripts did not get copied onto the image12:17
slaglekiller_prince: looks like a valid bug. argparse should be added to os-refresh-config's requirements.txt12:17
derekhslagle: Done, lets give it a few minutes and see if it helps12:19
killer_princeSo can i log a bug..? or is it specific to RHEL images only..12:21
openstackgerritJames Slagle proposed a change to openstack/os-refresh-config: Add argparse to requirements.txt  https://review.openstack.org/7684412:23
*** mrunge_away is now known as mrunge12:23
slaglekiller_prince: ^^ that patch should help12:23
derekh12:24
slaglederekh: ok, cool12:26
derekhslagle: that seems to have help, looks like zuul is spinning up more instances and the error ones are being delete, so I think we're back waiting on CI, I stall havn't figured out why the overcloud job is failing got a bit distracted12:26
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Add set -eux  https://review.openstack.org/7684712:29
*** CaptTofu has quit IRC12:30
killer_princeslagle: how do i test it as it gets pulled during the build.. any ideas..?12:32
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Fix of Overcloud Update  https://review.openstack.org/7685012:33
slaglekiller_prince: you need to set some environment variables that the source-repositories element will use12:33
slaglekiller_prince: take a look at https://github.com/openstack/diskimage-builder/tree/master/elements/source-repositories/README.md12:34
slaglekiller_prince: about half way down the page there is an example of using a change from a review in gerrit12:34
killer_princeslagle: okay.. thanks12:36
*** CaptTofu has joined #tripleo12:40
*** weshay has joined #tripleo12:44
*** cd-undercloud has joined #tripleo12:45
cd-undercloud************** overcloud complete status=1 ************12:45
*** cd-undercloud has quit IRC12:45
openstackgerritLadislav Smola proposed a change to openstack/tuskar-ui: Working scaling  https://review.openstack.org/7685412:48
*** saju_m has quit IRC12:53
openstackgerritLadislav Smola proposed a change to openstack/tuskar-ui: Working scaling  https://review.openstack.org/7685412:56
*** morazi has joined #tripleo13:09
*** hashar has joined #tripleo13:11
*** hashar_ has joined #tripleo13:12
*** ohadlevy_ is now known as ohadlevy13:13
*** ohadlevy has joined #tripleo13:13
*** cd-undercloud has joined #tripleo13:15
cd-undercloud************** overcloud complete status=128 ************13:15
*** cd-undercloud has quit IRC13:15
*** hashar has quit IRC13:16
*** gcha has joined #tripleo13:24
openstackgerritA change was merged to openstack/tripleo-incubator: Avoid races with Neutron by waiting for it.  https://review.openstack.org/7575413:24
*** jdob has joined #tripleo13:25
Ngmorning13:25
killer_princeslagle: i tried DIB_REPOLOCATION_os\-refresh\-config=https://review.openstack.org/openstack/os-refresh-config13:27
killer_princeslagle: but i get bash: DIB_REPOLOCATION_os-refresh-config=https://review.openstack.org/openstack/os-refresh-config: No such file or directory13:27
killer_princeslagle: am i doing it in the right way..?13:28
openstackgerritA change was merged to openstack/tripleo-image-elements: Add fedora-rdo-icehouse element  https://review.openstack.org/7341113:28
*** lucasagomes is now known as lucas-hungry13:28
slaglekiller_prince: it should be DIB_REPOLOCATION_os_refresh_config and DIB_REPOREF_os_refresh_config (replace dashes with underscores)13:29
*** jang has quit IRC13:29
killer_princeslagle: aha.. k..13:30
slaglekiller_prince: appears that the README doesn't mention that's required...could be a good doc update patch :)13:30
killer_princeslagle: yup..13:30
openstackgerritA change was merged to openstack/tripleo-image-elements: Fix typo  https://review.openstack.org/7365513:31
openstackgerritA change was merged to openstack/tripleo-image-elements: Install lvm2 package for cinder  https://review.openstack.org/7365613:32
*** andreaf has joined #tripleo13:33
*** ramishra has quit IRC13:40
*** jcooley_ has joined #tripleo13:40
*** ramishra has joined #tripleo13:42
openstackgerritDmitry Tantsur proposed a change to openstack/tuskar-ui: Prevent node profile of deployed role from being deleted  https://review.openstack.org/7686413:46
*** jcooley_ has quit IRC13:46
*** hashar_ is now known as hashar13:49
* Ng still pretty unwell today, so will not be super useful. got a doctor's appointment in a couple of hours to see if they can medicate me at all (feeling a lot like my plague is building into an ear infection, yay)13:51
*** dprince has joined #tripleo13:51
rpodolyakaget well soon, Ng!13:52
*** cd-undercloud has joined #tripleo13:53
cd-undercloud************** overcloud complete status=1 ************13:53
*** cd-undercloud has quit IRC13:53
*** noslzzp has joined #tripleo13:53
slagledon't share too many details, TSA might deny you entry at the border13:55
*** lblanchard has joined #tripleo13:58
*** sballe has joined #tripleo14:02
*** tzumainn has quit IRC14:03
*** tzumainn has joined #tripleo14:03
*** hewbrocca has joined #tripleo14:10
killer_princeslagle: i used DIB_REPOLOCATION_os_refresh_config and DIB_REPOREF_os_refresh_config but i still get same error about argparse14:14
rpodolyakakiller_prince: isn't os-refresh-config installed by pip rather than by source-repository element?14:17
killer_princerpodolyaka: yes.. you are right. but slagle suggested me to use this.14:20
slagleos-refresh-config doesn't use source-repository?14:22
killer_princemay be we can check in the logs that i provided a while back..14:23
*** ramishra has quit IRC14:23
slaglei checked, it doesn't14:24
slagleso, i'm not sure how to test this review14:24
slaglelikely there is not an exposed way14:24
rpodolyakaslagle: indeed, 8a2a8c71741348a3b2a77bfa077c68ea84a918bb14:24
rpodolyaka(https://review.openstack.org/#/c/66352/)14:24
openstackgerritDmitry Tantsur proposed a change to openstack/tuskar-ui: Prevent node profile of deployed role from being deleted  https://review.openstack.org/7686414:24
*** CaptTofu has quit IRC14:25
rpodolyakaslagle: killer_prince: from what I see, we can only fix this in os-refresh-config and then cut a new release14:25
rpodolyakaslagle: killer_prince: or try an ugly hack to test it - modify os-refresh-config element to install it from gerrit review :)14:26
killer_princerpodolyaka: and when is that gonna happen..14:26
rpodolyakakiller_prince: we cut new releases each week, so I can just release it today, if needed14:26
killer_princerpodolyaka: would be great if you could do that. I need this change for some custom element that I am writing now.. its kind of blocker for me..14:27
*** cd-undercloud has joined #tripleo14:29
cd-undercloud************** overcloud complete status=128 ************14:29
*** cd-undercloud has quit IRC14:29
*** lucas-hungry is now known as lucasagomes14:30
rpodolyakakiller_prince: no problem, we just need two core reviewers to get slagle's fix approved14:31
killer_princerpodolyaka: thanks a lot..14:32
*** jcooley_ has joined #tripleo14:35
*** ftcjeff has joined #tripleo14:36
fungiderekh: so it looks like you've got the ci cloud up and running again in the past couple hours... i see a handful of nodes nodepool thinks are in there for a very long time, so i'll get those cleaned up to open up a little more capacity for you14:39
derekhfungi: cool, thanks do I need to delete vm's in error state ?14:39
fungiderekh: well, i'll submit deletes through nodepool for any nodes in your cloud in any one state for more than 4 hours and we'll see where that leaves us. it will at least submit nova deletes for them if they exist on the provider end14:40
derekhfungi: ok14:41
*** jcooley_ has quit IRC14:41
fungiover a dozen at least 4 hours in delete or ready state (some up to ~15 hours)14:42
fungioh, actually there was one in delete for over 35 hours, and another over 2814:43
fungionce those are clear i'll look at any which are in any state other than used for at least an hour (that will allow us to avoid killing nodes which are actively running jobs)14:43
derekh15 hours is probably when the outage started. ok thanks14:44
fungiassuming that basically anything which has been building, deleting or in a ready state for an hour is probably broken14:44
derekhsounds about right14:46
*** e0ne_ has joined #tripleo14:46
*** CaptTofu has joined #tripleo14:48
*** spzala has joined #tripleo14:49
*** e0ne has quit IRC14:49
openstackgerritA change was merged to openstack/tuskar-ui: Manage image ID's for node profile  https://review.openstack.org/7582314:52
*** ramishra has joined #tripleo14:54
*** ramishra has quit IRC14:56
*** ramishra has joined #tripleo14:57
*** matty_dubs|gone is now known as matty_dubs15:00
*** ramishra has quit IRC15:02
killer_princeyeah.. so now i can see the scripts there under /opt/stack/os-config-refresh/configure.d/ but how do I know if they were executed.. any log file..15:05
slaglekiller_prince: they'll be executed when the image boots. os-refresh-config is triggered by the os-collect-config service15:06
*** ramishra has joined #tripleo15:07
slaglei *think* there is a /var/log/os-collect-config.log15:07
killer_princei have the instance running... so is there any log that i can check..15:07
slagleis this rhel 6?15:07
killer_princei do not have that file in running instance..15:07
*** noslzzp has quit IRC15:07
killer_princeslagle: yes..15:07
*** noslzzp has joined #tripleo15:08
*** cd-undercloud has joined #tripleo15:08
cd-undercloud************** overcloud complete status=1 ************15:08
*** cd-undercloud has quit IRC15:08
slagletypically, i would say check the stdout of the os-collect-config service, but rhel 6 does not use systemd, so i'm not sure where the stdout has gone...15:08
killer_princeRHEL 6.5 x86_64 to be precise..15:08
killer_princei am asking as the changes it was supposed to make did not happen.. so is there any way i can run it manually and check what went wrong..15:09
killer_princesorry.. i am new to this and so lots of dumb doubts..15:10
slaglei think the issue is that we only install a systemd service file for os-collect-config, which is obviously not going to work on rhel 615:10
slagleyou can run 'os-collect-config' as root to trigger it15:11
slagleos-collect-config --one-time to just run it once15:11
killer_princeso i have one doubt.. the scripts that we put under os-refresh-config is run by os-collect-config.. correct..?15:13
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Ceilo/Ironic/Tuskar: don't hard code venv dir  https://review.openstack.org/7688715:13
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Heat/Horizon/Tempest: don't hard code venv dir  https://review.openstack.org/7688815:13
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Neutron: don't hard code the venv dir  https://review.openstack.org/7662615:14
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Link db sync utilities to /usr/local/bin...  https://review.openstack.org/7657915:14
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Glance/Cinder/Keystone/Swift: don't hard code venv  https://review.openstack.org/7666615:14
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Nova: don't hard code the venv dir  https://review.openstack.org/7661915:14
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Stop using the os-svc-install -n option.  https://review.openstack.org/7660415:14
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Drop the os-svc-install -n -c options.  https://review.openstack.org/7660515:14
slaglekiller_prince: os-collect-config runs os-refresh-config, which runs the scripts under /opt/stack/os-config-refresh15:14
killer_princeI see that os-collect-config is quering for user data and instance meta data..15:15
*** blamar_ has joined #tripleo15:16
*** rdopiera1ski has joined #tripleo15:16
*** SpamapS_ has joined #tripleo15:16
*** dprince_ has joined #tripleo15:16
*** vipuls has joined #tripleo15:21
*** dprince has quit IRC15:21
*** gcha has quit IRC15:21
*** jprovazn has quit IRC15:21
*** athomas has quit IRC15:21
*** d0ugal has quit IRC15:21
*** rdopieralski has quit IRC15:21
*** blamar has quit IRC15:21
*** SpamapS has quit IRC15:21
*** vipul has quit IRC15:21
*** andreaf has quit IRC15:21
*** blamar_ is now known as blamar15:21
*** ramishra has quit IRC15:21
*** ramishra has joined #tripleo15:22
*** funzo has quit IRC15:22
*** funzo_ has joined #tripleo15:22
killer_princei also tried running os-refresh-config and it executed my scripts.. but os-collect-config did not execute os-refresh-config. to confirm i tried grep os-refresh-config on os-collect-config /opt/stack/venvs/os-collect-config but didnt get any thing.. am i missing anything..?15:23
*** mrunge has quit IRC15:24
slagleit's configured in /etc/os-collect-config.conf15:27
*** CaptTofu has quit IRC15:28
*** jprovazn has joined #tripleo15:28
*** CaptTofu has joined #tripleo15:28
*** athomas has joined #tripleo15:28
*** d0ugal has joined #tripleo15:28
killer_princeslagle: aha.. so there its hidden..15:29
*** gcha has joined #tripleo15:29
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Neutron: don't hard code the venv dir  https://review.openstack.org/7662615:29
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Link db sync utilities to /usr/local/bin...  https://review.openstack.org/7657915:29
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Heat/Horizon/Tempest: don't hard code venv dir  https://review.openstack.org/7688815:29
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Glance/Cinder/Keystone/Swift: don't hard code venv  https://review.openstack.org/7666615:29
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Ceilo/Ironic/Tuskar: don't hard code venv dir  https://review.openstack.org/7688715:29
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Nova: don't hard code the venv dir  https://review.openstack.org/7661915:29
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Stop using the os-svc-install -n -c options.  https://review.openstack.org/7660415:30
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Drop the os-svc-install -n -c options.  https://review.openstack.org/7660515:30
*** hashar has quit IRC15:30
*** hashar has joined #tripleo15:32
*** CaptTofu has quit IRC15:32
*** funzo_ is now known as funzo15:33
*** killer_prince has quit IRC15:41
*** lazy_prince is now known as killer_prince15:41
*** ramishra has quit IRC15:41
*** ramishra has joined #tripleo15:42
*** martyntaylor has joined #tripleo15:42
*** panda has joined #tripleo15:45
*** cd-undercloud has joined #tripleo15:46
cd-undercloud************** overcloud complete status=1 ************15:46
*** cd-undercloud has quit IRC15:46
*** panda_ has quit IRC15:48
derekhso now the the cloud is up and running again, looks like network access from the VM's is extreemly slow, so expect things to be failing15:52
derekhe.g. this job has been running for about 2 hours https://jenkins03.openstack.org/job/check-tripleo-overcloud-precise/42/consoleFull15:52
derekhand still hasn't finished download the cloud image....15:53
*** CaptTofu has joined #tripleo15:53
derekhthis looks like what we hit a while back with a MTU problem https://bugs.launchpad.net/neutron/+bug/1270646 but setting the MTU on vm's doesn't seem to have an effect15:54
derekheth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc pfifo_fast state UP qlen 100015:55
*** ramishra has quit IRC16:00
derekhlifeless, devananda, Ng , SpamapS_ , jog0 , GheRivero , derekh , dprince_ , slagle ^^^ any ideas ?16:00
jog0derekh: sorry my network foo is very weak16:01
derekhjog0: as is mine16:01
*** andreaf has joined #tripleo16:08
*** coolsvap has quit IRC16:10
slaglederekh: i've never seen slow network as a symptom of that bug, only dropped packets due to exceeding mtu size16:15
slagleand for that, i typically *increase* mtu to 154216:15
slaglebut, i don't think that applies in this case16:16
derekhslagle: ok, tried it just incase didn't help16:21
*** bauzas has quit IRC16:22
cody-somervillelifeless: Are you around?16:22
*** cd-undercloud has joined #tripleo16:25
cd-undercloud************** overcloud complete status=1 ************16:25
*** cd-undercloud has quit IRC16:25
*** jcooley_ has joined #tripleo16:29
*** cd-undercloud has joined #tripleo16:31
cd-undercloud************** overcloud complete status=128 ************16:31
*** cd-undercloud has quit IRC16:31
*** vkozhukalov has quit IRC16:36
*** jcooley_ has quit IRC16:36
*** jcooley_ has joined #tripleo16:38
*** vkozhukalov has joined #tripleo16:39
*** rdopiera1ski has quit IRC16:54
*** pbelanyi has quit IRC16:56
*** d0ugal has quit IRC16:58
*** morazi has quit IRC16:58
*** jcooley_ has quit IRC16:59
*** jistr has quit IRC17:00
*** morazi has joined #tripleo17:00
*** killer_prince is now known as lazy_prince17:01
*** vkozhukalov has quit IRC17:05
*** e0ne_ has quit IRC17:06
*** lblanchard has quit IRC17:06
*** andreaf has quit IRC17:08
*** jwalls has joined #tripleo17:08
*** jp_at_hp has quit IRC17:08
*** ftcjeff has quit IRC17:08
*** jp_at_hp has joined #tripleo17:09
openstackgerritDustin J. Mitchell proposed a change to openstack/diskimage-builder: Update README formatting and content  https://review.openstack.org/7548017:10
*** martyntaylor has quit IRC17:13
*** jcooley_ has joined #tripleo17:17
*** sdake_ has joined #tripleo17:24
*** bauzas has joined #tripleo17:26
*** matty_dubs is now known as matty_dubs|lunch17:26
derekhlifeless, devananda, Ng , SpamapS_ , jog0 , GheRivero , derekh , dprince_ , slagle gotta run, didn't make much progress https://etherpad.openstack.org/p/cloud-outage17:42
*** jcooley_ has quit IRC17:45
*** lblanchard has joined #tripleo17:45
*** derekh has quit IRC17:46
SpamapS_ugh17:48
*** SpamapS_ is now known as SpamapS17:48
*** cd-undercloud has joined #tripleo17:48
cd-undercloud************** overcloud complete status=1 ************17:48
*** cd-undercloud has quit IRC17:48
*** SpamapS has quit IRC17:48
*** SpamapS has joined #tripleo17:48
*** rlandy is now known as rlandy|bbl17:52
SpamapS64 bytes from 10.10.16.172: icmp_seq=1 ttl=64 time=0.577 ms17:55
SpamapS64 bytes from 10.10.16.172: icmp_seq=1 ttl=64 time=0.599 ms (DUP!)17:55
*** rsacharya has quit IRC17:55
SpamapSweird17:55
*** rsacharya has joined #tripleo17:58
*** cadenzajon has joined #tripleo18:00
*** hashar has quit IRC18:00
*** gcha has quit IRC18:01
*** andreaf has joined #tripleo18:03
*** jp_at_hp has quit IRC18:06
*** jcooley_ has joined #tripleo18:18
*** pblaho has quit IRC18:18
*** andreaf has quit IRC18:21
*** akrivoka has quit IRC18:21
openstackgerritDmitry Tantsur proposed a change to openstack/tuskar-ui: Prevent node profile of deployed role from being deleted  https://review.openstack.org/7686418:21
SpamapSwhat is 10.10.16.172 anyway? The old seed?18:21
* SpamapS is a bit confused why DHCP is setting a DNS server18:22
*** gcha has joined #tripleo18:22
*** shakayumi has joined #tripleo18:22
openstackgerritBen Nemec proposed a change to openstack/tripleo-incubator: Stop devtest if user was not in libvirtd group  https://review.openstack.org/7692618:24
*** rsacharya has left #tripleo18:25
*** athomas has quit IRC18:25
*** cd-undercloud has joined #tripleo18:27
cd-undercloud************** overcloud complete status=1 ************18:27
*** cd-undercloud has quit IRC18:27
*** shakayumi has quit IRC18:29
*** shakayumi has joined #tripleo18:30
*** shakayumi has quit IRC18:30
*** saurabhs has joined #tripleo18:30
* SpamapS struggles to bend brain to map jenkins -> instances18:32
lifelessjust toggled gro off on the ci-overcloud18:32
lifelessshould address network performance18:33
SpamapSlifeless: I just turned it off about 15 minutes18:33
SpamapSago18:33
* SpamapS noted it in the etherpad like a good boy ;)18:33
*** cadenzajon_ has joined #tripleo18:33
SpamapSlifeless: thank god you're here.. so how do I map tripleo-precise-tripleo-test-cloud-1728731 to an instance in ci-overcloud ?18:33
lifelessSpamapS: ah haven't looked there yet, just saw derekh's pings18:33
lifelessnova show tripleo-precise-tripleo-test-cloud-172873118:33
lifelessas admin18:33
SpamapSit.. can't be.. that easy18:33
SpamapSERROR: No server with a name or ID of 'tripleo-precise-tripleo-test-cloud-1728731' exists.18:34
lifelessuse the uuid18:34
*** matty_dubs|lunch is now known as matty_dubs18:34
SpamapSnova list --all-tenants grepping for it works18:34
lifelessnov show $(nova list --all-tenants | awk '/tripleo-precise-tripleo-test-cloud-1728731/ {print $2}')18:34
*** cadenzajon has quit IRC18:35
*** e0ne has joined #tripleo18:36
*** saju_m has joined #tripleo18:37
SpamapSlifeless: and the job executing on that VM did in fact work out.18:39
SpamapSfirst green \o/18:39
SpamapSlifeless: so I think I saw you asking how we can get gro forced off..18:39
*** athomas has joined #tripleo18:39
SpamapSlifeless: we can just put in an if-up.d script, and I think sysconfig has something similar that they can set in variables.18:40
*** saju_m has quit IRC18:41
*** d0ugal has joined #tripleo18:41
*** d0ugal has quit IRC18:41
*** d0ugal has joined #tripleo18:41
lifelessSpamapS: no, I wasn't asking that18:46
lifelessSpamapS: https://etherpad.openstack.org/p/tripleo-network-configuration18:47
SpamapSoh18:47
lifelessmight have given you that impression18:47
SpamapSmmmk18:47
lifelessits also a substantial scaling issue to have it off18:47
lifelessSpamapS: hello metadata :)18:47
SpamapSlifeless: right we're putting more work on the kernel right? But isn't this basically a GRE/OpenVswitch limitation?18:48
lifelessSpamapS: see the df bit bug18:48
*** shakayumi has joined #tripleo18:50
*** shakayumi has joined #tripleo18:51
*** shakayumi has quit IRC18:51
SpamapSlifeless: I really don't understand the bug, but the symptom, as I'm understanding it, is that we are setting DF and thus causing our packets to be dropped without recourse?18:51
*** saju_m has joined #tripleo18:51
*** shakayumi has joined #tripleo18:51
*** shakayumi has quit IRC18:52
*** shakayumi has joined #tripleo18:52
*** shakayumi has quit IRC18:53
jdobhaving an issue on my devtest setup: https://gist.github.com/jdob/9256555   does that look familiar to anyone?18:53
*** shakayumi has joined #tripleo18:53
*** vkozhukalov has joined #tripleo18:55
*** cd-undercloud has joined #tripleo19:04
cd-undercloud************** overcloud complete status=1 ************19:04
*** cd-undercloud has quit IRC19:04
*** cwolferh has quit IRC19:13
*** lblanchard has quit IRC19:19
*** cwolferh has joined #tripleo19:20
*** saju_m has quit IRC19:35
*** cd-undercloud has joined #tripleo19:37
cd-undercloud************** overcloud complete status=128 ************19:37
*** cd-undercloud has quit IRC19:37
*** cd-undercloud has joined #tripleo19:42
cd-undercloud************** overcloud complete status=128 ************19:42
*** cd-undercloud has quit IRC19:42
*** akuznetsov has quit IRC19:43
*** akuznetsov has joined #tripleo19:43
bnemecjdob: I sometimes have issues with connectivity to 192.0.2.1.  If you ping it do you get something about Next-Hop?19:45
*** jang1 has joined #tripleo19:50
*** rlandy|bbl is now known as rlandy19:50
openstackgerritRichard Su proposed a change to openstack/tripleo-image-elements: rsync configurations to support multi-node swift  https://review.openstack.org/7610519:51
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Add -i options to os-svc-install/os-svc-daemon...  https://review.openstack.org/7695219:59
*** lblanchard has joined #tripleo20:00
*** dshurtle has joined #tripleo20:01
*** hashar has joined #tripleo20:10
*** bauzas has quit IRC20:14
jdobbnemec: yes!20:14
jdobFrom 192.168.122.80: icmp_seq=2 Redirect Host(New nexthop: 192.168.122.1)20:14
bnemecjdob: Okay, I'm not entirely sure what causes that, but I think re-creating the route generally fixes it for me.20:17
jdobhrm, I had tried recreating the route20:17
lifelessyour traffic is being picked up on the wrong interface20:19
lifelessso its telling you where the router is rather than routing it20:19
*** cd-undercloud has joined #tripleo20:20
cd-undercloud************** overcloud complete status=128 ************20:20
*** cd-undercloud has quit IRC20:20
lifelessone reason it might think its teh wrong interface is because the seed doesn't have a  better route to 192.0.2.0/2420:21
lifelessso you need to recreate the route as bnemec says - on the seed node, not on the host20:21
jdobman is my networking knowledge way too limited20:24
lifelessjdob: live n learn :)20:27
jdobthis reminds me of how I got started in computers as a kid, I'd break my dad's and have to learn how to fix it before he killed me20:27
jdoblifeless: when you say seen node, you mean the undercloud guest right?20:29
lifelessseed20:29
jdober, crap, typo.20:29
*** epim has joined #tripleo20:31
*** newell has joined #tripleo20:38
*** jtomasek has quit IRC20:42
*** dshurtle has quit IRC20:47
jdobthere we go :D20:54
jdob\o/20:54
jdobthanks lifeless and bnemec20:55
*** ifarkas has quit IRC20:56
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Update elements to use os-svc-daemon -i.  https://review.openstack.org/7696321:00
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Update elements to use os-svc-install -i.  https://review.openstack.org/7696421:00
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Wire in OPENSTACK_VENV_DIR default if set  https://review.openstack.org/7696521:00
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Wire in _EXTRA_INSTALL_OPTS...  https://review.openstack.org/7696621:00
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Add a new common-venv element  https://review.openstack.org/7696721:00
*** cd-undercloud has joined #tripleo21:00
cd-undercloud************** overcloud complete status=1 ************21:00
*** cd-undercloud has quit IRC21:00
*** cd-undercloud has joined #tripleo21:02
cd-undercloud************** overcloud complete status=7 ************21:02
*** cd-undercloud has quit IRC21:02
*** e0ne has quit IRC21:05
*** rbrady is now known as rbrady-bbl21:09
fungilifeless: devananda: Ng: SpamapS_: jog0_: GheRivero: derekh: dprince: slagle: reviewing scrollback, it's unclear to me whether you expect ci-overcloud should be up or not. just in case you believe it should, the nodepool server is still getting...21:09
fungirequests.exceptions.ConnectionError: HTTPSConnectionPool(host='ci-overcloud.tripleo.org', port=13000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 110] Connection timed out)21:09
fungiand indeed, 'telnet ci-overcloud.tripleo.org 13000' times out for me as well21:12
lifelessit should be up21:21
lifelessnuts, I ahve some elocal but will look shortly21:22
fungik21:23
*** spzala has quit IRC21:24
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Wire in OPENSTACK_VENV_DIR default if set  https://review.openstack.org/7696521:26
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Wire in _EXTRA_INSTALL_OPTS...  https://review.openstack.org/7696621:26
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Add a new common-venv element  https://review.openstack.org/7696721:26
*** dprince_ has quit IRC21:28
*** vipuls is now known as vipul21:28
*** d0ugal has quit IRC21:29
*** athomas has quit IRC21:32
*** shakayumi has quit IRC21:35
lifelessfungi: https://etherpad.openstack.org/p/cloud-outage21:37
fungisuch fun! it's your old friend colonel panic21:40
fungidied from "not much information"21:40
*** shakayumi has joined #tripleo21:41
*** shakayumi has quit IRC21:41
fungithis is the second time in 24 hours, presumably? or did derekh recover it from a different failure earlier today?21:41
lifelesssecond on 24h yes21:44
lifelessNg was looking into a bios upgrade21:44
lifelessthen got sick21:44
lifelessand I've not had cycles to pursue that21:44
*** d0ugal has joined #tripleo21:47
*** d0ugal has quit IRC21:47
*** d0ugal has joined #tripleo21:47
* lifeless stabs the ci control node harder21:51
*** cd-undercloud has joined #tripleo21:51
cd-undercloud************** overcloud complete status=1 ************21:51
*** cd-undercloud has quit IRC21:51
*** jprovazn has quit IRC21:53
*** cd-undercloud has joined #tripleo21:54
cd-undercloud************** overcloud complete status=128 ************21:54
*** cd-undercloud has quit IRC21:54
lifelesshah21:56
lifelessilo do your thing21:56
lifeless'power off'...21:56
lifelessminute later21:56
lifelesspower still on21:56
lifelesspower off hard time21:57
tchaypookay, i have a registered token. now its time to set up the VPN21:57
openstackgerritJonathan Brownell proposed a change to openstack/diskimage-builder: Fix syntax error in GRUB_OPTS env var handling  https://review.openstack.org/7698322:01
*** rlandy has quit IRC22:04
*** jdob has quit IRC22:07
lifelessnetboot failure22:07
tchaypoStevenK: figured out how to trust the CA22:10
*** e0ne has joined #tripleo22:16
*** e0ne has quit IRC22:21
*** shardy is now known as shardy_afk22:23
StevenKtchaypo: Oooh22:25
*** gcha has quit IRC22:31
*** rbrady-bbl is now known as rbrady22:35
*** spzala has joined #tripleo22:35
*** cd-undercloud has joined #tripleo22:38
cd-undercloud************** overcloud complete status=1 ************22:38
*** cd-undercloud has quit IRC22:38
*** bauzas has joined #tripleo22:40
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Wire in _EXTRA_INSTALL_OPTS...  https://review.openstack.org/7696622:41
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Add a new common-venv element  https://review.openstack.org/7696722:41
*** lblanchard has quit IRC22:46
*** vkozhukalov has quit IRC22:47
*** al has quit IRC22:48
*** CaptTofu has quit IRC22:48
*** jang1 has quit IRC22:49
*** al has joined #tripleo22:50
*** hashar has quit IRC22:55
*** al has quit IRC22:57
*** al has joined #tripleo22:57
*** al has quit IRC22:57
*** CaptTofu has joined #tripleo22:59
*** al has joined #tripleo23:02
tchayposo here's a neat trick23:04
tchaypoI spotted https://review.openstack.org/#/c/71996/12/stack.sh and wondered if it would help me track down the errors I'm having running devtest23:04
tchaypoit turns out that with set -o errexit, every time I run devtest_setup.sh, tmux exits23:04
lifelesslol23:05
SpamapSdoh23:06
greghayneswierd, I do all my stuff in tmux23:06
greghaynesand havent had that23:06
tchaypoever tried set -o errexit?23:06
greghaynesoh, its new in that rev23:06
greghaynesthis sounds like fun :)23:06
*** matty_dubs is now known as matty_dubs|gone23:09
tchaypoIt seems like it's this line23:14
tchaypoif [ $? != 0 ] ; then echo "Terminating..." >&2 ; exit 1 ; fi23:14
SpamapSwell yes, there's an error, so it is exiting with an error.23:14
*** cd-undercloud has joined #tripleo23:14
cd-undercloud************** overcloud complete status=1 ************23:14
*** cd-undercloud has quit IRC23:14
tchayponot even that.23:15
tchaypoIf I'm running with set -o errexit and I execute " '[' 0 '!=' 0 ']' ", tmux goes away23:15
tchaypooh, of course - because I'm running that in my shell and so my shell exits. ie, what you said.23:16
tchaypoyep, and that makes perfect sense then. If I was using ./devtest_variables.sh, errexit would be set in a subshell. Because we source it, it gets set in my top-level shell. Once there's an error inside the script, the script exits with an error, which then exits the top-level shell.23:18
SpamapStchaypo: yeah, sourcing scripts is the devil :)23:18
tchaypoSo in short - if anyone suggests set -o errexit for devtest, we'll give them an embarrassed look and tell them we can't.23:19
tchaypono, that's not right, we can.23:20
tchaypowe just can't set it in anything that gets sourced23:20
tchaypodevtest_setup.sh already does set -eux23:21
lifelessright23:24
lifelessthis is why I pushed hard to get away from sourcing so much23:24
lifelessvariables needs to be sourced, by definition23:24
lifelessnothing else, in princple23:24
*** CaptTofu has quit IRC23:24
lifelesswell, cloud creds etc23:24
lifelessnothing complex23:24
SpamapSlifeless: I'm on board with "no more variables" btw.23:27
SpamapSlifeless: give me a config file over a sourced thing any day :)23:27
tchaypo+123:27
tchaypoeven if it's a config file that gets sourced inside the script23:27
SpamapShah yeah23:27
tchayposorry, i missed the air-quotes around "config file"23:28
greghaynesA file with var=val is kinda like a config file ;)23:28
tchaypoWe have a turing-complete config language, yay!23:28
greghayneshaha23:28
greghaynesI did notice we chose mustache for the templating, I havent ever seen that outside of web shops23:30
*** mahipal has joined #tripleo23:30
lifelessgreghaynes: we're a web shop :P23:31
greghaynestouche23:31
openstackgerritJonathan Brownell proposed a change to openstack/diskimage-builder: Fix spelling error in "vm" element README.md  https://review.openstack.org/7700723:32
*** spzala_ has joined #tripleo23:37
*** sdake__ has joined #tripleo23:40
*** sseago__ has joined #tripleo23:41
*** SpamapS_ has joined #tripleo23:43
*** spzala has quit IRC23:45
*** sdake_ has quit IRC23:45
*** SpamapS has quit IRC23:45
*** sseago_ has quit IRC23:45
*** openstackgerrit has quit IRC23:45
*** jwalls has quit IRC23:45
*** jwalls has joined #tripleo23:45
*** jwalls has quit IRC23:49
*** cd-undercloud has joined #tripleo23:53
cd-undercloud************** overcloud complete status=1 ************23:53
*** cd-undercloud has quit IRC23:53
*** SpamapS_ is now known as SpamapS23:53
*** SpamapS has quit IRC23:53
*** SpamapS has joined #tripleo23:53
StevenKlifeless: Can I steal spot number #4?23:56
lifelesssure23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!