Thursday, 2014-07-17

lifelessand a tripleo experimental queue00:00
lifelessstevebaker: ^00:00
stevebakerok00:00
lifelesswhy?00:01
stevebakerlifeless: I've been away for a week and wondered if the world had changed00:03
lifelessnup00:03
*** tserong_ has joined #tripleo00:09
*** tserong_ has joined #tripleo00:09
*** xuhaiwei has joined #tripleo00:10
* tchaypo just put in contacts for the first time00:11
lifelesswelcome to ios ?00:11
tchaypoeyeos00:11
tchaypoit's weird being able to read my screen and yet not be able to see the edges of the frames00:12
*** tserong has quit IRC00:12
*** tserong_ is now known as tserong00:13
* tchaypo waits for overcloud to come up00:15
tchaypoah00:15
tchaypoI typed too soon00:15
lifelesstchaypo doesn't wait?00:15
tchaypoas soon as i tpyed that it failed00:16
tchaypohttp://paste.openstack.org/show/86830/00:16
lifelessshouldn't have typed that then00:16
tchaypobut still. progress is being made.00:16
lifelessthis is what I warned about00:16
lifeless:)00:16
lifelesswhen I said we had been depending on the presumption of a default gateway00:17
tchaypoyou warn about many things, it's hard to keep track of them all00:17
lifelessthe right fix is to change teh call site thats calling setup-neutron00:17
lifelessI think00:17
tchaypo*scrolls up*00:18
tchaypooh there it is. Change it to do what?00:18
lifelesspass in a gateway00:18
lifeless10.0.0.100:19
lifelessor whatever00:19
lifelessalternatively you could make the test for --no-gateway be that its a) on bm and b) not set00:19
lifelessrather than just b)00:19
tchaypoah00:20
tchaypoif we don't provide --no-gateway, wouldn't we have to provide a gateway?00:21
lifelessno00:21
tchaypoand if we have to provide a gateway, the only way I can see that we'd be safeish just making one up is if this netowkr is for purely internal purposes and should never have outbound traffic on it so will never need the gateway00:22
lifelessneutron has three behaviours00:22
lifelessa) explicit gateway00:22
lifelessb) explicit no-gateway00:22
lifelessc) default gateway00:22
tchaypoand we'd be relying on c)00:22
tchaypookay.00:22
lifelessthe new code path to utilise b) is kicking in in places where we relied on c)00:22
tchaypoto check if it's on bm, look at $PHYSICAL_NETWORK?00:22
lifelessyeah00:23
lifelessbut personally00:23
lifelessI think avoiding c entirely is better00:23
lifelesssubtle defaults like that break peoples brains00:23
tchaypookay. if we're going to set a gateway, how do we know what to set?00:24
lifelesscase by case question00:24
lifelesswhich case are we looking at right now?00:24
lifelessor perhaps I can say it differently.00:25
lifelessmake the gateway parameter required00:25
tchaypooh00:25
lifelesssee what breaks00:25
lifelessthats a place that needs to supply it. Change that place.00:25
StevenKtchaypo: For default-net on overcloud, the gateway is 10.0.0.100:25
tchaypoyep. don't make setup-neutron guess things, make setup-neutron require a gateway, or an explicit --no-gateway00:25
tchaypoStevenK: yep, so we can stick that in the thing that calls setup-neutron00:25
tchaypohard to have setup-neutron make a sensible guess though.00:26
lifelesstchaypo: I don't think it needs to guess00:26
StevenKWhich is all of devtest_{seed,undercloud,overcloud}.sh00:26
lifelesstchaypo: personally00:26
lifelesstchaypo: I would take this error as 'it breaking' and look at the call site of setup-neutron and fix it00:26
lifelesstchaypo: treat "" as == to --no-gateway (which the code now does in your tree)00:26
lifelesstchaypo: and treat incompatible situations as a place to fix00:27
lifelessI expect it to be only _overcloud that needs changing00:27
tchaypoall these pings are making my phone go wild00:27
tchaypoto check if it's on bm, look at $PHYSICAL_NETWORK?00:28
StevenKYes00:28
*** marun is now known as marun_afk00:28
lifelesstchaypo: I don't think you need to check if its on bm00:28
StevenKIf [ -z ... ] for non-BM00:28
* StevenK sets up a script that says "tchaypo: ping!" every 31.5 seconds00:28
tchaypobah00:28
tchayposorry00:28
tchaypoi was trying to scroll up in another window00:28
tchaypobut my up-arrows went here, followed by an enter, causing me to repeat a now irrelevant question i asked earlier00:29
*** rameshg87 has joined #tripleo00:30
tchaypoOVERCLOUD_IP=$(echo $OVERCLOUD_ENDPOINT | awk -F '[/:]' '{print $4}')00:34
tchaypoi'm almost certain that could be done without the echo00:34
rameshg87hello lifeless00:34
lifelesstchaypo: awk <<< $OVERCLOUD_ENDPOINT00:37
tchaypobut okay, while that run is in progress I'm going to go for a swim. If this build gets past that point I'll push the patch00:39
tchaypobut just in case greghaynes is looking for a quick patch00:41
tchaypohttp://paste.openstack.org/show/86835/00:41
*** jml has quit IRC00:45
*** jml has joined #tripleo00:49
*** yamahata has joined #tripleo00:54
lifelessrameshg87: hi :)00:56
rameshg87hello lifeless00:56
rameshg87lifeless: just had a question regarding your comment on https://review.openstack.org/#/c/97744/7/specs/juno/ironic-ilo-virtualmedia-driver.rst00:57
rameshg87lifeless: regarding the comment on two specs, i didn't get it completely00:57
rameshg87lifeless: do you mean we can have 2 specs - one for ironic changes (booting via virtual media) and other about diskimage-builder changes (using swift and partitioning locally) ?00:58
lifelesswell its all conceptually ironic changes01:01
lifelessbut yes two specs01:01
lifelessone about boot changes01:01
lifelessone about deploy changes01:01
rameshg87lifeless: so should both of them be under ironic-specs itself ?01:02
lifelessyes01:02
rameshg87lifeless: okay :-)01:02
rameshg87lifeless: and regarding the second comment on swap01:02
rameshg87lifeless: it was my mistake. that is not required. it was part of earlier spec01:03
rameshg87lifeless: the issue with lesser memory was addressed by handling uncompressed images01:03
greghaynespcrews: Re: your tripleo-HA job comment - I have absolutely no idea01:03
rameshg87lifeless: L301-305 https://review.openstack.org/#/c/97744/7/specs/juno/ironic-ilo-virtualmedia-driver.rst01:04
lifelessrameshg87: so it needs cleaning up regardless01:04
lifelessrameshg87: I'm still not at all convinced that http is a better transport than iscsi in this situation01:04
rameshg87lifeless: yes, i will remove that.01:04
lifelessrameshg87: so at a minium I'm going to urge that it be made optional01:04
*** Yanyanhu has joined #tripleo01:06
rameshg87lifeless: the proposed changes will be used only by ilo deploy driver in ironic initially (which is mostly supposed to be provisioning high memory systems[proliant gen8 systems])01:06
lifelessrameshg87: we deploy 6G images to 2G VMs in tripleo-gate hundreds of times a day01:06
lifelessrameshg87: this is not a hypothetical concern01:07
rameshg87lifeless: i agree.  :-)01:07
rameshg87lifeless: but i didn't get "optional". the change in deploy-ironic will be used by ilo driver only for now01:08
lifelessrameshg87: I don't understand why you want to force it to be just one thing01:08
rameshg87lifeless: do you mean it should be able to provision both via conductor node (via iscsi) and via baremetal node(locally) ?01:09
lifelessrameshg87: dd is extremely fast, its a C data plane; swift is a python data plane.01:09
lifelessrameshg87: I get that you have for whatever reason a desire to use swift, and thats fine - I just don't want it tied to the driver01:10
lifelessrameshg87: because I would quite like to be able to use a more secure boot driver01:10
lifelessrameshg87: but I really don't want to use a swift image transfer at this stage, not without a whole bunch of testing and perf analysis01:10
rameshg87lifeless: okay :-)01:11
lifelessrameshg87: for tripleo we have the exact same number of swift servers as ironic servers in the deploy cloud01:12
rameshg87lifeless: okay, so in your opinion, the ilo driver should be able to do deployments in both ways (conductor vs baremetal) and it should be a choice point for the user ?01:12
rameshg87lifeless: choice point for the admin infact01:12
lifelessrameshg87: I think the ilo driver is a replacement for pxe01:12
lifelessand there is an open question about whether the deploy engine in that should change at all, or if we should just go straight to IPA01:13
lifelesswhich also has swift etc etc in its design01:13
lifelessand/or multicast01:13
lifelesswhich is what I actually want as the next transfer method01:14
rameshg87lifeless: okay01:15
lifelessanyhow - by splitting the conversation about image transfer from the one about boot methods01:17
pcrewsgreghaynes: ack :)01:17
lifelesswe can unblock the boot methods and let it go forward01:17
lifelesswe also need to consider SANs too at some point01:17
rameshg87lifeless: i guess this was asked before and the thought point was extension to current deploy-ironic element was much simpler01:17
lifelesswhere boot method would be separate to connecting-the-image-up01:17
rameshg87lifeless: we weren't thinking about your multicast proposition then :-)01:17
lifelessrameshg87: seems like no change at all to the deploy-ironic element is *even simpler*01:17
rameshg87lifeless: hmm..okay01:18
lifelessrameshg87: it takes current Ironic 8 seconds to transfer a 6G image (we get 6Gbps on a mlx4n card)01:18
lifelessrameshg87: it takes the SL servers 5+ minutes to reboot01:19
lifelesscompute images with much smaller sizes are correspondingly faster01:19
lifelessthe bottlenecks today on deploy are:01:19
lifeless - machine reboot01:19
lifeless - the SATA disks in the machine01:19
rameshg87lifeless: okay01:20
lifelessI'm not saying we can't do swift01:21
lifelessjust that swift seems very unlikely to make anything better vs e.g. IPA and host standby servers01:21
lifelessI also want to look into doing iscsi or better still multicast with IPA, because IPA has the image size problem too01:22
rameshg87lifeless: yeah, it too will have some problem.01:23
rameshg87lifeless: but it acts on your bottleneck but putting servers on standby waiting for instructions01:23
rameshg87lifeless: and avoiding reboot. am i correct ?01:23
lifelessyes01:23
lifelessit trades off latency for power01:24
rameshg87lifeless: but i thought there were plans to put the server on low power mode wakable by lan or something.01:24
rameshg87lifeless: i don't know clearly :-)01:24
rameshg87lifeless: as you said i can split up the specs into two - an ironic deploy driver having virtual media boot and local deploy mechanism using swift01:25
lifelessyes01:26
rameshg87lifeless: but the deploy driver has to be refer to a deploy mechanism (like ironic pxe driver in ironic currently depends on iscsi)01:26
lifelessindeed01:28
lifelessso the second spec can talk about how we might make that more flexible01:28
*** tzumainn has quit IRC01:29
rameshg87lifeless: flexible as in a choose a deploy mechanism - like swift vs multicast ?01:30
lifelessright01:30
rameshg87lifeless: if i understand it correctly, the first spec shouldn't be concerned about deploy mechanism (it just knows that deploy-ironic will do it)01:31
lifelessright01:31
rameshg87lifeless: the second spec should talk about the currently proposed mechanism (swift) and its alternatives (multicast)01:31
rameshg87lifeless: am i correct ?01:31
lifelessI think the second spec should talk about how to allow deployers to choose between the current deployment engine (iscsi) and a new one (e.g. swift) without code duplication; and also - possibly same spec, possibly a third; the design of a swift mechanism01:33
rameshg87lifeless: okay, that means we will have to provide a config for the deployers by which they can configure the deployment engine01:35
rameshg87lifeless: with the same driver, the deployers should be able to  choose different deploy engines01:35
lifelesswell01:36
lifelessit might be just a wider set of drivers01:36
lifelesspxe_iscsi_ipmi01:36
lifelessilo_iscsi_ilo01:36
lifelessI don't know - thats the point of the spec01:36
rameshg87lifeless: okay. i think i have enough info now.01:39
rameshg87lifeless: will try to write the spec as you had suggested01:39
rameshg87lifeless: thanks a lot for your feedback :-)01:39
lifelessyw01:39
*** rameshg87 has left #tripleo01:41
* lifeless goesback to staring at booting serveres01:42
lifelessservers01:42
*** nosnos has joined #tripleo01:49
vinshsevering severe servers.01:51
*** saurabhs has quit IRC01:57
lifelessStevenK: where did you end up bunking for pyconau ?02:12
lifelesstchaypo: ^02:13
StevenKlifeless: Oaks Casino02:15
StevenK151 George St, Brisbane02:15
*** killer_prince is now known as lazy_prince02:18
tchaypoi would tell you all about how i just turned on my laptop and it couldn't find the wifi02:23
tchaypobut obviously it did in the end so the story is pointless02:23
tchaypolatest stack got stuck trying to build overcloud again02:23
* tchaypo pokes02:23
*** dshurtle has quit IRC02:33
openstackgerritpatrick-crews proposed a change to openstack-infra/tripleo-ci: Use /opt/git instead of /opt/stack/new  https://review.openstack.org/8997402:34
*** weshay has quit IRC02:41
*** ccrouch has joined #tripleo02:41
*** ramishra has joined #tripleo02:53
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134008703:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134077603:00
uvirtbotLaunchpad bug 1340087 in tripleo "PXE deploy failed with nova.virt.baremetal" [Undecided,New]03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/133998603:00
uvirtbotLaunchpad bug 1340776 in tripleo "Swift fails to start in overcloud" [Undecided,In progress]03:00
uvirtbotLaunchpad bug 1339986 in tripleo "when CONTROLSCALE>1 , services auth is still pointing at the local ip instead of virtual" [Undecided,In progress]03:00
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/134193603:00
*** untriaged-bot has quit IRC03:00
uvirtbotLaunchpad bug 1341936 in os-collect-config "Require keystone v3 auth_url in heat collector" [Undecided,New]03:00
*** Penick has joined #tripleo03:01
openstackgerritpatrick-crews proposed a change to openstack-infra/tripleo-ci: Use /opt/git instead of /opt/stack/new  https://review.openstack.org/8997403:04
*** Penick has quit IRC03:05
*** nosnos has quit IRC03:06
*** pcrews has quit IRC03:24
*** akuznetsov has joined #tripleo03:29
*** nosnos has joined #tripleo03:33
*** akuznetsov has quit IRC03:34
*** rakesh_hs has joined #tripleo03:37
*** Azher has joined #tripleo03:47
*** cwolferh_ has quit IRC03:54
lifelessok 43 machines up, finally04:00
* lifeless deletes them all04:00
*** cwolferh_ has joined #tripleo04:01
*** akuznetsov has joined #tripleo04:03
mordredwow04:04
mordredthat's a great thing to see when popping in to channel04:04
mordred04:00:15               * | lifeless deletes them all04:04
lifelessmordred: well its taken me all day to convince 43 machines to boot04:04
lifelessmordred: I found another how-did-this-ever-work bug this morning04:05
mordredlifeless: _excellent_04:05
mordredlifeless: I used to enjoy finding those during the early drizzle days04:06
lifelessregression in a nova refactoring of rescheduling04:06
*** rakesh_hs has quit IRC04:11
*** cwolferh_ has quit IRC04:13
*** lazy_prince is now known as killer_prince04:17
*** cwolferh_ has joined #tripleo04:18
*** bmahalakshmi has joined #tripleo04:19
*** ramishra has quit IRC04:26
*** nati_ueno has quit IRC04:26
*** ramishra has joined #tripleo04:26
*** ramishra has quit IRC04:30
*** ramishra_ has joined #tripleo04:30
*** pcrews has joined #tripleo04:30
StevenKGrr! http://paste.openstack.org/show/86864/04:32
*** noslzzp has quit IRC04:46
*** pcrews has quit IRC04:49
*** ramishra_ has quit IRC04:58
* tchaypo sets mode +train05:01
tchaypolifeless: is that a schrodingbug?05:02
tchaypoas in, as soon as you observe it, it's *defiitely* dead and you aren't sure how it ever worked?05:02
tchaypoi have a full stack build.05:02
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Add public VLAN support.  https://review.openstack.org/10375705:03
tchayponow to go away and work on my presentation a bit more..05:03
StevenKPort a0a199cb-dc0f-4f28-87ff-3a23924398b0 has owner network:router_interface and therefore cannot be deleted directly via the port API.05:04
StevenKRouter d3c587f8-8894-449b-951a-afb200e4bb3a still has ports05:04
StevenKSo I get to delete the port via magic?05:04
*** bashok has joined #tripleo05:07
StevenKI see. I have to router-interface-delete, but give the subnet id.05:08
*** ramishra has joined #tripleo05:08
*** killer_prince is now known as lazy_prince05:10
*** cody-somerville has quit IRC05:12
* tchaypo is so clever05:15
* tchaypo got on the wrong train05:15
tchaypoStevenK: well duh. wasn't that obvious?05:15
StevenKtchaypo: ... no? :-)05:17
*** loki184 has joined #tripleo05:19
tchaypo(Sarcasm obviously)05:21
StevenKI think my new setup-neutron works05:25
*** shausy has joined #tripleo05:25
StevenKBut it's missing lifeless' VLAN work05:25
tchaypoDo nothing important then05:27
tchaypoAt least the unit tests will give you confidence you have replicated the behaviour05:27
StevenK| 1a272143-1c59-4295-88de-526be5914558 | demo | ACTIVE | -          | Running     | default-net=10.0.0.2, 192.0.2.46 |05:29
StevenKThat gives me some confidence05:29
*** loki184 has quit IRC05:33
*** morganfainberg is now known as morganfainberg_Z05:33
*** dshulyak_ has joined #tripleo05:34
*** nati_ueno has joined #tripleo05:34
*** nati_ueno has quit IRC05:34
*** rakesh_hs has joined #tripleo05:41
*** chuckC has joined #tripleo05:43
*** loki184 has joined #tripleo05:57
*** jcoufal has joined #tripleo05:57
*** mrunge has joined #tripleo05:59
*** andreaf has quit IRC06:05
lifelessStevenK: so whats your setup-neutron change?06:05
*** andreaf has joined #tripleo06:06
StevenKlifeless: A new script in os-cloud-config06:07
lifelessStevenK: cool06:08
xuhaiweilifeless: hi, about the heat stack-create problem i asked you yesterday, I login the baremetal machine, and saw the /var/log/upstart/os-collect-config.log, there are a lot of warning like 'os_collect_config [-] No metadata_url configured06:13
xuhaiweido the warnings matter?06:13
xuhaiweithere are no errors in the log06:14
*** ramishra has quit IRC06:16
*** ramishra has joined #tripleo06:16
lifelessthose warnings are fine06:18
*** akuznetsov has quit IRC06:18
lifelessyou should be able to look back and see if it actually posted a success callback to heat06:18
*** akuznetsov has joined #tripleo06:20
*** k4n0 has joined #tripleo06:22
*** andreaf has quit IRC06:23
*** andreaf has joined #tripleo06:24
*** sballe_ has joined #tripleo06:28
*** pcrews has joined #tripleo06:28
*** michchap_ has joined #tripleo06:29
*** ramishra has quit IRC06:33
*** pmyers_ has joined #tripleo06:33
*** pleia2_ has joined #tripleo06:33
*** lynxman_ has joined #tripleo06:33
*** pmyers has quit IRC06:34
*** sballe has quit IRC06:34
*** jtomasek has quit IRC06:34
*** d0ugal has quit IRC06:34
*** jrist has quit IRC06:34
*** panda has quit IRC06:34
*** lynxman has quit IRC06:34
*** michchap has quit IRC06:34
*** pleia2 has quit IRC06:34
*** k4n0 has quit IRC06:34
*** robsparker has quit IRC06:34
*** sseago has quit IRC06:34
*** rbrady has quit IRC06:34
*** panda___ has joined #tripleo06:34
*** lynxman_ is now known as lynxman06:34
*** jtomasek has joined #tripleo06:34
*** lynxman has quit IRC06:34
*** lynxman has joined #tripleo06:34
*** shardy_afk is now known as shardy06:34
*** sseago has joined #tripleo06:35
*** robsparker has joined #tripleo06:35
*** jrist has joined #tripleo06:35
*** rbrady has joined #tripleo06:35
*** k4n0 has joined #tripleo06:35
*** d0ugal has joined #tripleo06:36
*** ramishra has joined #tripleo06:36
*** bashok has quit IRC06:36
xuhaiweilifeless: sorry, can you give a hint, how to check if there is a success callback to heat?06:38
*** ramishra has quit IRC06:38
lifelessxuhaiwei: have a look in the log for curl06:41
xuhaiweiin /var/log/upstart ?06:42
lifelessos-collect-config.log, yes06:42
xuhaiweilifeless: in os-collect-config.log, I also saw [Errno 111] Connection refused06:44
xuhaiweilifeless: does this mean it can't send a callback to heat?06:45
*** ramishra has joined #tripleo06:45
*** pcrews has quit IRC06:49
lifelessxuhaiwei: possibly, need to see more info06:51
xuhaiweilifeless: thank you06:52
kviiriHow would you recommend tweaking the ntp of the images created with disk-image-builder?06:55
*** ifarkas has joined #tripleo06:59
*** ramishra has quit IRC06:59
*** ramishra has joined #tripleo07:00
*** andreaf has quit IRC07:02
lifelesskviiri: ntp? its metadata driven, no?07:03
*** regebro has joined #tripleo07:05
*** rdopieralski has joined #tripleo07:17
kviiriDunno :)07:18
*** jistr has joined #tripleo07:18
*** jang2 has joined #tripleo07:20
*** bashok has joined #tripleo07:24
*** ramishra has quit IRC07:26
lifelesskviiri: see the ntp element07:37
*** dshulyak_ has quit IRC07:41
openstackgerritEamonn O'Toole proposed a change to openstack/tripleo-image-elements: Set venv ownership for services  https://review.openstack.org/10638007:45
*** jp_at_hp has joined #tripleo07:51
*** ramishra has joined #tripleo07:54
*** andreaf has joined #tripleo07:55
*** andreaf has quit IRC08:01
*** pleia2_ is now known as pleia208:04
*** derekh_ has joined #tripleo08:08
*** rwsu has quit IRC08:13
*** andreaf has joined #tripleo08:17
fabbioneCommand output matched '(CREATE|UPDATE)_FAILED'. Exiting...08:21
fabbionenot very promising08:21
*** ramishra has quit IRC08:22
*** lucasagomes has joined #tripleo08:23
*** bmahalakshmi has quit IRC08:24
*** bmahalakshmi has joined #tripleo08:30
lifelessfabbione: means one of the physical node deploys failed08:32
lifelessfabbione: heat is quite sensitive to API fails at the moment, its being worked on08:32
openstackgerritA change was merged to openstack/tuskar-ui: Updating Baremetal Node UUIDs with realistic values for display  https://review.openstack.org/10747408:32
*** pelix has joined #tripleo08:38
*** martyntaylor has joined #tripleo08:40
*** athomas has joined #tripleo08:42
fabbionelifeless: feh....08:44
*** cody-somerville has joined #tripleo08:44
fabbionelifeless: can i just re-run overcloud deployment or do I need to start from scratch? (devtest)08:45
*** ramishra has joined #tripleo08:47
derekh_Does anybody else see bug 1343162 locally ?08:55
uvirtbotLaunchpad bug 1343162 in tripleo "Failed to fetch bzip2:... Hash Sum mismatch" [High,Triaged] https://launchpad.net/bugs/134316208:55
derekh_I'm wondering if it may be a problem with the proxy having a stale cache08:56
*** dtantsur|afk is now known as dtantsur08:59
fabbioneclearly no.. needs manual recovery..08:59
*** untriaged-bot has joined #tripleo09:00
untriaged-botUntriaged bugs so far:09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134008709:00
uvirtbotLaunchpad bug 1340087 in tripleo "PXE deploy failed with nova.virt.baremetal" [Undecided,New]09:00
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/134193609:00
uvirtbotLaunchpad bug 1341936 in os-collect-config "Require keystone v3 auth_url in heat collector" [Undecided,New]09:00
*** untriaged-bot has quit IRC09:00
*** ramishra has quit IRC09:04
lifelessfabbione: you can rerun the overcloud09:04
lifelessfabbione: long as you've got the variables sourced09:04
fabbionelifeless: nah i had to destroy the old overcloud first, now it's re-running09:05
fabbionebut yeah. i have all sourced09:05
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Allow capabilites to be registered to ironic nodes  https://review.openstack.org/10730809:13
*** mkerrin has quit IRC09:16
*** mkerrin has joined #tripleo09:23
*** yamahata has quit IRC09:25
*** markmc has joined #tripleo09:28
*** martyntaylor has left #tripleo09:28
*** ramishra has joined #tripleo09:29
openstackgerritYanyan Hu proposed a change to openstack/os-collect-config: Using keystone discover to find V3 auth_url  https://review.openstack.org/10761109:30
openstackgerritEndre Karlson proposed a change to openstack/diskimage-builder: Cleanup apt cache after grub install  https://review.openstack.org/10712809:37
*** akrivoka has joined #tripleo09:46
*** Yanyanhu has quit IRC09:46
*** pmyers_ is now known as pmyers09:51
*** nosnos has quit IRC09:54
*** nosnos has joined #tripleo10:01
*** cody-somerville has quit IRC10:08
*** loki184 has quit IRC10:08
*** rlandy has joined #tripleo10:10
*** nosnos has quit IRC10:10
*** nosnos has joined #tripleo10:12
*** martyntaylor1 has joined #tripleo10:19
openstackgerritsandhya proposed a change to openstack/diskimage-builder: Fix for host env leak into grub  https://review.openstack.org/10763610:25
openstackgerritsandhya proposed a change to openstack/diskimage-builder: Fix for host env leak into grub  https://review.openstack.org/10763610:27
*** derekh_ has quit IRC10:31
*** andreaf has quit IRC10:34
*** andreaf has joined #tripleo10:35
*** lazy_prince is now known as killer_prince10:38
*** nosnos has quit IRC10:38
*** derekh_ has joined #tripleo10:40
*** killer_prince is now known as lazy_prince10:41
*** dguerri is now known as _dguerri10:44
*** _dguerri is now known as dguerri10:46
*** Azher has quit IRC10:48
*** martyntaylor1 has quit IRC10:50
*** martyntaylor has joined #tripleo10:51
openstackgerritAna Krivokapic proposed a change to openstack/tuskar-ui: Ceilometer graphs for nodes  https://review.openstack.org/10418811:05
*** dguerri is now known as _dguerri11:05
*** lucasagomes is now known as lucas-hungry11:07
*** Yanyanhu has joined #tripleo11:08
lifelesstchaypo: so hi11:08
lifelesstchaypo: did you confirm your tweak works ?11:09
openstackgerritYanyan Hu proposed a change to openstack/os-collect-config: Using keystone discover to find V3 auth_url  https://review.openstack.org/10761111:10
*** cody-somerville has joined #tripleo11:13
*** julim has joined #tripleo11:15
lifelessgreghaynes: there's a stack  you might want to look at - 3 node HA + vlan + public VIP, more than somewhat broken :P - on hp111:15
lifelessVIPs came up, yay11:17
*** martyntaylor has left #tripleo11:19
*** ramishra has quit IRC11:20
*** _dguerri is now known as dguerri11:20
*** ramishra has joined #tripleo11:20
derekh_lifeless: So I think the ci-overcloud problem I was talking about at the meeting are because either dbus or openvswitch is locked up, going to reboot it later (systemctl restart times out), but it holds the geard broker and the ci proxy so it will probably disrupt running jobs11:21
*** jcoufal has quit IRC11:22
*** ramishra has quit IRC11:25
lifelessderekh_: ack11:26
lifelessderekh_: the ctlplane?11:26
*** ramishra has joined #tripleo11:26
*** robsparker has quit IRC11:28
lifelesshmm11:28
lifeless| controller1AllNodesConfig | 3a003079-3942-4035-9844-c9938d84513a | OS::Heat::StructuredDeployment | CREATE_IN_PROGRESS | 2014-07-17T11:14:23Z |11:28
lifelessbut 0 is done, wonder whats up11:28
lifelesshmm11:30
lifeless[WARNING] 197/112858 (28915) : config : log format ignored for proxy 'horizon' since it has no log address.11:30
*** xuhaiwei has quit IRC11:31
lifelessgreghaynes: mysql wasn't coming up - suspec tthe same on both ctl1 and ctl2; haven't looked closely11:33
lifelessalso wow we shouldn't be running11:34
lifelessdib-run-parts Thu Jul 17 11:33:52 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/69-ceilometer-api11:34
lifeless+ ceilometer-dbsync --config-file /etc/ceilometer/ceilometer.conf11:34
lifelesson non-bootstrap-host today :)11:34
*** lblanchard has joined #tripleo11:34
lifelesslooks like it couldn't talk to the master so gave up11:41
lifelessfrom error.log11:41
lifelessgreghaynes: TheJulia: ^ fodder for thought.11:42
lifelessgnight11:42
*** robsparker has joined #tripleo11:43
*** akrivoka has quit IRC11:44
*** akrivoka has joined #tripleo11:45
*** lazy_prince has quit IRC11:47
openstackgerritJulia Kreger proposed a change to openstack/tripleo-image-elements: Prevent multiple MySQL masters  https://review.openstack.org/10441411:50
TheJuliagood morning11:52
*** cody-somerville has quit IRC11:54
TheJulia\o/ for more than somewhat broken11:54
*** tzumainn has joined #tripleo11:56
rdopieralskitzumainn: what did you mean?11:56
tzumainnrdopieralski, hm - you don't get an error when submitting that form?11:57
TheJulialifeless: mysql giving up is actually due to /etc/init.d/mysql not returning 1 upon a failed restart.  We've got two bugs that relate to that as well as a bug open with Percona.  I've got a patchset that fixes it, but Percona's next release for the series we're on should have the fix included.11:58
fabbionelifeless: i keep getting that error in loop.. where I can find logs and do a proper report?11:58
rdopieralskitzumainn: the tests would fail then11:58
fabbionelifeless: or you have an automated error reporting tool?11:58
tzumainnrdopieralski, give me a sec11:59
*** jistr|mobi has joined #tripleo11:59
* rdopieralski hands tzumainn a large, shiny sec12:01
tzumainnrdopieralski, hm, you're right, I'm suddenly confused again12:01
tzumainnlet me test it once more, sec12:01
*** ci-overcloud has joined #tripleo12:05
ci-overcloud************** ci-overcloud complete status=1 ************12:05
*** ci-overcloud has quit IRC12:05
*** ramishra has quit IRC12:06
*** ramishra has joined #tripleo12:06
*** shausy has quit IRC12:07
*** ramishra has quit IRC12:09
*** morazi has joined #tripleo12:09
*** ramishra has joined #tripleo12:12
TheJuliafabbione: Do you mean the issue where a mysql server gave up upon start-up?12:12
fabbioneTheJulia: Command output matched '(CREATE|UPDATE)_FAILED'. Exiting...12:13
fabbionedeploying overcloud from devtest12:13
*** jdob has joined #tripleo12:14
TheJuliafabbione: first step would be to look at the undercloud's stack event list12:14
TheJuliasource $TRIPLEO_ROOT/tripleo-incubator/understackrc ; heat event-list overcloud12:15
TheJuliaThat should give a basic idea of what happened, from there you'll need to login to your controller(s) and look at the logs.12:16
tchaypolifeless: I confirmed that devtest.sh runs to completion12:16
fabbione| controller0             | a84fe469-a975-40bb-bde4-f8b9855b6acd | Error: Creation of server overcloud-controller0-th2pd7f4rgpr failed. | CREATE_FAILED      | 2014-07-17T11:20:20Z |12:17
fabbione| NovaCompute0            | 09642e02-ab31-40f8-b86e-ee3e12924031 | CREATE aborted                                                       | CREATE_FAILED      | 2014-07-17T11:20:20Z |12:17
fabbioneTheJulia: ack thanks12:17
tzumainnrdopieralski, so, I still get the error - infrastructure/nodes/forms.py  passes in variables such as 'cpus' and 'memory', but I think api/node.py expects 'cpu' and 'ram' ?12:18
*** ramishra has quit IRC12:19
TheJuliafabbione: so you'll need to login to your undercloud and look at the logs there.  I think the heat-api log is your first stop once inside.12:19
fabbioneTheJulia: yeps thanks. first time running TO.. need to find my way around :)12:20
*** ramishra has joined #tripleo12:20
rdopieralskihuh?12:21
rdopieralskithat's clearly my mistake12:21
rdopieralskibut why didn't the test fail?12:22
TheJuliaAnyone willing to review, one thing that might HA mode happier in the long run: https://review.openstack.org/#/c/105959/12:23
tzumainnrdopieralski, hm, I'm not sure12:24
TheJuliafabbione: I know that feeling, I'm relatively new myself.  Feel free to ping me if you have any questions :)12:24
*** cody-somerville has joined #tripleo12:25
*** cody-somerville has joined #tripleo12:25
openstackgerritTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Add Discovered nodes tab  https://review.openstack.org/10750912:25
rdopieralskiI don't like having tests that never fail :/12:26
tzumainnrdopieralski, lol, I don't blame you, it's rather confusing12:27
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Allow user-defined flavors on the undercloud  https://review.openstack.org/10766712:28
*** Yanyanhu has quit IRC12:29
*** ramishra has quit IRC12:30
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Allow user-defined flavors on the undercloud  https://review.openstack.org/10766712:32
*** andreaf has quit IRC12:36
*** dprince has joined #tripleo12:37
*** Yanyanhu has joined #tripleo12:37
*** andreaf has joined #tripleo12:37
openstackgerritLucas Alvares Gomes proposed a change to openstack/diskimage-builder: Ironic auto-discovery ramdisk  https://review.openstack.org/10767212:37
openstackgerritLucas Alvares Gomes proposed a change to openstack/diskimage-builder: Ironic auto-discovery ramdisk  https://review.openstack.org/10767212:39
*** lucas-hungry is now known as lucasagomes12:39
*** weshay has joined #tripleo12:40
*** jcoufal has joined #tripleo12:44
*** noslzzp has joined #tripleo12:46
*** dguerri is now known as _dguerri13:00
derekh_dprince: can you add my key to the squid server please https://launchpad.net/~derekh/+sshkeys13:00
derekh_dprince: I want to see if whats its cahing might be causeing bug 134316213:00
uvirtbotLaunchpad bug 1343162 in tripleo "Failed to fetch bzip2:... Hash Sum mismatch" [High,Triaged] https://launchpad.net/bugs/134316213:01
*** k4n0 has quit IRC13:01
TheJuliaderekh_: I saw that locally yesterday, I deleted the files and whatever got served back up through my local proxy was right13:03
dprincederekh_: yep, one minute13:04
derekh_TheJulia: ok, so maybe deleting them in diskimage-builder might be the correct solution13:04
derekh_dprince: thanks13:04
TheJuliathat doesn't seem like a solution, more a work-around, I was suspecting stale data myself yesterday13:05
openstackgerritDougal Matthews proposed a change to openstack/tripleo-specs: Updated the reference to the Tuskar REST API spec  https://review.openstack.org/10768313:06
openstackgerritDougal Matthews proposed a change to openstack/tripleo-specs: Update the Tuskar storage spec to reference Heat  https://review.openstack.org/10768413:06
*** _dguerri is now known as dguerri13:06
derekh_TheJulia: ya, it doesn't seem right, will poke at it a bit and see if I discover anything13:07
*** athomas has quit IRC13:07
*** bmahalakshmi has quit IRC13:07
TheJuliaCool, I thought it was a fluke, but I'm seeing it in CI jobs that are failing :(13:09
TheJulias/seeing/now seeing/13:09
derekh_TheJulia: ya, its causing about 25 jobs per day to fail13:10
TheJulia:(13:10
*** bashok has quit IRC13:11
*** athomas has joined #tripleo13:11
*** jistr|mobi has quit IRC13:16
derekh_hmmm, a mirror not updated ?13:19
derekh_suppose a link would be handy http://paste.openstack.org/show/86934/13:20
TheJuliaoh awesome13:25
dprincederekh_: your key should be on the squid now13:26
derekh_dprince: thanks13:26
dprincederekh_: you might should add mine to the te-broker when you get a chance (no rush)13:27
derekh_dprince: will do13:27
derekh_dprince: done13:30
derekh_So the offending server appears to be, nova.clouds.archive.ubuntu.com (91.189.91.13)13:31
derekh_anybody know the place to report these things?13:31
derekh_launchpad ?13:31
dprincederekh_: yep, I've seen that in the logs before13:32
dprincederekh_: is it possible we could pin the Ubuntu tests to a specific mirror?13:32
dprincederekh_: outside of the rack...13:32
derekh_dprince: maybe a temp CI fix ?    $ echo 91.189.91.14 nova.clouds.archive.ubuntu.com >> /etc/hosts13:34
dprincederekh_: if need be, we can do that13:36
TheJuliacheckout https://jenkins04.openstack.org/job/check-tripleo-ironic-undercloud-precise/971/console security.ubuntu.com13:38
TheJuliacould be bad timing if its a new update being pushed... but still little weird13:40
derekh_TheJulia: looks like security.ubuntu.com also resolves to 91.189.91.13 (1 of its 6 IP's), so looks like that server hasn't updated for some reason13:42
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Add a "driver" field to the Register Nodes dialog  https://review.openstack.org/10705213:47
TheJuliayay13:48
funzoderekh_: ping13:51
derekh_funzo: ack13:53
funzoderekh_: do you know if there is a way to run the tempest tests in openstack-infra with a custom tempest.conf?13:53
*** hewbrocca has joined #tripleo13:54
funzoderekh_: I'm going to configure the nova-docker repo to run the tempest tests which will run the stock config to begin with, but in order to have it make more sense I need to turn off a few services in the tempest.conf13:54
derekh_funzo: not that I've ever done, but looking at the devstack gate code would show it if its possible, I can take a look in a few minutes and see if its possible13:55
funzoderekh_: wow, thanks very much13:55
derekh_dprince: TheJulia ok I'm going to add this to /etc/hosts on the proxy and see if it helps13:55
derekh_91.189.91.14 archive.ubuntu.com13:56
derekh_91.189.91.14 nova.clouds.archive.ubuntu.com13:56
derekh_91.189.91.14 security.ubuntu.com13:56
dprincederekh_: fair enough, we'll never remember to do this again on the other rack though13:57
dprincederekh_: unless you do13:57
derekh_dprince: I'll set a reminder to check the server in a few days and see if the problem is resolved, hopfully we shouldn't need it there for long (if it even works that is)13:58
derekh_hmm, does squid have a graceful restart...13:58
dprincederekh_: not sure, if not just do a quick bounce13:59
derekh_done, lets see if that helps http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSGFzaCBTdW0gbWlzbWF0Y2hcIiBBTkQgYnVpbGRfbmFtZToqdHJpcGxlbyoiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwNTYwNTYxNDA1OH0=14:00
*** dguerri is now known as _dguerri14:05
*** _dguerri is now known as dguerri14:07
*** lazy_prince has joined #tripleo14:19
*** lazy_prince has joined #tripleo14:20
*** jdob has quit IRC14:31
*** jcoufal has quit IRC14:32
*** cinerama has joined #tripleo14:33
*** Yanyanhu has quit IRC14:33
*** marun_afk is now known as marun14:34
*** morganfainberg_Z is now known as morganfainberg14:35
*** robsparker has quit IRC14:35
openstackgerritA change was merged to openstack/tripleo-incubator: Improve readability of long JQ expression  https://review.openstack.org/9968114:35
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Separate Heat BM and VM configs for Nova-BM.  https://review.openstack.org/9003314:37
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of JQ for novaBM and ironic  https://review.openstack.org/10455914:37
*** mrunge has quit IRC14:37
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Wait for os-collect-config to complete on the seed  https://review.openstack.org/9651514:37
*** jdob has joined #tripleo14:38
*** morganfainberg is now known as morganfainberg_Z14:40
greghayneslifeless: that mysql fail is... odd14:40
greghayneslifeless: seems like controller2 connected to the master fine but not controller 114:41
*** morganfainberg_Z is now known as morganfainberg14:41
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Allow user-defined flavors on the undercloud  https://review.openstack.org/10766714:42
greghaynesugh14:42
greghaynesI think we might have a bug where were no longer restarting mysql after a failed start14:43
dprincegreghaynes: we do, in fact we no longer restart MySQL on a reboot14:43
* dprince ranted about this in the meeting Tuesday14:44
TheJuliawe do, but only because we're not able to detect the failed start14:44
greghaynesyes, thats somewhat intentional14:44
greghaynesbut I mean after a failed start ever14:44
greghaynesdprince: we now restart mysql if you have controlscale of 114:44
greghaynesor at least we should14:44
*** cinerama has quit IRC14:44
dprincegreghaynes: I don't think we do14:44
dprincegreghaynes: my take is with HA disabled (on a single undercloud for example or seed VM) mysql should start automatically14:45
dprincegreghaynes: after it has been initially configured that is14:45
dprincegreghaynes: anything else is unacceptable14:45
greghaynesYes, I had a patch for that, maybe its not merged yet...14:46
dprincegreghaynes: can you point me to it?14:46
*** pcrews has joined #tripleo14:46
*** ddieterly has joined #tripleo14:48
greghaynesoh, it merged: Id41c2fcf9602828b60692d866ee33737e33f87aa14:49
greghaynesMy internet is less than ideal ATM so links are hard14:50
dprincegreghaynes: https://review.openstack.org/#/c/104455/14:50
dprincegreghaynes: so that actually won't quite do it because on a reboot os-collect-config won't necissarily run right?14:51
greghaynesaye14:51
greghaynescorrect14:51
dprincegreghaynes: what we really need is an os-svc-enable there too14:51
openstackgerritA change was merged to openstack/tuskar-ui: Add a "driver" field to the Register Nodes dialog  https://review.openstack.org/10705214:52
dprincegreghaynes: glad we are talking about this cause I was about to have a look today anyways...14:52
dprincegreghaynes: has anyone pushed a fix for that yet, if not I'm up for giving it a go...14:52
greghaynesnot AFAIK, im +1 on the svc-enable14:53
greghaynesmaybe only do it for the single node setupalso14:53
lxsliplease could we get +2 on https://review.openstack.org/#/c/104559/ ? It has clean tests14:53
lxsliI'd +2 it myself but I don't have the button yet14:53
openstackgerritpatrick-crews proposed a change to openstack-infra/tripleo-ci: Use /opt/git instead of /opt/stack/new  https://review.openstack.org/8997414:53
*** ddieterl_ has joined #tripleo14:54
dprincegreghaynes: sure. For HA setups though what do we rely on to ensure this configure.d re-runs?14:54
greghayneswe dont14:54
greghaynesIts an unsolved problem for multiple node setups14:55
dprincegreghaynes: gotcha, just making sure I wasn't missing something. So... we could solve all of this (both problems) with our own systemd script for this14:55
dprincegreghaynes: which could used cached heat metadata if it needs too...14:56
greghaynesThe multi-node setup case requires us to re-perform leader election14:56
greghayneswhich is why we are intentionally not restarting in that case14:56
dprincegreghaynes: okay14:56
* dprince doesn't have his head wrapped around HA this morning14:56
dprincegreghaynes: so for now an inline os-svc-enable for the single node case doesn't seem too bad then. Lets do that...14:57
greghaynes+114:57
*** jp_at_hp has quit IRC15:01
*** morganfainberg is now known as morganfainberg_Z15:02
*** ddieterly has quit IRC15:02
*** akrivoka has quit IRC15:02
*** andreaf has quit IRC15:02
*** derekh_ has quit IRC15:02
*** rlandy has quit IRC15:02
*** ifarkas has quit IRC15:02
*** rbrady has quit IRC15:02
*** jtomasek has quit IRC15:02
openstackgerritTomas Sedovic proposed a change to openstack/tripleo-heat-templates: Deprecate Role and ImageBuilder metadata handling  https://review.openstack.org/10252115:03
*** andreaf has joined #tripleo15:03
rdopieralskitzumainn: I can't see newly_discovered on that api link that you posted?15:04
tzumainnrdopieralski, it doesn't exist yet, it's part of dtantsur's spec15:04
tzumainnhe mentioned that he believed that part was relatively uncontroversial15:04
rdopieralskitzumainn: then why did you link to it?15:04
tzumainnoh, for the 'maintenance' attribute15:05
rdopieralskiType:boolean15:05
rdopieralskimaintenance (boolean) – Optional boolean value that indicates whether to get nodes in maintenance mode (“True”), or not in maintenance mode (“False”).15:05
*** rbrady has joined #tripleo15:05
rdopieralskiit doesn't mention None anywhere?15:06
tzumainnif you look at the returned json15:06
rdopieralskithe returned json has that undefined everywhere15:06
tzumainnI wish I could use line number, search for "maintenance" with the quotes15:06
rdopieralskithat is, not set15:06
*** jp_at_hp has joined #tripleo15:07
*** akrivoka has joined #tripleo15:07
*** jtomasek has joined #tripleo15:07
*** derekh_ has joined #tripleo15:07
tzumainnoh, is that what that means?  I might have been confused15:07
*** ifarkas has joined #tripleo15:07
*** rlandy has joined #tripleo15:07
*** ifarkas has quit IRC15:07
*** ifarkas has joined #tripleo15:07
rdopieralskiI guess that's what the "optional value" means15:07
*** ddieterl_ has quit IRC15:07
tzumainnI figured optional value meant that it was an optional argument into the list function15:08
*** ddieterly has joined #tripleo15:08
rdopieralskibut I don't like this "True", "False", "File Not Found" three-state logic :/15:08
tzumainnbut the json snippets made it seem like the returned json from the rest api would be null15:08
rdopieralskithey probably made it optional by defaulting to None15:08
tzumainnso, yeah, I'm just following what I thought I saw in the ironic docs15:09
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: mysql: call os-svc-enable for single node installs  https://review.openstack.org/10773715:09
tzumainndid I misunderstand them?15:09
rdopieralskiwell, 'if not foo' is going to catch both None and False, so it should be fine15:10
greghayneslifeless: oh, I bet that mysql issue is fixed by TheJulia's patch to make percona init script return error on startup fail15:14
*** mestery has quit IRC15:16
greghaynesdprince: FWIW theres a ML thread about re-running o-r-c on boot15:17
greghaynesIf you havent seen15:18
*** andreaf has quit IRC15:23
*** morganfainberg_Z is now known as morganfainberg15:24
openstackgerritGerry Drudy proposed a change to openstack/tripleo-image-elements: Add proxy-logging to proxy-server pipeline  https://review.openstack.org/9153315:24
*** ramishra has joined #tripleo15:25
*** rakesh_hs has quit IRC15:28
*** akuznetsov has quit IRC15:30
openstackgerritpatrick-crews proposed a change to openstack-infra/tripleo-ci: Use /opt/git instead of /opt/stack/new  https://review.openstack.org/8997415:33
*** ramishra has quit IRC15:34
*** akuznetsov has joined #tripleo15:35
*** jcoufal has joined #tripleo15:38
*** cinerama has joined #tripleo15:40
*** rdopieralski has quit IRC15:40
*** ifarkas has quit IRC15:47
openstackgerritJon-Paul Sullivan (jp_at_hp) proposed a change to openstack/tripleo-incubator: Provide useful error early if not enough nodes  https://review.openstack.org/9359715:50
*** akrivoka has quit IRC15:52
openstackgerritJames Slagle proposed a change to openstack/tripleo-heat-templates: Remove extra list  https://review.openstack.org/10775915:54
openstackgerritJames Slagle proposed a change to openstack/tripleo-heat-templates: Add user_data_format  https://review.openstack.org/10776015:54
openstackgerritJames Slagle proposed a change to openstack/tripleo-heat-templates: Add signal_transport property  https://review.openstack.org/10776115:54
cineramamorning guys15:56
*** cody-somerville has quit IRC15:57
*** lazy_prince is now known as killer_prince15:57
*** jcoufal has quit IRC15:59
*** jistr has quit IRC16:01
TheJuliaGood morning, although it is now afternoon here :)16:01
cineramatime zones are hard16:04
*** jcoufal has joined #tripleo16:08
*** IvanBerezovskiy has left #tripleo16:09
*** markmc has quit IRC16:15
greghaynescinerama: 'morning16:21
cineramahi there!16:22
*** johnma has joined #tripleo16:23
TheJuliacinerama: If only we could convince the world to go to using one time zone :)16:24
*** mkerrin has quit IRC16:24
*** lucasagomes has quit IRC16:30
*** ddieterly has quit IRC16:36
*** ddieterly has joined #tripleo16:36
greghaynescan I get someone to review https://review.openstack.org/#/c/104569/16:37
greghaynesfixes a prettybad bug for HA setups16:37
TheJuliaAdditionally, https://review.openstack.org/#/c/105959/ which fixes the VRRP configuration in HA16:38
*** rameshg87 has joined #tripleo16:43
openstackgerritClint 'SpamapS' Byrum proposed a change to openstack/tripleo-image-elements: Do not let Upstart start services automatically  https://review.openstack.org/10777216:43
*** sseago has quit IRC16:43
rameshg87hello derekh_16:44
rameshg87derekh_: request your review on https://review.openstack.org/#/c/97301/16:44
greghaynesdprince: ^ that t-i-e change may be of interest to you16:44
greghaynesfrom SpamapS16:44
derekh_rameshg87: sorry, I got sidetracked and am about to run but will look again ASAP16:45
rameshg87thanks derekh_  :-)16:45
*** derekh_ has quit IRC16:49
openstackgerritA change was merged to openstack/tripleo-image-elements: Add new sosreport element  https://review.openstack.org/10459316:51
*** rwsu has joined #tripleo16:55
SpamapSgreghaynes: is there a similar effort underway that I'm not aware of?16:59
SpamapSgreghaynes: I did it in response to the ML post about starting services before o-c-c has run.16:59
*** sseago has joined #tripleo16:59
greghaynesSpamapS: Not a similar effort, dprince was hoping to fix starting mysql on single controller node setups by adding it to start on init17:00
greghayneser, starting after reboot17:00
greghaynesWas there a resolution for diong an o-r-c run on reboot? I saw it mentioned in the email but the spec that was linked didnt seem to mention it at all :/17:02
*** dshulyak_ has joined #tripleo17:07
*** akrivoka has joined #tripleo17:17
*** regebro has quit IRC17:23
greghaynesIs anyone familiar with how were using the tripleo-end-to-end reporefs deal in hp ci?17:30
*** chuckC has quit IRC17:30
greghaynesTrying to add a patch to the rollup for one of the repos17:30
*** ramishra has joined #tripleo17:32
*** cadenzajon has quit IRC17:35
*** ramishra has quit IRC17:35
SpamapSgreghaynes: yes, there's a patch up to have o-c-c run the command once per boot guaranteed17:39
greghaynesdprince: ^17:40
greghaynesdprince: with that mysql should 'just work' on reboot for 1 node setups17:40
greghaynes\O/ conflicts with the patches we need to run in CI17:41
dprincegreghaynes: let me think about it, we might should do both for this one...17:41
greghaynesFor mysql long-term using os-r-c is really the only viable option17:42
greghaynesdue to the leader election deal17:42
greghaynesI guess the alternative would be trying to write a highly intelligent init/upstart/systemd script17:42
dprincegreghaynes: that is for HA, if you aren't using HA then autostarting MySQL would seem to be fine17:42
greghaynesYes, so IMO if we can get away with just doing it one way then we should17:43
*** ramishra has joined #tripleo17:44
* greghaynes is afraid of what he just did to the repos in hp ci region17:44
greghaynesthis is why we cant have nice things17:44
*** pelix has quit IRC17:45
*** markmc has joined #tripleo17:46
*** regebro has joined #tripleo17:51
*** ramishra has quit IRC17:52
*** jcoufal has quit IRC17:58
*** jp_at_hp has quit IRC18:04
*** spzala has joined #tripleo18:04
*** chuckC has joined #tripleo18:10
*** dtantsur is now known as dtantsur|afk18:11
vinsh:)18:14
openstackgerritA change was merged to openstack/tripleo-heat-templates: add Horizon caches backend definition (memcached)  https://review.openstack.org/10528918:15
openstackgerritA change was merged to openstack/tripleo-heat-templates: use ControlVirtualInterface for keepalived VRRP  https://review.openstack.org/10595918:19
openstackgerritA change was merged to openstack/tripleo-heat-templates: make glance.host point to controller_virtual_ip  https://review.openstack.org/10596418:19
*** jcoufal has joined #tripleo18:22
*** akrivoka has quit IRC18:28
jomaraany of you devtest experts help me for a hot minute? im setting it up for the first time, and im getting slightly different results than the guide suggests18:31
jomaranamely i dont have /dev/kvm after installing kvm18:32
jomara(kvm installs qemu-system-x86 now)18:32
openstackgerritA change was merged to openstack/tripleo-image-elements: Move libvirt's qemu configuration dir to state fs  https://review.openstack.org/10746118:34
*** rameshg87 has left #tripleo18:34
lifelessjomara: hey18:37
lifelessjomara: that should be ok18:37
lifelessmorning all18:38
lifelessgreghaynes: I am18:38
lifelessgreghaynes: I manually started mysql on ctl118:38
jomaralifeless: easy followup question: is this guide horribly stale and doomed? https://wiki.openstack.org/wiki/Tuskar/Devtest and should i use this one instead? http://docs.openstack.org/developer/tripleo-incubator/devtest.html18:38
lifelessjomara: it looks a little stale but not super18:41
lifelessjomara: its kindof odd its not in the main devtest docs tho :)18:41
lifelessgreghaynes: so I see the new ref you want18:43
lifeless10456918:43
lifelessgreghaynes: . seed.hp1.tripleo.org.rc18:43
lifelessgreghaynes: is the invokation to kick it into place18:43
*** noslzzp has quit IRC18:45
*** noslzzp has joined #tripleo18:46
*** noslzzp has quit IRC18:46
*** noslzzp has joined #tripleo18:47
lifelessgreghaynes: I replied to your -1 on https://review.openstack.org/#/c/105045/2/scripts/setup-endpoints18:49
openstackgerritpatrick-crews proposed a change to openstack-infra/tripleo-ci: Use /opt/git instead of /opt/stack/new  https://review.openstack.org/8997418:51
*** noslzzp has quit IRC18:53
lifelessgreghaynes: just rebasing the incubator stuff19:00
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Add public VLAN support.  https://review.openstack.org/10375719:09
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Add --public to setup-endpoints.  https://review.openstack.org/10504519:09
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Fix setup-endpoints handling of public URLs.  https://review.openstack.org/10504419:09
*** marun is now known as marun_afk19:15
*** markmc has quit IRC19:26
openstackgerritJulia Kreger proposed a change to openstack/tripleo-image-elements: Network init order fixes for VIP  https://review.openstack.org/10586219:29
Nglifeless: so fyi, adam_g managed to get a deployment with scheduler_host_subset_size and scheduler_max_attempts set to the number of nodes (29 in this case)19:32
lifelesshah19:33
lifelessthanks19:33
lifelessmakes me want to sob19:33
*** nibalizer has quit IRC19:38
*** nibalizer has joined #tripleo19:38
*** nati_ueno has joined #tripleo19:39
*** ddieterly has quit IRC19:46
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Add public VLAN support.  https://review.openstack.org/10375719:53
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Add public VLAN support.  https://review.openstack.org/10375719:56
*** nati_ueno has quit IRC20:00
*** nati_ueno has joined #tripleo20:02
*** nati_ueno has quit IRC20:05
*** nati_ueno has joined #tripleo20:07
*** dshurtle has joined #tripleo20:08
*** nati_ueno has quit IRC20:08
*** jtomasek has quit IRC20:11
openstackgerritRichard Su proposed a change to openstack/tripleo-image-elements: Correct SELinux file contexts  https://review.openstack.org/9924220:21
*** jdob has quit IRC20:25
openstackgerritRichard Su proposed a change to openstack/tripleo-image-elements: Correct SELinux file contexts  https://review.openstack.org/9924220:27
cinerama /back20:27
openstackgerritGhe Rivero proposed a change to openstack/tripleo-image-elements: Add pkg_map support to openvswitch elements  https://review.openstack.org/10696020:28
openstackgerritpatrick-crews proposed a change to openstack-infra/tripleo-ci: Test the upgrade codepath works as well.  https://review.openstack.org/8775820:28
lifelesscinerama: wotchya20:29
cineramalifeless: yo20:29
openstackgerritGhe Rivero proposed a change to openstack/tripleo-image-elements: Add pkg_map support to openvswitch elements  https://review.openstack.org/10696020:29
openstackgerritGhe Rivero proposed a change to openstack/tripleo-image-elements: Make use of openvswitch elements  https://review.openstack.org/10411520:29
*** rlandy has quit IRC20:30
lifelessso now I have a fun thing20:33
lifelessinstances are just dropping into nowhere20:33
lifeless| a2a69839-74d4-448f-8593-1d0626c5df7a | ci-overcloud-NovaCompute1-tgsrmhkp4lzo  | BUILD  | -          | NOSTATE     |                       |20:33
lifelessno task state20:33
lifelessnot in ERROR20:33
lifelessw.t.f.20:33
lifelessI want nice things, so much20:34
openstackgerritGhe Rivero proposed a change to openstack/diskimage-builder: Avoid to install a blank list of packages  https://review.openstack.org/10414020:34
*** michchap_ has quit IRC20:35
lifelesswaaaait, what.20:40
lifelessrescheduling is done by the compute node now20:40
lifelesshope it doesn't crash :/20:40
lifelessgreghaynes: new image, deploying again20:48
*** jml has quit IRC20:55
*** jml has joined #tripleo20:59
openstackgerritA change was merged to openstack-infra/tripleo-ci: Add some docs descibing tripleo CI  https://review.openstack.org/9970820:59
*** regebro has quit IRC20:59
*** untriaged-bot has joined #tripleo21:00
untriaged-botUntriaged bugs so far:21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134008721:00
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/134193621:00
*** untriaged-bot has quit IRC21:00
uvirtbotLaunchpad bug 1340087 in tripleo "PXE deploy failed with nova.virt.baremetal" [Undecided,New]21:00
uvirtbotLaunchpad bug 1341936 in os-collect-config "Require keystone v3 auth_url in heat collector" [Undecided,In progress]21:00
*** regebro has joined #tripleo21:01
*** dprince has quit IRC21:02
*** nati_ueno has joined #tripleo21:03
*** dshulyak_ has quit IRC21:03
*** johnma has left #tripleo21:20
*** chuckC has quit IRC21:22
*** rebrego has joined #tripleo21:27
*** regebro has quit IRC21:27
*** rebrego is now known as regebro21:27
*** nati_ueno has quit IRC21:41
*** jang2 has quit IRC21:43
*** gcha has quit IRC21:46
*** andreaf has joined #tripleo21:55
*** regebro has quit IRC21:55
*** andreaf has quit IRC21:56
*** andreaf has joined #tripleo21:56
*** ci-overcloud has joined #tripleo21:57
ci-overcloud************** ci-overcloud complete status=1 ************21:57
*** ci-overcloud has quit IRC21:57
*** gcha has joined #tripleo22:05
*** andreaf has quit IRC22:11
lifelesswow22:18
lifelessits still happening22:18
* lifeless tries a compute restart22:18
lifelessnova-conductor logging is approximately useless :(22:19
*** hewbrocca has quit IRC22:23
*** chuckC has joined #tripleo22:23
*** andreaf has joined #tripleo22:27
*** andreaf has quit IRC22:27
*** andreaf has joined #tripleo22:27
*** andreaf has quit IRC22:27
*** andreaf has joined #tripleo22:28
*** andreaf has quit IRC22:28
*** andreaf has joined #tripleo22:28
*** andreaf has quit IRC22:28
*** andreaf has joined #tripleo22:29
*** andreaf has quit IRC22:29
*** andreaf has joined #tripleo22:29
*** andreaf has quit IRC22:30
*** andreaf has joined #tripleo22:30
*** andreaf has quit IRC22:30
*** andreaf has joined #tripleo22:31
*** andreaf has quit IRC22:31
*** andreaf has joined #tripleo22:31
*** andreaf has quit IRC22:31
*** andreaf has joined #tripleo22:32
*** andreaf has quit IRC22:32
*** dsneddon is now known as dsneddon_gone22:32
*** andreaf has joined #tripleo22:32
*** andreaf has quit IRC22:32
*** andreaf has joined #tripleo22:33
*** andreaf has quit IRC22:33
*** andreaf has joined #tripleo22:33
*** andreaf has quit IRC22:34
*** andreaf has joined #tripleo22:34
*** andreaf has quit IRC22:34
*** andreaf has joined #tripleo22:35
*** andreaf has quit IRC22:35
*** andreaf has joined #tripleo22:35
*** andreaf has quit IRC22:35
*** andreaf has joined #tripleo22:36
*** andreaf has quit IRC22:36
*** andreaf has joined #tripleo22:36
*** andreaf has quit IRC22:37
*** andreaf has joined #tripleo22:37
*** andreaf has quit IRC22:37
*** andreaf has joined #tripleo22:38
*** andreaf has quit IRC22:38
*** andreaf has joined #tripleo22:38
*** andreaf has quit IRC22:38
*** andreaf has joined #tripleo22:39
*** andreaf has quit IRC22:39
*** andreaf has joined #tripleo22:39
*** andreaf has quit IRC22:40
*** andreaf has joined #tripleo22:40
*** andreaf has quit IRC22:40
*** andreaf has joined #tripleo22:41
*** andreaf has quit IRC22:41
*** andreaf has joined #tripleo22:41
*** andreaf has quit IRC22:41
*** andreaf has joined #tripleo22:42
*** andreaf has quit IRC22:42
*** andreaf has joined #tripleo22:42
*** andreaf has quit IRC22:43
*** andreaf has joined #tripleo22:43
*** andreaf has quit IRC22:43
*** andreaf has joined #tripleo22:44
*** andreaf has quit IRC22:44
*** andreaf has joined #tripleo22:44
*** andreaf has quit IRC22:45
*** andreaf has joined #tripleo22:45
*** andreaf has quit IRC22:45
*** andreaf has joined #tripleo22:46
*** andreaf has quit IRC22:46
*** andreaf has joined #tripleo22:46
*** andreaf has quit IRC22:47
*** andreaf has joined #tripleo22:47
*** andreaf has quit IRC22:47
*** andreaf has joined #tripleo22:48
*** andreaf has quit IRC22:48
*** andreaf has joined #tripleo22:48
*** andreaf has quit IRC22:49
*** andreaf has joined #tripleo22:49
*** andreaf has quit IRC22:49
*** morganfainberg has quit IRC22:50
*** andreaf has joined #tripleo22:50
*** andreaf has quit IRC22:50
*** morganfainberg has joined #tripleo22:50
*** andreaf has joined #tripleo22:50
*** jerryz_ has quit IRC22:50
*** morganfainberg has quit IRC22:50
*** andreaf has quit IRC22:50
*** jerryz_ has joined #tripleo22:51
*** pcrews has quit IRC22:51
*** ekarlso has quit IRC22:51
*** GheRivero has quit IRC22:51
*** boris-42 has quit IRC22:51
*** morganfainberg has joined #tripleo22:51
*** andreaf has joined #tripleo22:51
*** andreaf has quit IRC22:51
*** morganfainberg has quit IRC22:51
*** andreaf has joined #tripleo22:52
*** morganfainberg has joined #tripleo22:52
*** andreaf has quit IRC22:52
*** chuckC has quit IRC22:52
*** sseago has quit IRC22:52
*** panda___ has quit IRC22:52
*** BadCub has quit IRC22:52
*** kevinbenton has quit IRC22:52
*** greghaynes has quit IRC22:52
*** bhunter71 has quit IRC22:52
*** gilliard has quit IRC22:52
*** otoolee has quit IRC22:52
*** jomara has quit IRC22:52
*** uvirtbot has quit IRC22:52
*** davidlenwell has quit IRC22:52
*** andreaf has joined #tripleo22:52
*** andreaf has quit IRC22:53
*** chuckC has joined #tripleo22:53
*** sseago has joined #tripleo22:53
*** panda___ has joined #tripleo22:53
*** BadCub has joined #tripleo22:53
*** kevinbenton has joined #tripleo22:53
*** gilliard has joined #tripleo22:53
*** greghaynes has joined #tripleo22:53
*** bhunter71 has joined #tripleo22:53
*** otoolee has joined #tripleo22:53
*** jomara has joined #tripleo22:53
*** uvirtbot has joined #tripleo22:53
*** davidlenwell has joined #tripleo22:53
*** dickson.freenode.net sets mode: +v davidlenwell22:53
*** andreaf has joined #tripleo22:53
*** pcrews has joined #tripleo22:53
*** boris-42 has joined #tripleo22:53
*** ekarlso has joined #tripleo22:53
*** GheRivero has joined #tripleo22:53
*** andreaf has quit IRC22:53
*** andreaf has joined #tripleo22:53
*** mikal has quit IRC22:54
*** lxsli has quit IRC22:54
*** andreaf has quit IRC22:54
*** mikal has joined #tripleo22:54
*** lxsli has joined #tripleo22:54
*** jrist has quit IRC22:54
*** lynxman has quit IRC22:54
*** pleia2 has quit IRC22:54
*** jang has quit IRC22:54
*** stevebaker has quit IRC22:54
*** dtantsur|afk has quit IRC22:54
*** zigo has quit IRC22:54
*** adam_g has quit IRC22:54
*** therve has quit IRC22:54
*** jpeeler has quit IRC22:54
*** fungi has quit IRC22:54
*** gonery has quit IRC22:54
*** agordeev has quit IRC22:54
*** StevenK has quit IRC22:54
*** andreaf has joined #tripleo22:54
*** andreaf has quit IRC22:54
*** pmyers has quit IRC22:55
*** tserong has quit IRC22:55
*** dsneddon_gone has quit IRC22:55
*** gothicmindfood has quit IRC22:55
*** tchaypo has quit IRC22:55
*** Hazelesque has quit IRC22:55
*** vinsh has quit IRC22:55
*** al has quit IRC22:55
*** tteggel has quit IRC22:55
*** dkehn has quit IRC22:55
*** Ng has quit IRC22:55
*** bkero has quit IRC22:55
*** chuckC has quit IRC22:55
*** jrist has joined #tripleo22:55
*** lynxman has joined #tripleo22:55
*** pleia2 has joined #tripleo22:55
*** jang has joined #tripleo22:55
*** stevebaker has joined #tripleo22:55
*** dtantsur|afk has joined #tripleo22:55
*** zigo has joined #tripleo22:55
*** adam_g has joined #tripleo22:55
*** therve has joined #tripleo22:55
*** jpeeler has joined #tripleo22:55
*** fungi has joined #tripleo22:55
*** gonery has joined #tripleo22:55
*** agordeev has joined #tripleo22:55
*** StevenK has joined #tripleo22:55
*** andreaf has joined #tripleo22:55
*** andreaf has quit IRC22:55
*** andreaf has joined #tripleo22:55
*** andreaf has quit IRC22:56
*** pmyers has joined #tripleo22:56
*** tserong has joined #tripleo22:56
*** dsneddon_gone has joined #tripleo22:56
*** gothicmindfood has joined #tripleo22:56
*** tchaypo has joined #tripleo22:56
*** Hazelesque has joined #tripleo22:56
*** vinsh has joined #tripleo22:56
*** bkero has joined #tripleo22:56
*** al has joined #tripleo22:56
*** tteggel has joined #tripleo22:56
*** dkehn has joined #tripleo22:56
*** Ng has joined #tripleo22:56
*** dickson.freenode.net sets mode: +v Ng22:56
*** andreaf has joined #tripleo22:56
*** andreaf has quit IRC22:56
*** andreaf has joined #tripleo22:57
*** andreaf has quit IRC22:57
*** andreaf has joined #tripleo22:57
*** andreaf has quit IRC22:58
*** andreaf has joined #tripleo22:58
*** andreaf has quit IRC22:58
*** andreaf has joined #tripleo22:59
*** andreaf has quit IRC22:59
*** andreaf has joined #tripleo22:59
*** andreaf has quit IRC23:00
*** andreaf has joined #tripleo23:00
*** chuckC has joined #tripleo23:03
lifelesswow something vvvery wrong here23:06
*** nati_ueno has joined #tripleo23:06
lifelesshas anyone else see reschedules not happenign23:23
lifelesslike, when it should happen it goes nowhere23:23
lifelessstate BUILD task -23:23
tchaypolifeless: good morning23:26
*** cinerama has quit IRC23:27
lifelesstchaypo: hai23:27
tchaypoI'm not sure why that was directed at lifeless23:27
tchaypothe rest of you deserve a good morning too23:27
*** cody-somerville has joined #tripleo23:28
*** cody-somerville has quit IRC23:28
*** cody-somerville has joined #tripleo23:28
tchaypoguy behind me on the train says "I'm on the phone so there might be some background noise"23:30
tchaypono dude, the background noise is *you*23:30
tchaypohe's attracting lots of stares23:30
tchaypomy plan for today is to poke at the stack I built yesterday and see if it's working as expected23:34
tchaypoat a gross level that means i should be able to connect through to the services bound to the public IP. If that doesn't work I'll have to drill down on the networking setup on all the layers23:35
tchaypoand fwiw, using 32bit vms on a server (no firefox running) I'm sitting at 15298700 used after b/c taken out. There's no way that's going to fit on a 16Gb laptop23:36
tchaypoand that's my stop so I'll be getting out now. back in 20-30 when I find the room23:36
lifelessbc?23:37
*** bashok has joined #tripleo23:45
*** jerryz_ has quit IRC23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!