*** ctayal has quit IRC | 00:02 | |
*** mburned is now known as mburned_out | 00:10 | |
*** penick_ has quit IRC | 00:16 | |
*** dmarlin has quit IRC | 00:17 | |
*** liverpooler has quit IRC | 00:32 | |
*** lblanchard has joined #tripleo | 00:32 | |
openstackgerrit | Merged openstack/instack-undercloud: Changed author and author-email https://review.openstack.org/399367 | 00:33 |
---|---|---|
openstackgerrit | Merged openstack/python-tripleoclient: Clean up image exceptions https://review.openstack.org/404408 | 00:33 |
*** numans has quit IRC | 00:37 | |
*** limao has joined #tripleo | 00:37 | |
openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Improve checksum performance for images https://review.openstack.org/400311 | 00:48 |
*** flepied has joined #tripleo | 00:58 | |
openstackgerrit | Merged openstack/python-tripleoclient: Exit before post-deploy code if we are updating the plan only https://review.openstack.org/402025 | 01:00 |
openstackgerrit | Merged openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/400911 | 01:00 |
openstackgerrit | Merged openstack/puppet-tripleo: pacemaker: create Mysql_user once Galera is ready (puppet4) https://review.openstack.org/404437 | 01:00 |
*** nyechiel has joined #tripleo | 01:13 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: deployment updates for containerized compute https://review.openstack.org/400986 | 01:14 |
*** rajinir has quit IRC | 01:16 | |
*** dmacpher has quit IRC | 01:20 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: config for containerized-compute https://review.openstack.org/393348 | 01:27 |
rook | mwhahaha should the # of workers be 1/2 the cores or 1/2 one socket? | 01:41 |
mwhahaha | It uses processor count so cores | 01:43 |
mwhahaha | And it's count/4 | 01:43 |
rook | ahh ok mwhahaha | 01:43 |
rook | mwhahaha: how did you come up with /4 | 01:45 |
*** nyechiel has quit IRC | 01:46 | |
mwhahaha | Ask pradk | 01:47 |
*** dhill_ has quit IRC | 01:48 | |
*** dhill__ has joined #tripleo | 01:48 | |
mwhahaha | rook: so these are defaults and not necessarily the most optimized possible settings. The goal is to not do stupid stuff by default. Any tunings should be handled outside of the defaults | 01:49 |
rook | the /4th is not a default. | 01:51 |
mwhahaha | Trying to come up with a single formula for all services is probably not realistic so we started with a conservative approach. If we need to adjust please provided data for an alternative | 01:51 |
rook | a default is based on cores... I don't see a good reason to /4 -- but I am happy to review data you guys have | 01:51 |
rook | to suggest otherwise. | 01:51 |
rook | mwhahaha: yeah, I am running against trunk | 01:51 |
*** Goneri has quit IRC | 01:52 | |
mwhahaha | For keystone, gnocchi and other services we found our formula to be a better alternative to the python cores only default | 01:52 |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: WIP: Create a demo Mistral workflow for Tendlr integration https://review.openstack.org/404499 | 01:52 |
mwhahaha | Also things like Solaris where cores are in the hundreds makes that a non starter | 01:53 |
mwhahaha | The formula is is a min of 2 with a sliding scale of cores/4 to a max of 8 | 01:53 |
mwhahaha | We can adjust it to /2 with a max of 16 if you find that to work better | 01:54 |
rook | mwhahaha ill get some data. | 01:54 |
mwhahaha | But I have yet to see additional data to indicate something better | 01:54 |
rook | mwhahaha will be on the back burner during some other things | 01:54 |
rook | mwhahaha it would be good to know what and how you tested | 01:55 |
rook | ie, using Rally at a single concurrency and times, or multiple | 01:55 |
rook | and what scenarios | 01:55 |
rook | anyway, its late here. | 01:55 |
rook | tahnks mwhahaha | 01:55 |
mwhahaha | K | 01:56 |
rook | *thanks | 01:56 |
*** ccamacho has quit IRC | 01:57 | |
thrash | EmilienM: thanks. I'll look at it tomorrow. | 02:00 |
*** thrash is now known as thrash|g0ne | 02:02 | |
*** chem has quit IRC | 02:02 | |
*** ccamacho has joined #tripleo | 02:02 | |
*** chem has joined #tripleo | 02:04 | |
*** ctayal has joined #tripleo | 02:09 | |
*** lblanchard has quit IRC | 02:22 | |
*** bana_k has quit IRC | 02:24 | |
*** TSCHAK has quit IRC | 02:26 | |
*** dsneddon_ has quit IRC | 02:32 | |
*** dsneddon_ has joined #tripleo | 02:35 | |
*** dmacpher has joined #tripleo | 02:39 | |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: WIP: Create a demo Mistral workflow for Tendlr integration https://review.openstack.org/404499 | 02:42 |
*** fzdarsky_ has joined #tripleo | 02:42 | |
*** fzdarsky has quit IRC | 02:46 | |
*** ctayal has quit IRC | 02:48 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Fix grammar https://review.openstack.org/404442 | 02:49 |
*** flepied has quit IRC | 02:59 | |
*** achadha has quit IRC | 03:01 | |
*** achadha has joined #tripleo | 03:01 | |
*** ayoung has joined #tripleo | 03:02 | |
*** achadha has quit IRC | 03:06 | |
*** achadha has joined #tripleo | 03:08 | |
*** pmannidi has quit IRC | 03:09 | |
*** achadha has quit IRC | 03:12 | |
*** links has joined #tripleo | 03:28 | |
openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore https://review.openstack.org/365449 | 03:30 |
openstackgerrit | Merged openstack/instack-undercloud: Fix package update cases https://review.openstack.org/400956 | 03:37 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Improve checksum performance for images https://review.openstack.org/400311 | 03:43 |
*** dmacpher has quit IRC | 03:47 | |
*** achadha has joined #tripleo | 03:48 | |
*** achadha has quit IRC | 03:53 | |
*** achadha has joined #tripleo | 04:02 | |
*** dmacpher has joined #tripleo | 04:04 | |
*** dsneddon_ has quit IRC | 04:06 | |
*** numans has joined #tripleo | 04:06 | |
*** bana_k has joined #tripleo | 04:14 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Perform package install outside of debootstrap https://review.openstack.org/392253 | 04:16 |
openstackgerrit | Merged openstack/diskimage-builder: Fix a typo https://review.openstack.org/399341 | 04:20 |
*** bana_k has quit IRC | 04:24 | |
*** misty-cloud has joined #tripleo | 04:27 | |
*** sudipto has joined #tripleo | 04:31 | |
*** sudipto_ has joined #tripleo | 04:31 | |
*** prateek has joined #tripleo | 04:31 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Fix grammar https://review.openstack.org/404442 | 04:33 |
*** yamahata has quit IRC | 04:35 | |
*** Vijayendra_ has joined #tripleo | 04:36 | |
*** pmannidi has joined #tripleo | 04:39 | |
*** dsneddon_ has joined #tripleo | 04:46 | |
*** achadha has quit IRC | 04:48 | |
*** achadha has joined #tripleo | 04:50 | |
*** dsneddo__ has joined #tripleo | 04:51 | |
*** misty-cloud has quit IRC | 04:51 | |
*** rlandy|fallout has quit IRC | 04:51 | |
*** udesale has joined #tripleo | 04:51 | |
*** misty-cloud has joined #tripleo | 04:51 | |
*** achadha_ has joined #tripleo | 04:52 | |
*** dsneddon_ has quit IRC | 04:52 | |
*** prateek has quit IRC | 04:53 | |
*** sudipto_ has quit IRC | 04:54 | |
*** sudipto has quit IRC | 04:54 | |
*** dsneddo__ has quit IRC | 04:54 | |
*** prateek has joined #tripleo | 04:54 | |
*** achadha has quit IRC | 04:55 | |
*** dsneddon_ has joined #tripleo | 04:56 | |
*** ola has joined #tripleo | 04:58 | |
*** ola is now known as Guest9347 | 04:58 | |
*** yprokule has joined #tripleo | 05:05 | |
*** dsneddo__ has joined #tripleo | 05:06 | |
openstackgerrit | Merged openstack/diskimage-builder: Improve checksum performance for images https://review.openstack.org/400311 | 05:07 |
*** dsneddon_ has quit IRC | 05:07 | |
*** masco has joined #tripleo | 05:07 | |
*** ramishra has quit IRC | 05:16 | |
*** Guest9347 has quit IRC | 05:17 | |
*** ramishra has joined #tripleo | 05:18 | |
*** saneax-_-|AFK is now known as saneax | 05:19 | |
*** sudipto has joined #tripleo | 05:19 | |
*** sudipto_ has joined #tripleo | 05:20 | |
*** dsneddo__ has quit IRC | 05:23 | |
*** iranzo has joined #tripleo | 05:24 | |
*** iranzo has joined #tripleo | 05:24 | |
*** dsneddon_ has joined #tripleo | 05:25 | |
*** ctayal has joined #tripleo | 05:28 | |
*** chandankumar|off is now known as chandankumar | 05:29 | |
*** dsneddon_ has quit IRC | 05:30 | |
*** dsneddo__ has joined #tripleo | 05:32 | |
openstackgerrit | Merged openstack/diskimage-builder: Perform package install outside of debootstrap https://review.openstack.org/392253 | 05:32 |
*** dsnedd___ has joined #tripleo | 05:35 | |
*** dsneddo__ has quit IRC | 05:39 | |
*** ctayal has quit IRC | 05:42 | |
*** markmc has quit IRC | 05:42 | |
*** markmc has joined #tripleo | 05:43 | |
*** ctayal has joined #tripleo | 05:43 | |
*** Vijayendra__ has joined #tripleo | 05:48 | |
*** Vijayendra_ has quit IRC | 05:50 | |
*** achadha_ has quit IRC | 05:52 | |
*** pkovar has joined #tripleo | 05:59 | |
*** abregman has joined #tripleo | 05:59 | |
*** ctayal has quit IRC | 06:06 | |
*** jaosorior has joined #tripleo | 06:06 | |
*** dsnedd___ has quit IRC | 06:10 | |
*** dsneddon_ has joined #tripleo | 06:17 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: debugging - do not merge https://review.openstack.org/404560 | 06:17 |
*** achadha has joined #tripleo | 06:21 | |
*** yamahata has joined #tripleo | 06:26 | |
*** dsneddo__ has joined #tripleo | 06:40 | |
*** dsneddon_ has quit IRC | 06:43 | |
*** dmacpher has quit IRC | 06:47 | |
*** dsneddon has quit IRC | 06:48 | |
*** dsneddo__ has quit IRC | 06:49 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Add option to set undercloud dns nameserver https://review.openstack.org/400937 | 06:49 |
*** chem has quit IRC | 06:53 | |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use the overcloudrc generated in a Mistral action https://review.openstack.org/398975 | 06:55 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Only start the deploy if the Heat stack isn't already in progress https://review.openstack.org/396567 | 06:56 |
*** oshvartz has joined #tripleo | 06:58 | |
openstackgerrit | Merged openstack/python-tripleoclient: Show team and repo badges on README https://review.openstack.org/402583 | 07:01 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Add option to set undercloud dns nameserver https://review.openstack.org/400937 | 07:02 |
*** bana_k has joined #tripleo | 07:05 | |
jaosorior | d0ugal: is there a timeout for the plan upload? | 07:06 |
*** lmiccini has joined #tripleo | 07:07 | |
d0ugal | jaosorior: not exactly - plan upload happens directly with swift. so there is no zaqar queue etc. | 07:07 |
d0ugal | jaosorior: swiftclient may have some timeouts, so maybe - but nothing we add | 07:07 |
*** jaosorior has quit IRC | 07:08 | |
* d0ugal waits for jaosorior to return | 07:08 | |
*** jaosorior has joined #tripleo | 07:09 | |
d0ugal | jaosorior: not exactly - plan upload happens directly with swift. so there is no zaqar queue etc. | 07:09 |
*** aufi has joined #tripleo | 07:09 | |
d0ugal | jaosorior: swiftclient may have some timeouts, so maybe - but nothing we add | 07:09 |
jaosorior | damn flaky internet.. | 07:09 |
jaosorior | aah, alright | 07:09 |
jaosorior | gotta check that out then | 07:09 |
jaosorior | recently saw a timeout there | 07:09 |
d0ugal | jaosorior: but it depends what you mean by "plan upload" :) | 07:09 |
jaosorior | that was the last thing in the tripleoclient log | 07:09 |
jaosorior | trying to reproduce it now | 07:09 |
d0ugal | jaosorior: creating a plan is a series of three steps - you call a mistral action that creates the swift container, you upload the files directly to swift, you call a mistral workflow to process the files | 07:10 |
d0ugal | jaosorior: only the last step has a timeout, as it is the only bit that uses zaqar for the response | 07:10 |
d0ugal | jaosorior: Can you link me to the log? | 07:10 |
jaosorior | d0ugal: http://logs.openstack.org/38/391738/47/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/ce04860/console.html#_2016-11-29_19_10_01_473354 | 07:11 |
jaosorior | d0ugal: and the problem is that since it didn't have a timeout, it just got killed by devstack and we got no more relevant logs back :/ | 07:11 |
* d0ugal just remembered it is a public holiday here | 07:11 | |
jaosorior | d0ugal: ah, nice! what's celebrated? | 07:12 |
d0ugal | jaosorior: St. Andrew's Day | 07:12 |
d0ugal | jaosorior: I am going to guess that the files uploaded to swift and the workflow was triggered. Just looking at the logs. | 07:13 |
d0ugal | oh, since it timed out the logs are missing? | 07:13 |
jaosorior | d0ugal: yep | 07:14 |
d0ugal | dammit! This is the worst :P | 07:14 |
jaosorior | indeed | 07:14 |
jaosorior | well, it seems that it worked this time | 07:14 |
jaosorior | now I see 2016-11-30 07:13:58.977726 | Waiting for messages on queue '665e026c-61b8-42ea-86ab-138c84050154' with no timeout. | 07:15 |
d0ugal | oh | 07:15 |
d0ugal | Maybe I missed one. | 07:15 |
d0ugal | jaosorior: that message coule mean the deploy has been started - we don't timeout that. | 07:18 |
jaosorior | 2016-11-30 07:13:20.217316 | Removing the current plan files | 07:18 |
jaosorior | 2016-11-30 07:13:31.874595 | Uploading new plan files | 07:18 |
jaosorior | 2016-11-30 07:13:43.194591 | Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: 086f9cd4-0841-40c9-919d-fb09d094d164 | 07:18 |
jaosorior | 2016-11-30 07:13:55.009823 | Plan updated | 07:18 |
jaosorior | 2016-11-30 07:13:55.334107 | Deploying templates in the directory /tmp/tripleoclient-K9WQ19/tripleo-heat-templates | 07:18 |
jaosorior | 2016-11-30 07:13:58.054448 | Started Mistral Workflow tripleo.deployment.v1.deploy_plan. Execution ID: 05e62ab5-d5c2-4fce-a85e-5a92e92aca5e | 07:18 |
jaosorior | 2016-11-30 07:13:58.977726 | Waiting for messages on queue '665e026c-61b8-42ea-86ab-138c84050154' with no timeout. | 07:18 |
jaosorior | d0ugal: ^^ | 07:18 |
d0ugal | jaosorior: cool, so yeah, it is for the deploy | 07:19 |
jaosorior | yay | 07:19 |
d0ugal | jaosorior: that message is probably a bit noisy, alarming. I wonder if I should change it | 07:19 |
jaosorior | strange I'm not seeing any heat events print out. | 07:19 |
d0ugal | jaosorior: that is the start of the deploy workflow | 07:20 |
d0ugal | jaosorior: so I would expect you to see Heat events soon... | 07:20 |
d0ugal | but yeah, that was 8 mins ago already :/ | 07:20 |
jaosorior | yep | 07:20 |
jaosorior | pretty strange | 07:20 |
jaosorior | but I think I had seen issues with this before | 07:20 |
jaosorior | AFAIK the overcloud deploy does timeout nicely | 07:20 |
jaosorior | so some logs will be had | 07:20 |
*** rasca has joined #tripleo | 07:22 | |
*** fzdarsky_ is now known as fzdarsky | 07:24 | |
*** rcernin has joined #tripleo | 07:28 | |
*** dciabrin|away is now known as dciabrin | 07:28 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Add option to set undercloud dns nameserver https://review.openstack.org/400937 | 07:29 |
d0ugal | jaosorior: any Heat events yet? | 07:30 |
*** jprovazn has joined #tripleo | 07:31 | |
d0ugal | jaosorior: I am going to add a timeout to the deploy workflow - probably 6 mins again | 07:31 |
d0ugal | jaosorior: it should be really fast, here it was about 25 seconds. http://logs.openstack.org/75/398975/9/check/gate-tripleo-ci-centos-7-nonha-multinode/de52a7c/console.html#_2016-11-29_18_08_46_926367 | 07:31 |
d0ugal | (just to pick a randon CI run) | 07:31 |
jaosorior | d0ugal: no heat events | 07:31 |
*** mcornea has joined #tripleo | 07:32 | |
openstackgerrit | Christian Schwede proposed openstack/puppet-tripleo: Set memcache_servers in /etc/swift/object-expirer.conf https://review.openstack.org/404598 | 07:33 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Add option to set undercloud dns nameserver https://review.openstack.org/400937 | 07:34 |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add a timeout when waiting for messages from the deploy workflow https://review.openstack.org/404601 | 07:35 |
d0ugal | jaosorior: You might want to depend-on ^ for your next try | 07:35 |
*** florianf has joined #tripleo | 07:35 | |
jaosorior | nice | 07:36 |
jaosorior | d0ugal: will do, thanks dude | 07:36 |
d0ugal | jaosorior: no worries, it's quite alarming that this all fails so badly | 07:36 |
d0ugal | We really need to rewrite some of the CLI so that waiting for messages isn't a blocking operation. Then it could check the workflow status periodically | 07:38 |
d0ugal | but I've not had time to figure out how to do that yet... | 07:38 |
jaosorior | d0ugal: well, partly it's my fault, the TLS everywhere job I'm testing changes the fqdn of the undercloud, and that's where I started stumbling with zaqar errors, since it autodetects its hostname on startup, so the first issue was that, that it was trying to send notifications to itself but it was failing. | 07:39 |
jaosorior | really hitting some corner cases here | 07:39 |
d0ugal | hah, nice | 07:40 |
d0ugal | You should write this up at the end, sounds like it could make a good war story :) | 07:40 |
jaosorior | d0ugal: haha maybe when it's over | 07:40 |
d0ugal | once you have recovered! | 07:41 |
jaosorior | d0ugal: I'm pretty sure I'm gonna hit selinux issues (again) when I finally get a proper overcloud deploy to start | 07:41 |
d0ugal | :/ | 07:41 |
jaosorior | it'll be fiiine | 07:42 |
jaosorior | at some point this has to work | 07:43 |
*** pmannidi has quit IRC | 07:43 | |
*** bana_k has quit IRC | 07:45 | |
*** achadha has quit IRC | 07:46 | |
*** anshul has joined #tripleo | 07:49 | |
*** sudipto has quit IRC | 07:50 | |
*** sudipto_ has quit IRC | 07:50 | |
*** dsariel has joined #tripleo | 07:51 | |
*** leanderthal|afk is now known as leanderthal | 07:51 | |
*** ealcaniz has joined #tripleo | 07:56 | |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use the overcloudrc generated in a Mistral action https://review.openstack.org/398975 | 08:00 |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] overcloudrc command https://review.openstack.org/404354 | 08:00 |
openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates: WIP -- Add pacemaker ansible module of composable upgrade https://review.openstack.org/403397 | 08:01 |
*** limao has quit IRC | 08:03 | |
*** Vijayendra__ has quit IRC | 08:04 | |
*** shardy has joined #tripleo | 08:04 | |
*** artom_ has joined #tripleo | 08:04 | |
*** artom_ has quit IRC | 08:05 | |
*** artom_ has joined #tripleo | 08:05 | |
*** links has quit IRC | 08:06 | |
*** artom has quit IRC | 08:07 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart: Add option to set DNS name servers on the undercloud https://review.openstack.org/401452 | 08:09 |
*** dsneddon has joined #tripleo | 08:11 | |
ccamacho | morning guys! | 08:11 |
matbu | ccamacho: hello | 08:12 |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] overcloudrc command https://review.openstack.org/404354 | 08:12 |
d0ugal | Morning! | 08:13 |
matbu | sshnaidm: hey, when you will be there, could you point me to an example of your oooq/tripleo-ci jobs ? | 08:13 |
jaosorior | ccamacho: morning dude, how's it going? | 08:13 |
*** dsneddon_ has joined #tripleo | 08:14 | |
jaosorior | shardy: in CI, do we have a timeout for the overcloud deploy? or does it just keep going until the job itself times out? | 08:15 |
ccamacho | jaosorior matbu , quick question, do you know in which cases I can have "hiera -c /etc/puppet/hiera.yaml bootstrap_nodeid" == nil http://paste.openstack.org/show/590919/ Im running the upgrade but it hangs, because is not starting the pcs cluster | 08:16 |
ccamacho | And I only have one controller | 08:16 |
ccamacho | o/ | 08:16 |
jaosorior | I actually don't know in which cases you could get that, AFAIK it shouldn't be nil | 08:17 |
*** jlinkes has joined #tripleo | 08:17 | |
*** Vijayendra has joined #tripleo | 08:17 | |
*** dsneddon has quit IRC | 08:17 | |
jaosorior | ccamacho: try sudo hiera bootstrap_nodeid | 08:17 |
matbu | ccamacho: jaosorior yep, i shouldn't, but it doesn't help you :) | 08:17 |
matbu | s/i/it/ | 08:17 |
jaosorior | ccamacho: might be a permissions issue | 08:18 |
matbu | ccamacho: i have one node and mine is overcloud-controller-0 | 08:18 |
shardy | jaosorior: we do see OVERCLOUD_DEPLOY_TIMEOUT in toci_gate_test.sh | 08:18 |
*** links has joined #tripleo | 08:18 | |
matbu | ccamacho: yep , try with sudo | 08:18 |
matbu | ccamacho: you are logged at heat-admin | 08:19 |
*** arxcruz has joined #tripleo | 08:19 | |
sshnaidm | matbu, hi | 08:19 |
sshnaidm | matbu, it's two experimental jobs | 08:19 |
*** limao has joined #tripleo | 08:19 | |
*** pgadiya has joined #tripleo | 08:19 | |
sshnaidm | matbu, example run: https://review.openstack.org/#/c/403848/ | 08:20 |
jaosorior | shardy: I see it now, thanks | 08:21 |
sshnaidm | matbu, any of ovb jobs there, i.e. http://logs.openstack.org/48/403848/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/c9e93c5/console.html | 08:21 |
sshnaidm | matbu, right now it's broken until this patch will be merged: https://review.openstack.org/#/c/403843/ | 08:21 |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] overcloudrc command https://review.openstack.org/404354 | 08:22 |
*** jpena|off is now known as jpena | 08:22 | |
sshnaidm | btw, jaosorior , shardy could you please merge fixes for quickstart job? https://review.openstack.org/#/c/403843/ thanks | 08:22 |
openstackgerrit | Merged openstack/tripleo-common: Fix invalid tests. https://review.openstack.org/404185 | 08:23 |
jaosorior | sshnaidm: it seems the oooq jobs are still failing in that CR | 08:24 |
ccamacho | matbu jaosorior, yeahp, that made the trick,:) | 08:24 |
sshnaidm | jaosorior, I run them here: https://review.openstack.org/#/c/403848 | 08:24 |
jaosorior | sshnaidm: right, so I guess they are still running. Is it cool if we wait for the results of that CR? | 08:25 |
*** amoralej|off is now known as amoralej | 08:25 | |
matbu | sshnaidm: ack thank you :) | 08:25 |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] overcloudrc command https://review.openstack.org/404354 | 08:25 |
sshnaidm | jaosorior, I mean I ran them :) but no problem to recheck again | 08:26 |
*** cylopez has joined #tripleo | 08:26 | |
jaosorior | sshnaidm: ah, I didn't see the oooq results there | 08:27 |
jaosorior | sshnaidm: aaah now I see | 08:27 |
jaosorior | hadn't seen that you run oooq instead of ovb there | 08:27 |
sshnaidm | jaosorior, yeah, little trick there | 08:28 |
jaosorior | sshnaidm: done | 08:28 |
sshnaidm | jaosorior, cool, thanks! | 08:29 |
jaosorior | shardy: I am trying to create a job and for some reason the overcloud deployment keeps timing out; I'm not sure what's causing that, could you help me out figuring it out? http://logs.openstack.org/38/391738/47/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/9bb4796/ really stuck here :/ | 08:31 |
*** ebarrera has joined #tripleo | 08:31 | |
*** dsneddon has joined #tripleo | 08:32 | |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] overcloudrc command https://review.openstack.org/404354 | 08:32 |
*** dsneddon_ has quit IRC | 08:35 | |
*** aufi has quit IRC | 08:37 | |
*** aufi has joined #tripleo | 08:40 | |
*** sudipto_ has joined #tripleo | 08:40 | |
*** sudipto has joined #tripleo | 08:40 | |
*** akrivoka has joined #tripleo | 08:42 | |
*** tremble has joined #tripleo | 08:45 | |
*** numans has quit IRC | 08:47 | |
*** achadha has joined #tripleo | 08:47 | |
*** tremble has quit IRC | 08:51 | |
*** abregman is now known as abregman|mtg | 08:52 | |
*** achadha has quit IRC | 08:52 | |
shardy | http://logs.openstack.org/38/391738/47/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/9bb4796/logs/undercloud/var/log/heat/heat-engine.txt.gz#_2016-11-30_08_21_23_312 | 08:56 |
shardy | jaosorior: looks like heat can't connect to rabbit | 08:56 |
shardy | so it 's just got stuck | 08:56 |
jaosorior | shardy: but that happened way down the logs. could it have been that it timed out and the rabbit connection was closed? | 08:57 |
shardy | jaosorior: well the timestamp says the first rabbit timeout happend about 1 minute after the stack create started | 08:58 |
shardy | perhaps rabbit got OOM killed or something? | 08:58 |
*** b00tcat has joined #tripleo | 08:58 | |
*** b00tcat has quit IRC | 08:58 | |
*** b00tcat has joined #tripleo | 08:58 | |
shardy | Hmm, can't see that happening | 08:59 |
shardy | http://logs.openstack.org/38/391738/47/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/9bb4796/logs/undercloud/var/log/rabbitmq/rabbit@tripleo-centos-7-tripleo-test-cloud-rh1-5877384-sasl.txt.gz | 08:59 |
shardy | That has a bunch of crash reports in it | 08:59 |
*** numans has joined #tripleo | 09:00 | |
shardy | looks like a name lookup failure? | 09:00 |
jaosorior | shardy: name lookup failure? | 09:01 |
shardy | jaosorior: well there's an nxdomain in the error | 09:01 |
shardy | Reason: {badmatch,{error,nxdomain}} | 09:02 |
shardy | looks dns-ish to me, but I guess you'll need to reproduce locally and see | 09:02 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 09:02 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: debugging - do not merge https://review.openstack.org/404560 | 09:02 |
jaosorior | shardy: haven't been able to reproduce it... | 09:03 |
jaosorior | shardy: but I do change the undercloud's fqdn after the undercloud install, which is the only variable I know of that I don't do in my local deployment | 09:03 |
jaosorior | trying to do that before the undercloud install, lets see if that works | 09:04 |
*** numans has quit IRC | 09:04 | |
shardy | jaosorior: aha, sounds like it's worth trying | 09:04 |
openstackgerrit | Aparna proposed openstack/diskimage-builder: Update hpssacli to ssacli in proliant-tools element https://review.openstack.org/396504 | 09:05 |
openstackgerrit | Anshul Jain proposed openstack/diskimage-builder: DIB element to support cinder local attach/detach functionality https://review.openstack.org/385880 | 09:06 |
*** yprokule has quit IRC | 09:08 | |
*** yamahata has quit IRC | 09:09 | |
jaosorior | By the way, adding encrypted volumes to the scenario tests, can someone check this out? https://review.openstack.org/#/c/404223/ | 09:11 |
*** jaosorior is now known as jaosorior_lunch | 09:13 | |
*** zoli|gone is now known as zoli|tng | 09:25 | |
*** lucas-afk is now known as lucasagomes | 09:33 | |
*** Vijayendra has quit IRC | 09:36 | |
*** zoli|tng is now known as zoli|trng | 09:39 | |
*** oshvartz has quit IRC | 09:40 | |
*** oshvartz has joined #tripleo | 09:42 | |
openstackgerrit | yolanda.robla proposed openstack/tripleo-quickstart: Always upload overcloud image when has been customized https://review.openstack.org/396849 | 09:47 |
openstackgerrit | vikrant proposed openstack/tripleo-validations: Fixes undercloud-tokenflush validation https://review.openstack.org/404688 | 09:48 |
*** yprokule has joined #tripleo | 09:49 | |
*** paramite has joined #tripleo | 09:49 | |
*** bkopilov_ has quit IRC | 09:49 | |
*** Vijayendra has joined #tripleo | 09:54 | |
openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart: WIP: Update OOOQ to ansible 2.2 https://review.openstack.org/398194 | 09:55 |
*** chem has joined #tripleo | 09:57 | |
*** abregman|mtg is now known as abregman | 09:59 | |
*** derekjhyang has joined #tripleo | 09:59 | |
openstackgerrit | Christopher Brown proposed openstack/tripleo-docs: Remove WIP message https://review.openstack.org/404697 | 09:59 |
*** links has quit IRC | 10:05 | |
*** achadha has joined #tripleo | 10:09 | |
openstackgerrit | Alejandro Andreu proposed openstack/puppet-tripleo: Adds new tripleo::network::midonet::config class https://review.openstack.org/401342 | 10:12 |
*** milan has joined #tripleo | 10:13 | |
openstackgerrit | Alejandro Andreu proposed openstack/puppet-tripleo: Adds new tripleo::network::midonet::mem class https://review.openstack.org/401928 | 10:14 |
openstackgerrit | Alejandro Andreu proposed openstack/puppet-tripleo: Adds new tripleo::network::midonet::mem class https://review.openstack.org/401928 | 10:15 |
openstackgerrit | Alejandro Andreu proposed openstack/puppet-tripleo: Adds class tripleo::network::midonet::gateway https://review.openstack.org/401983 | 10:15 |
*** jlinkes_ has joined #tripleo | 10:16 | |
*** achadha has quit IRC | 10:16 | |
*** brault has joined #tripleo | 10:17 | |
*** andreaf has quit IRC | 10:17 | |
*** milan has quit IRC | 10:18 | |
*** andreaf has joined #tripleo | 10:18 | |
openstackgerrit | Alejandro Andreu proposed openstack/puppet-tripleo: Adds tripleo::network::midonet::insights class https://review.openstack.org/402062 | 10:19 |
*** larstobi has quit IRC | 10:19 | |
*** jlinkes has quit IRC | 10:19 | |
*** brault_ has quit IRC | 10:19 | |
*** rook has quit IRC | 10:19 | |
*** jaosorior_lunch has quit IRC | 10:19 | |
*** iranzo has quit IRC | 10:19 | |
*** ayoung has quit IRC | 10:19 | |
*** bnemec has quit IRC | 10:19 | |
*** soc_off has quit IRC | 10:19 | |
*** timothyb89 has quit IRC | 10:19 | |
*** soc_off_ has joined #tripleo | 10:19 | |
b00tcat | sorry, rebasing a lot ^^^ | 10:19 |
*** colonwq has quit IRC | 10:19 | |
*** bswartz has quit IRC | 10:19 | |
*** iranzo has joined #tripleo | 10:19 | |
*** iranzo has joined #tripleo | 10:20 | |
openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras: Expose the tempest results to the top level of the collected logs https://review.openstack.org/404710 | 10:20 |
*** timothyb89 has joined #tripleo | 10:20 | |
*** bnemec has joined #tripleo | 10:21 | |
*** links has joined #tripleo | 10:21 | |
*** limao_ has joined #tripleo | 10:25 | |
*** vkmc has quit IRC | 10:25 | |
*** limao has quit IRC | 10:25 | |
*** rook has joined #tripleo | 10:26 | |
*** rook is now known as Guest68279 | 10:26 | |
*** limao_ has quit IRC | 10:27 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 10:27 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: debugging - do not merge https://review.openstack.org/404560 | 10:27 |
*** vkmc has joined #tripleo | 10:27 | |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] overcloudrc command https://review.openstack.org/404354 | 10:28 |
*** larstobi has joined #tripleo | 10:30 | |
*** larstobi has quit IRC | 10:31 | |
*** zoli|trng is now known as zoli|lunch | 10:31 | |
*** jaosorior_lunch has joined #tripleo | 10:31 | |
*** colonwq has joined #tripleo | 10:32 | |
*** pcaruana has joined #tripleo | 10:35 | |
*** derekh has joined #tripleo | 10:36 | |
*** larstobi has joined #tripleo | 10:36 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 10:38 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: debugging - do not merge https://review.openstack.org/404560 | 10:38 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use command instead of env var for setting the hostname in the undercloud https://review.openstack.org/404717 | 10:38 |
*** jaosorior_lunch is now known as jaosorior | 10:41 | |
*** Vijayendra has quit IRC | 10:42 | |
*** Vijayendra has joined #tripleo | 10:43 | |
*** nyechiel has joined #tripleo | 10:45 | |
*** gfidente has joined #tripleo | 10:45 | |
*** tosky has joined #tripleo | 10:45 | |
*** apetrich has joined #tripleo | 10:48 | |
jprovazn | gfidente: hi | 10:49 |
openstackgerrit | yolanda.robla proposed openstack-infra/tripleo-ci: Add debug to image builds https://review.openstack.org/404723 | 10:55 |
*** tremble has joined #tripleo | 10:57 | |
*** tremble has joined #tripleo | 10:57 | |
*** udesale has quit IRC | 10:57 | |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Add validation parameters to overcloud deploy https://review.openstack.org/404726 | 11:03 |
*** links has quit IRC | 11:07 | |
*** links has joined #tripleo | 11:07 | |
flaper87 | shardy: so, I think I've tracked the issue down (the one I mentioend yday about not being able to fetch the contents of my env file). I think heatclient is having troubles with relative paths and they were not entirely fixed by f13d36ae0114e159663ea2a3fa07260df9d9407d :/ | 11:12 |
flaper87 | brb, lunch | 11:12 |
flaper87 | will keep poking later | 11:12 |
shardy | flaper87: ugh, OK thanks for the update | 11:16 |
shardy | flaper87: there's various nasty path mangling in tripleoclient which I'm hoping to remove in due course | 11:16 |
*** misty-cloud has quit IRC | 11:16 | |
shardy | flaper87: if you can ping me a reproducer I'll take a look later | 11:16 |
*** Vijayendra has quit IRC | 11:21 | |
*** sudipto has quit IRC | 11:23 | |
*** sudipto_ has quit IRC | 11:23 | |
*** flepied has joined #tripleo | 11:23 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Disable all repos during rhel registration https://review.openstack.org/381233 | 11:26 |
*** Vijayendra has joined #tripleo | 11:26 | |
*** nyechiel has quit IRC | 11:27 | |
*** tremble has quit IRC | 11:27 | |
jaosorior | bandini: hey dude, could you check this one out? https://review.openstack.org/#/c/403518/3 | 11:28 |
openstackgerrit | yolanda.robla proposed openstack/tripleo-quickstart: Allow to consume external overcloud images https://review.openstack.org/404734 | 11:31 |
*** tremble has joined #tripleo | 11:31 | |
*** tremble has quit IRC | 11:31 | |
*** tremble has joined #tripleo | 11:31 | |
openstackgerrit | Alejandro Andreu proposed openstack/puppet-tripleo: Adds tripleo::network::midonet::insights class https://review.openstack.org/402062 | 11:32 |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Set validate_image_dir relative to working dir https://review.openstack.org/404179 | 11:33 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Move the overcloudrc generation from tripleoclient to a Mistral action https://review.openstack.org/397211 | 11:34 |
*** thrash|g0ne is now known as thrash | 11:34 | |
*** jprovazn has quit IRC | 11:40 | |
yolanda | hi , derekh or other people that know ci... i'm trying to see failures on http://logs.openstack.org/51/401251/13/check/gate-tripleo-ci-centos-7-nonha-multinode/a54816b/console.html#_2016-11-30_11_26_16_421958 | 11:43 |
yolanda | i see error like http://logs.openstack.org/51/401251/13/check/gate-tripleo-ci-centos-7-nonha-multinode/a54816b/console.html#_2016-11-30_11_26_16_421958 | 11:43 |
yolanda | how shall these images be generated? | 11:43 |
yolanda | i just see code falls to "generate dummy overcloud image", that doesn't create those | 11:44 |
jaosorior | bandini: thanks dude | 11:45 |
yolanda | nah, i saw it's a red herring :( | 11:46 |
openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Enable cinder-backup service https://review.openstack.org/396587 | 11:47 |
bandini | jaosorior: for some reason I was 100% sure I reviewed that one yesterday | 11:48 |
jaosorior | bandini: it's cool, there were many CRs, sorry about that | 11:49 |
bandini | jaosorior: thank you for that work ;) | 11:49 |
*** zoli|lunch is now known as zoli|trng | 11:49 | |
yolanda | can i get help debugging failure on http://logs.openstack.org/51/401251/13/check/gate-tripleo-ci-centos-7-nonha-multinode/a54816b/console.html ? | 11:50 |
yolanda | just fails with 2016-11-30 11:29:55.234839 | 2016-11-30 11:29:47Z [overcloud.ControllerAllNodesValidationDeployment.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 | 11:51 |
yolanda | but i don't have a clue about where to find the failures | 11:51 |
jaosorior | yolanda: so, you gotta check the postci logs | 11:52 |
jaosorior | those are more explicit about the failure | 11:52 |
jaosorior | so one directory above http://logs.openstack.org/51/401251/13/check/gate-tripleo-ci-centos-7-nonha-multinode/a54816b/ | 11:52 |
jaosorior | then go to logs | 11:52 |
jaosorior | and the file is postci.txt.gz | 11:52 |
yolanda | oh | 11:52 |
yolanda | ok | 11:52 |
jaosorior | yolanda: http://logs.openstack.org/51/401251/13/check/gate-tripleo-ci-centos-7-nonha-multinode/a54816b/logs/postci.txt.gz | 11:52 |
jaosorior | yolanda: this is the issue http://logs.openstack.org/51/401251/13/check/gate-tripleo-ci-centos-7-nonha-multinode/a54816b/logs/postci.txt.gz#_2016-11-30_11_30_33_000 | 11:52 |
yolanda | ImportError: No module named ipaddr | 11:53 |
yolanda | weird... | 11:53 |
yolanda | i have not done any changes for that | 11:53 |
jaosorior | very weird indeed, seems that the image is missing packages | 11:53 |
*** misty-cloud has joined #tripleo | 11:53 | |
yolanda | i'm trying to test dib feature/v2 branch | 11:54 |
yolanda | but i thought it was not messing with packages | 11:54 |
jaosorior | yolanda: I'm sorry, I'm not sure what that means | 11:54 |
yolanda | diskimage-builder | 11:54 |
yolanda | there is a new feature branch, called v2 | 11:54 |
jaosorior | I meant the feature/v2 branch | 11:54 |
yolanda | and CI crashes on it | 11:54 |
jaosorior | oh, right | 11:54 |
openstackgerrit | Merged openstack/tripleo-quickstart-extras: Only clone DLRN in role if needed https://review.openstack.org/402046 | 11:54 |
jaosorior | thought it was a specific version of dib or something of the sort | 11:54 |
jaosorior | ok | 11:54 |
yolanda | i chained several patches to fix that, but this is a new problem | 11:54 |
yolanda | jaosorior, this job just uses a dummy image there, as far as i could see, right? | 11:55 |
yolanda | i saw it was falling on | 11:55 |
yolanda | # Create dummy overcloud-full image since there is no way (yet) to disable | 11:55 |
yolanda | # this constraint in the heat templates | 11:55 |
jaosorior | yolanda: it does, since it's a multinode job | 11:58 |
yolanda | jaosorior, where are those packages installed and the real image generated then? | 11:58 |
jaosorior | yolanda: I don't know | 11:58 |
yolanda | :( | 11:58 |
jaosorior | BUT | 11:58 |
jaosorior | lets see if we can figure it out | 11:58 |
yolanda | thanks for giving a hand there, i'm so lost | 11:59 |
* jaosorior is usually lost as well | 11:59 | |
jaosorior | lets see | 11:59 |
*** Vijayendra has quit IRC | 11:59 | |
yolanda | less than me :) | 11:59 |
jaosorior | yolanda: so, the multinode job will attempt to spawn the overcloud node(s) in subnodes | 12:00 |
jaosorior | you can usually find the logs in such a way | 12:00 |
jaosorior | for instance, for this job the multinode deployment is an all-in-one node that contains the overcloud services and the compute as well | 12:01 |
jaosorior | that node is usually subnode-2 | 12:01 |
jaosorior | here are the logs http://logs.openstack.org/51/401251/13/check/gate-tripleo-ci-centos-7-nonha-multinode/a54816b/logs/subnode-2/ | 12:01 |
*** dsneddon has quit IRC | 12:01 | |
yolanda | but there are no image builds there as well | 12:01 |
jaosorior | yolanda: do you know what package provides the ipaddr python module? | 12:01 |
jaosorior | yolanda: I think the only jobs that are gonna get the image builds reflected are the ovb jobs | 12:02 |
yolanda | python-ipaddr | 12:02 |
yolanda | ovb logs are super useful :) | 12:03 |
yolanda | http://logs.openstack.org/61/404161/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/6b17776/console.html#_2016-11-30_11_51_09_790644 | 12:03 |
yolanda | no clue about why dib fails there | 12:03 |
jaosorior | dammit | 12:04 |
yolanda | i added a DIB_DEBUG_TRACE to see if i get more info, but seems the wrapper is just masking output from diskimage-builder | 12:04 |
jaosorior | the logs are not very explicit http://logs.openstack.org/61/404161/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/6b17776/logs/undercloud/var/log/image_build.txt.gz | 12:04 |
jaosorior | yolanda: where did you add DIB_DEBUG_TRACE ? | 12:04 |
yolanda | https://review.openstack.org/#/c/404723/ | 12:05 |
yolanda | not sure if that's going to take effect | 12:05 |
jaosorior | not sure either | 12:05 |
yolanda | i rechecked to get that output again. But i think that wrapper has a problem, was nested in some python subprocess call, just capturing the exit code and not the output itself | 12:07 |
*** jtomasek has joined #tripleo | 12:07 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 12:07 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: debugging - do not merge https://review.openstack.org/404560 | 12:07 |
openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Add Zaqar to scenario002 https://review.openstack.org/404737 | 12:08 |
openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart: Update OOOQ to ansible 2.2 https://review.openstack.org/398194 | 12:10 |
*** adarazs is now known as adarazs_lunch | 12:12 | |
flaper87 | shardy: tbh, the reproducer so far has been just running overcloud deploy with latest t-h-t and heatclient. It concatenates the base_path with the template path, which is a problem when the template path has '..' in it | 12:12 |
jaosorior | yolanda: I think you're right, the wrapper disregards the actual environment variables | 12:12 |
yolanda | jaosorior, what do you recomend to debug that? | 12:13 |
jaosorior | yolanda: this is where they're set https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_image.py#L449 | 12:13 |
jaosorior | yolanda: I recommend we fix the wrapper | 12:13 |
jaosorior | lets see | 12:13 |
yolanda | even without DIB_DEBUG_TRACE, but a proper capture of the output, it will give some hints | 12:14 |
flaper87 | shardy: it might very well be that it's failing at getting the template base ur; | 12:14 |
*** achadha has joined #tripleo | 12:14 | |
jaosorior | thrash: are you around? | 12:14 |
thrash | jaosorior: yah. wassup? | 12:14 |
jaosorior | thrash: you did some work with the tripleoclient, on the dib image wrapper, right? | 12:15 |
thrash | yes. | 12:15 |
thrash | the yaml one? | 12:15 |
thrash | in general too. | 12:16 |
jaosorior | thrash: I'm not entirely sure which are we using in CI | 12:16 |
jaosorior | thrash: but the deal is that we need more output from it | 12:16 |
thrash | I'm pretty sure it's the yaml version. | 12:16 |
thrash | from... what? dib? | 12:16 |
jaosorior | thrash: yolanda is trying to set DIB_DEBUG_TRACE but that's not being taken into account | 12:16 |
jaosorior | thrash: and it seems that the dib output is being discarded | 12:16 |
thrash | jaosorior: it would probably need to be set in the yaml itself. | 12:17 |
thrash | we pass a limited environment to dib. | 12:17 |
*** slagle has joined #tripleo | 12:17 | |
thrash | jaosorior: you should be able to just add another yaml file to the cli | 12:18 |
thrash | jaosorior: one sec, I'll show you where... | 12:18 |
thrash | jaosorior: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L134 | 12:18 |
thrash | those args... | 12:18 |
thrash | you could add '--config-file my_debug.yaml' | 12:19 |
yolanda | thrash, jaosorior , i could even ignore the DIB_DEBUG_TRACE flag, but i'd like to capture the real output of diskimage-builder | 12:19 |
*** misty-cloud has quit IRC | 12:19 | |
yolanda | because it fails silently on CI | 12:19 |
*** jtomasek has quit IRC | 12:19 | |
yolanda | thrash, look http://logs.openstack.org/61/404161/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/6b17776/console.html#_2016-11-30_11_51_09_790644 | 12:19 |
yolanda | so you can see "returned non-zero exit status 1", but you cannot see the real failure | 12:19 |
*** achadha has quit IRC | 12:19 | |
thrash | yolanda: is that happening everywhere? Or just on this job? | 12:20 |
thrash | yolanda: and that patch? | 12:20 |
*** nyechiel has joined #tripleo | 12:20 | |
yolanda | thrash, i've seen that on ovb jobs | 12:20 |
yolanda | but the other jobs are not calling disk-image-create directly | 12:20 |
thrash | yolanda: ok... let me put together a test patch. One sec. | 12:21 |
yolanda | thanks | 12:21 |
jaosorior | thrash: also, one thing, is there a reason why we're using shell=True here https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_image.py#L87 ? | 12:21 |
thrash | jaosorior: that's not even used anymore. | 12:22 |
* jaosorior has no knowledge about the image building side | 12:22 | |
thrash | jaosorior: real call is here: https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/image_builder.py#L95 | 12:22 |
flaper87 | shardy: so, it gets the env_base_url for a temp env file:///tmp/tripleoclient-bkEwZT/tripleo-heat-templates/env-docker-QRcd81.yaml | 12:23 |
flaper87 | not sure where that file is coming from but I'll figure it out | 12:24 |
*** limao has joined #tripleo | 12:24 | |
*** flepied has quit IRC | 12:25 | |
openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Add Mistral to scenario003 https://review.openstack.org/404743 | 12:25 |
*** mburned_out is now known as mburned | 12:26 | |
*** nyechiel has quit IRC | 12:26 | |
*** prateek has quit IRC | 12:28 | |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Add validation parameters to overcloud deploy https://review.openstack.org/404726 | 12:28 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add verify required and CA bundle to haproxy https://review.openstack.org/399591 | 12:28 |
*** nyechiel has joined #tripleo | 12:29 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: debugging - do not merge https://review.openstack.org/404560 | 12:29 |
*** pkovar has quit IRC | 12:32 | |
*** links has quit IRC | 12:34 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add verify required and CA bundle to haproxy https://review.openstack.org/399591 | 12:34 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add verify required and CA bundle to haproxy https://review.openstack.org/399591 | 12:34 |
*** jtomasek has joined #tripleo | 12:35 | |
EmilienM | hi | 12:36 |
jaosorior | EmilienM: sup dude | 12:36 |
thrash | jaosorior: yolanda oh frick... y'all look at overcloud-image-build.log? | 12:37 |
*** jrist has joined #tripleo | 12:37 | |
*** jrist has joined #tripleo | 12:37 | |
jaosorior | thrash: well, had checked this o ut http://logs.openstack.org/61/404161/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/6b17776/logs/undercloud/var/log/image_build.txt.gz | 12:37 |
*** dsneddon has joined #tripleo | 12:38 | |
thrash | jaosorior: we must not be capturing it... https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L524 | 12:39 |
thrash | hmmm | 12:39 |
yolanda | thrash, yep, image-build just gives that empty output | 12:39 |
thrash | maybe that doesn't have enough... With the tee and all. :) | 12:39 |
thrash | Ok.. Back to what I was getting ready for debug. :) | 12:39 |
EmilienM | thrash: can you review https://review.openstack.org/#/c/404223/ please? | 12:40 |
thrash | EmilienM: ack | 12:40 |
*** bfournie has quit IRC | 12:40 | |
jaosorior | thrash: so... where is this overcloud-image-build.log located usually? in the home dir? | 12:40 |
thrash | jaosorior: that would be my guess... | 12:41 |
yolanda | thrash, looks as that file is not copied to logs | 12:41 |
*** flepied has joined #tripleo | 12:42 | |
*** dsneddon has quit IRC | 12:42 | |
*** jprovazn has joined #tripleo | 12:43 | |
*** jkilpatr has joined #tripleo | 12:44 | |
*** dsariel has quit IRC | 12:44 | |
shardy | flaper87: sounds like there's a problem here https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L173 | 12:44 |
*** arxcruz has quit IRC | 12:44 | |
shardy | flaper87: that's where we rewrite the user environments to mangle paths to match the tmpdir | 12:45 |
flaper87 | shardy: yes, I think I just found it: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L179 this should be abs_rsr_path | 12:45 |
openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: tempest tests https://review.openstack.org/404345 | 12:45 |
shardy | flaper87: aha, good catch! :) | 12:45 |
*** ola has joined #tripleo | 12:45 | |
shardy | flaper87: apologies - I think that code is my fault, it was a late workaround for some blocker issues at newton GA - probably needs more/better tests | 12:46 |
*** ola is now known as Guest49746 | 12:46 | |
shardy | I hope we can clean that entire mess up this cycle by moving the file resolution logic into heat | 12:46 |
flaper87 | shardy: no worries, just glad I found it, it was becoming too frustrating hahahaha | 12:46 |
shardy | ramishra has started looking into that | 12:46 |
flaper87 | I think there's one more fix to do, though. still not working perfectly | 12:46 |
shardy | flaper87: we might want to add some different corner-case environment files to CI, even if they don't do anything - e.g relative/absolute paths etc | 12:47 |
EmilienM | FYI, we updated puppet version in tripleo, we are now deploying Puppet 4.6 thanks to RDO packaging | 12:49 |
EmilienM | we fixed a last minute bug yesterday but everything should be fine now. Please let me know if you see some strange puppet things today :P | 12:49 |
*** amoralej is now known as amoralej|lunch | 12:50 | |
*** Guest49746 has quit IRC | 12:50 | |
*** ola_ has joined #tripleo | 12:50 | |
openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: TEST: DO NOT MERGE - Verbose output for image building https://review.openstack.org/404751 | 12:51 |
thrash | yolanda: jaosorior ^^^^ | 12:51 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Include logs in the undercloud's home directory https://review.openstack.org/404752 | 12:52 |
jaosorior | thrash: trying to include the logs ^^ seems they're missing | 12:52 |
jaosorior | EmilienM: which bug was it? | 12:52 |
thrash | jaosorior: ack | 12:53 |
*** kaslcrof has joined #tripleo | 12:53 | |
EmilienM | jaosorior: https://review.openstack.org/#/c/404437/ | 12:53 |
EmilienM | jaosorior: took me 3 hours to find it :) | 12:53 |
jaosorior | EmilienM: wow, that's a tricky one | 12:53 |
*** lucasagomes is now known as lucas-hungry | 12:54 | |
*** panda|off is now known as panda | 12:55 | |
pradk | can i get another review on this please https://review.openstack.org/#/c/403766/ | 12:55 |
*** milan has joined #tripleo | 12:55 | |
*** jayg|g0n3 is now known as jayg | 12:56 | |
flaper87 | shardy: nvm, the other problem was my fault. Using abs_rsr_path does fix the issue | 12:58 |
shardy | flaper87: good news! :) | 12:58 |
pradk | thx jaosorior ! | 12:58 |
*** arxcruz has joined #tripleo | 12:58 | |
flaper87 | shardy: will write a test and push a patch | 12:59 |
shardy | flaper87: ack, sounds good - thanks for finding the problem | 13:00 |
flaper87 | shardy: it was either that or throwing my laptop out of the window | 13:01 |
shardy | lol :) | 13:01 |
*** athomas has joined #tripleo | 13:01 | |
*** noslzzp has joined #tripleo | 13:03 | |
*** jcoufal has joined #tripleo | 13:03 | |
*** trown|outtypewww is now known as trown | 13:05 | |
gfidente | shardy so I managed to get the parameters passed to a mistral workflow | 13:05 |
gfidente | (heat parameters) | 13:05 |
gfidente | but not to trigger the workflow execution | 13:05 |
gfidente | I think I can get the alarm_url | 13:05 |
gfidente | but can I GET it from a heat resource? | 13:05 |
*** dmacpher has joined #tripleo | 13:06 | |
*** adarazs_lunch is now known as adarazs | 13:06 | |
*** athomas has quit IRC | 13:06 | |
ccamacho | hey matbu, arounf | 13:06 |
ccamacho | around? | 13:06 |
matbu | ccamacho: yep | 13:06 |
ccamacho | quick question, im hitting http://paste.openstack.org/show/590953/ | 13:06 |
ccamacho | when upgrading newton to master | 13:07 |
ccamacho | there is a bug related | 13:07 |
*** pkovar has joined #tripleo | 13:07 | |
ccamacho | https://github.com/ibmdb/python-ibmdb/issues/166 | 13:07 |
*** bkopilov has joined #tripleo | 13:08 | |
matbu | ccamacho: hmm i didn't see that yet | 13:08 |
matbu | ccamacho: looks like the issue is closed | 13:09 |
ccamacho | yeahp, im trying to get from yum logs if alembic was updated | 13:09 |
ccamacho | and exclude it from the update | 13:09 |
ccamacho | to see if makes some difference | 13:10 |
ccamacho | matbu this is hard dude :P | 13:10 |
matbu | ccamacho: hehe | 13:10 |
matbu | ccamacho: i redeployed this mornign, cause my env was totally broken | 13:10 |
ccamacho | so after I make that stuff work ill move the pcm resources to your patch :) | 13:10 |
matbu | ccamacho: but i'll check the package version | 13:10 |
openstackgerrit | Flavio Percoco proposed openstack/python-tripleoclient: e the absolute path for temp templates https://review.openstack.org/404762 | 13:11 |
ccamacho | to see if we can make it :) | 13:11 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: scenario001: deploy Ceph https://review.openstack.org/404462 | 13:11 |
matbu | ccamacho: ack, did you check the current upgrade script ? | 13:11 |
openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart-extras: gate-quickstart: force repo checkout https://review.openstack.org/404230 | 13:12 |
*** jkilpatr has quit IRC | 13:12 | |
*** jtomasek has quit IRC | 13:12 | |
ccamacho | matbu yeahp the issue raise when executing https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/tasks/major_upgrade_controller_pacemaker_3.sh#L63 (From Ansible) | 13:12 |
matbu | ccamacho: it could be nice also to share an etherpad with notes and manual actions (or modification made in script) | 13:12 |
ccamacho | yeahp yeahp Im creating notes to share about "how to test it" | 13:13 |
matbu | ccamacho: k, so currently, you push and execute the upgrade script ? | 13:13 |
openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras: Enable cinder-backup service https://review.openstack.org/400787 | 13:13 |
*** dbecker has joined #tripleo | 13:14 | |
*** bfournie has joined #tripleo | 13:14 | |
ccamacho | im pushing the scripts with ansible (from shardy submission), but commenting it here (https://review.openstack.org/#/c/393448/15/puppet/services/neutron-api.yaml) for not executing it twice. | 13:14 |
ccamacho | matbu^ | 13:15 |
ccamacho | there are some duplicated bit that we need to extract | 13:15 |
openstackgerrit | Merged openstack/tripleo-quickstart-extras: gate-quickstart: force repo checkout https://review.openstack.org/404230 | 13:15 |
*** dsariel has joined #tripleo | 13:16 | |
matbu | ccamacho: ok, i think most the upgrade script are really specific to L to M and M to N | 13:16 |
matbu | ccamacho: im not sure we need to reuse all | 13:16 |
ccamacho | mmm i see | 13:17 |
*** ola_ has quit IRC | 13:17 | |
matbu | ccamacho: there is many workaround or speicific migration cases | 13:17 |
matbu | but maybe we will have to deal with new cases | 13:17 |
matbu | ccamacho: for now i just make it "stupid", stop pcs, update services, update packages, start the cluster, check the status and make db sync | 13:18 |
*** sudipto_ has joined #tripleo | 13:18 | |
*** sudipto has joined #tripleo | 13:18 | |
ccamacho | yeahp, but what im trying to get is the smallest HA deployment upgraded N -> O (master), and from that point we can add more cases | 13:19 |
*** athomas has joined #tripleo | 13:19 | |
*** jkilpatr has joined #tripleo | 13:19 | |
yolanda | jaosorior, added a comment | 13:20 |
openstackgerrit | Flavio Percoco proposed openstack/python-tripleoclient: Use the absolute path for temp templates https://review.openstack.org/404762 | 13:21 |
*** pgadiya has quit IRC | 13:25 | |
*** fultonj has joined #tripleo | 13:28 | |
jaosorior | yolanda: I used /home/stack since that's the usual user we run the overcloud parts with.... and I'm not sure what user is logging in there to retrieve those logs (if it's root it wouldn't work) | 13:29 |
*** masco has quit IRC | 13:29 | |
*** karthiks has quit IRC | 13:29 | |
*** dtantsur|afk is now known as dtantsur | 13:29 | |
*** rlandy has joined #tripleo | 13:30 | |
yolanda | it may be a safe bet, i remember trying to change the /home/stack directory on undercloud at some point, and was failing totally | 13:31 |
jaosorior | EmilienM: weren't we gonna put ceph in scenario004? | 13:32 |
EmilienM | jaosorior: scenario001 and 4 | 13:32 |
jaosorior | ok | 13:32 |
EmilienM | jaosorior: scenario001 for classic RBD scenario, and 4 for Rados Gateway | 13:33 |
*** links has joined #tripleo | 13:36 | |
*** links has quit IRC | 13:36 | |
*** achadha has joined #tripleo | 13:37 | |
*** dtrainor has joined #tripleo | 13:37 | |
ansiwen | EmilienM: "Could not find class ::ec2api::db::mysql for overcloud-controller-0.localdomain" http://logs.openstack.org/34/398634/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d6a61e8/logs/postci.txt.gz#_2016-11-29_15_05_26_000 | 13:38 |
ansiwen | EmilienM: what could be the reason for that? I have it included in puppet-tripleo-elements | 13:38 |
EmilienM | ansiwen: https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L148 | 13:39 |
EmilienM | I don't see puppet-ec2api | 13:39 |
*** lblanchard has joined #tripleo | 13:39 | |
EmilienM | ansiwen: tripleo-puppet-elements are not used anymore for puppet modules | 13:39 |
EmilienM | ansiwen: it's in RDO packaging | 13:39 |
ansiwen | EmilienM: oh dear... obviously I didn't hear the news... | 13:39 |
ansiwen | EmilienM: thanks! | 13:39 |
jaosorior | EmilienM: could you check these out? https://review.openstack.org/#/q/status:open+project:openstack/tripleo-heat-templates+branch:master+topic:rabbit-hostnames | 13:40 |
EmilienM | jaosorior: looking | 13:41 |
*** achadha has quit IRC | 13:41 | |
ansiwen | EmilienM: but why does it work in CI then? is there a different mechanism? | 13:43 |
*** michchap_alt has joined #tripleo | 13:43 | |
*** jcoufal_ has joined #tripleo | 13:44 | |
EmilienM | ansiwen: ec2api works in tripleo CI? | 13:45 |
*** jeckersb is now known as jeckersb_gone | 13:45 | |
EmilienM | ansiwen: where do you see the service running? | 13:45 |
*** karthiks has joined #tripleo | 13:45 | |
ansiwen | EmilienM: no, in openstack puppet ci | 13:45 |
ansiwen | EmilienM: we deplay it there and have the tempest run | 13:45 |
EmilienM | ansiwen: puppet openstack ci doesn't deploy puppet modules with RPM but with r10k | 13:46 |
EmilienM | ansiwen: https://github.com/openstack/puppet-openstack-integration/blob/master/Puppetfile | 13:46 |
ansiwen | EmilienM: I see... | 13:46 |
*** jtomasek has joined #tripleo | 13:47 | |
*** jcoufal has quit IRC | 13:47 | |
*** oshvartz has quit IRC | 13:47 | |
*** jpena is now known as jpena|lunch | 13:48 | |
*** rbrady-afk is now known as rbrady | 13:48 | |
yolanda | hi jaosorior , looking at your patch results. I guess we won't see anything until ovb jobs finish? | 13:49 |
*** lucas-hungry is now known as lucasagomes | 13:51 | |
ansiwen | EmilienM: and why does puppet-panko not appear in that file? I thought it has been merged already? | 13:51 |
EmilienM | ansiwen: https://github.com/openstack/puppet-openstack-integration/blob/master/Puppetfile#L108-L110 | 13:51 |
EmilienM | it is in that file | 13:51 |
jaosorior | yolanda: correct | 13:51 |
ansiwen | EmilienM: no, I mean in the rdo.yml | 13:52 |
EmilienM | ansiwen: because we need to add it | 13:52 |
EmilienM | pradk: ^ fyi | 13:52 |
EmilienM | pradk: if you want panko deployed in tripleo, we need to add puppet-panko in RDO | 13:52 |
ansiwen | EmilienM: I see ^_^ | 13:53 |
EmilienM | ansiwen: easy right? (lol) | 13:53 |
pradk | ah thought i did that with panko.. will do now | 13:54 |
ansiwen | EmilienM, pradk, so come puppet-projects are just with a "conf: rpmfactory-puppet" element, some have upstream: and tags: elements. What do I have to do for ec2-api? | 13:56 |
EmilienM | ansiwen: better to ask on #rdo what is the best | 13:56 |
ansiwen | EmilienM: I'll do, thanks | 13:56 |
ansiwen | pradk: while I'm at it I can also add panko | 13:57 |
pradk | https://review.rdoproject.org/r/3939 | 13:57 |
pradk | ansiwen, i just pushed | 13:57 |
pradk | thx though | 13:57 |
ansiwen | pradk: so, that's it? nothing else needed? | 13:58 |
*** rodrigods is now known as rodrigods_ | 13:58 | |
openstackgerrit | Lucas Alvares Gomes proposed openstack/tripleo-quickstart: VirtualBMC support for tripleo-quickstart https://review.openstack.org/399704 | 13:58 |
ansiwen | pradk: no "upatream:" or so? | 13:58 |
*** hrybacki is now known as hrybacki|mtg | 14:00 | |
*** numans has joined #tripleo | 14:00 | |
openstackgerrit | Babu Shanmugam proposed openstack/tripleo-heat-templates: OVN bridge mappings for tripleo https://review.openstack.org/402293 | 14:01 |
openstackgerrit | Babu Shanmugam proposed openstack/tripleo-heat-templates: OVN plugin configuration fixes https://review.openstack.org/397674 | 14:01 |
*** karthiks has quit IRC | 14:02 | |
*** ealcaniz has quit IRC | 14:02 | |
*** abregman is now known as abregman|mtg | 14:02 | |
slagle | shardy: hey, one of the things i'm trying to do for the split stack phase 2 work is move away from having to use admin creds on the deployed servers | 14:03 |
slagle | shardy: i've switched the software config transport to be POLL_SERVER_HEAT, but i'm still not sure how to get the generated keystone creds out of Heat | 14:04 |
*** ola has joined #tripleo | 14:04 | |
*** ola is now known as Guest63228 | 14:04 | |
*** pkovar has quit IRC | 14:04 | |
shardy | slagle: it's the same as with the CFN credentials, e.g you get them from the metadata for the dummy server after the stack is created | 14:06 |
shardy | https://etherpad.openstack.org/p/noop-softwareconfig | 14:07 |
shardy | slagle: the only difference is we put a username and randomly generated password (and the heat/keystone endpoint) in there instead of the CFN URL | 14:07 |
shardy | slagle: the other option is to use the swift transport (which is what we use by default now) then you get a tempurl | 14:08 |
shardy | still need to create the stack before the credentials exist tho | 14:08 |
shardy | bbiab, appointment | 14:09 |
slagle | shardy: for some reason resource metadata is empty in my stack | 14:09 |
shardy | slagle: make sure it's the metadata for the dummy resource, not the dummy stack | 14:09 |
shardy | slagle: sorry, will ping you in a bit, need to drop off for a few mins | 14:09 |
*** shardy has quit IRC | 14:09 | |
thrash | jaosorior: yolanda think I found a bug in the environment extension. Fixing it. | 14:10 |
*** Guest63228 has quit IRC | 14:10 | |
*** Vijayendra has joined #tripleo | 14:11 | |
*** amoralej|lunch is now known as amoralej | 14:11 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 14:12 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: debugging - do not merge https://review.openstack.org/404560 | 14:12 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use command instead of env var for setting the hostname in the undercloud https://review.openstack.org/404717 | 14:12 |
*** Goneri has joined #tripleo | 14:14 | |
*** kjw3 has joined #tripleo | 14:18 | |
*** karthiks has joined #tripleo | 14:19 | |
*** bswartz has joined #tripleo | 14:19 | |
*** ohamada has joined #tripleo | 14:20 | |
d0ugal | panda: btw, the backport has been released https://pypi.python.org/pypi/python-mistralclient/2.1.2 | 14:21 |
panda | d0ugal: \o/ | 14:21 |
panda | d0ugal: thanks. | 14:22 |
openstackgerrit | Lucas Alvares Gomes proposed openstack/tripleo-quickstart: VirtualBMC support for tripleo-quickstart https://review.openstack.org/399704 | 14:22 |
openstackgerrit | Merged openstack/tripleo-quickstart-extras: Set validate_image_dir relative to working dir https://review.openstack.org/404179 | 14:22 |
d0ugal | panda: so I guess something can be reverted now? | 14:23 |
*** [1]cdearborn has joined #tripleo | 14:24 | |
*** pradk has quit IRC | 14:24 | |
*** dsneddon has joined #tripleo | 14:26 | |
*** rhallisey has joined #tripleo | 14:26 | |
*** pkovar has joined #tripleo | 14:29 | |
*** ayoung has joined #tripleo | 14:29 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: deployment updates for containerized compute https://review.openstack.org/400986 | 14:30 |
openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: add the containers prep role to the quickstart-extras https://review.openstack.org/400983 | 14:30 |
*** rodrigods_ is now known as rodrigods | 14:31 | |
*** dsneddon has quit IRC | 14:31 | |
*** sudipto_ has quit IRC | 14:35 | |
*** sudipto has quit IRC | 14:35 | |
yolanda | jaosorior, i'm looking at some ovb logs for your job, for example http://logs.openstack.org/52/404752/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha-mitaka/0eb3ec6/ | 14:37 |
yolanda | and i don't see image build logs there as well | 14:37 |
*** jeckersb_gone is now known as jeckersb | 14:38 | |
*** yprokule_ has joined #tripleo | 14:38 | |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Replace hardcoded stack user by ansible_user https://review.openstack.org/404800 | 14:39 |
*** yprokule has quit IRC | 14:39 | |
*** yprokule_ is now known as yprokule | 14:39 | |
jaosorior | yolanda: not sure if they'll be available for mitaka | 14:41 |
jaosorior | yolanda: we gotta wait for the newton/master jobs | 14:41 |
yolanda | agh | 14:41 |
yolanda | i was too impatient | 14:41 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 14:43 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: debugging - do not merge https://review.openstack.org/404560 | 14:43 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use command instead of env var for setting the hostname in the undercloud https://review.openstack.org/404717 | 14:43 |
*** jpena|lunch is now known as jpena | 14:44 | |
thrash | yolanda: got a bug I'm fixing | 14:46 |
d0ugal | thrash: https://github.com/openstack/tripleo-common/blob/master/workbooks/baremetal.yaml#L40-L46 | 14:46 |
d0ugal | thrash: that waiting loop may be useful for you | 14:46 |
thrash | d0ugal: cool. I'll look at that. | 14:46 |
*** tremble has quit IRC | 14:48 | |
*** tremble has joined #tripleo | 14:51 | |
*** kjw3 has quit IRC | 14:52 | |
*** chlong has joined #tripleo | 14:54 | |
*** ohamada has quit IRC | 14:56 | |
*** ohamada has joined #tripleo | 14:56 | |
*** jaosorior has quit IRC | 14:57 | |
*** saneax is now known as saneax-_-|AFK | 14:58 | |
*** achadha has joined #tripleo | 14:59 | |
*** prateek has joined #tripleo | 14:59 | |
*** jaosorior has joined #tripleo | 14:59 | |
*** udesale has joined #tripleo | 15:00 | |
yolanda | jaosorior, i am looking at http://logs.openstack.org/52/404752/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d9c1851/ | 15:04 |
yolanda | and no logs | 15:04 |
*** achadha has quit IRC | 15:04 | |
*** hrybacki|mtg is now known as hrybacki | 15:04 | |
*** shardy has joined #tripleo | 15:05 | |
yolanda | ah.. may be on the tar, sorry i overlooked | 15:05 |
yolanda | mm no, the image build is for the undercloud | 15:06 |
jaosorior | no logs indeed | 15:06 |
*** Guest69699 has quit IRC | 15:06 | |
*** Guest69699 has joined #tripleo | 15:06 | |
jaosorior | dammit | 15:06 |
yolanda | they may be in another place, not home/stack? | 15:08 |
jaosorior | sshnaidm, panda hey guys, any idea where I could find the logs relevant to the image build? | 15:09 |
shardy | slagle: back, did you find your missing metadata? | 15:09 |
slagle | shardy: yes! | 15:09 |
slagle | shardy: your last comment helped me :) | 15:09 |
shardy | slagle: cool, sorry I had to drop off abruptly | 15:09 |
sshnaidm | jaosorior, I think we have problem with it now.. | 15:10 |
slagle | i had to go one nested stack deeper then i do when using the cfn transport | 15:10 |
slagle | at least, that's where the metadata was | 15:10 |
*** prateek has quit IRC | 15:10 | |
shardy | slagle: Hmm, it should be in the same place, has anything about how you define the dummy server also changed? | 15:10 |
shardy | we always update the metadata for the "server" resource, only in this case it's not an actual server | 15:11 |
sshnaidm | jaosorior, it was before in: logs/undercloud/var/log/image_build.txt.gz | 15:11 |
shardy | still, good that you found it | 15:11 |
yolanda | sshnaidm, but these logs are not really useful | 15:11 |
yolanda | they just say diskimage builder failed, but without any real output | 15:11 |
sshnaidm | yolanda, exactly | 15:11 |
sshnaidm | it happened after some last changes | 15:11 |
yolanda | ugh | 15:12 |
slagle | shardy: everyting else is the same... | 15:12 |
slagle | shardy: i just wasn't looking deep enough | 15:12 |
sshnaidm | yolanda, jaosorior I think this file is not collected in logs, let me fix it: https://github.com/openstack-infra/tripleo-ci/blob/2159b2fc5166c5f934ad5b51fce1af1f40f2acda/scripts/tripleo.sh#L524-L524 | 15:14 |
shardy | slagle: ack, cool well good to hear you got past it | 15:14 |
yolanda | sshnaidm, yep, i think that this was the intention of jaosorior patch | 15:15 |
yolanda | but is not on /home/stack? | 15:15 |
sshnaidm | yolanda, but anyway, it has "tee -a" so all output should have went to image_build.txt.. | 15:15 |
slagle | shardy: yea, i got the username/password now, but os-collect-config is still getting an unauthorized | 15:15 |
slagle | shardy: i'm looking into that now | 15:16 |
openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Don't rely on overcloudrc https://review.openstack.org/400800 | 15:16 |
slagle | ERROR: You are not authorized to use this action. | 15:16 |
*** anshul has quit IRC | 15:16 | |
sshnaidm | yolanda, in upstream ci we collect only /var/log, $workspace/logs and /etc. | 15:16 |
*** ealcaniz has joined #tripleo | 15:17 | |
*** sudipto has joined #tripleo | 15:17 | |
*** sudipto_ has joined #tripleo | 15:17 | |
slagle | i updated the project_id as well. hmm | 15:17 |
yolanda | sshnaidm, but look at jaosorior patch https://review.openstack.org/#/c/404734/1/roles/libvirt/setup/undercloud/tasks/main.yml | 15:17 |
yolanda | oh, bad link | 15:17 |
yolanda | https://review.openstack.org/#/c/404752/1/scripts/common_functions.sh | 15:17 |
yolanda | it shall have been collected on undercloud.tar.xz... | 15:18 |
sshnaidm | yolanda, it should be jenkins, not stack | 15:19 |
*** mburned is now known as mburned_out | 15:19 | |
*** dprince has joined #tripleo | 15:21 | |
yolanda | ah, that explains | 15:21 |
yolanda | jaosorior ^ | 15:21 |
sshnaidm | yolanda, although we have "tee" there, so it should be in image_builder.txt too, I'm afraid it may be something in additional args for tripleoclient (just guess) | 15:22 |
ccamacho | matbu now getting this after the nova upgrade "ConfigFileValueError: Value for option osapi_compute_workers is not valid: Should be greater than or equal to 1" :P I remember this issue from before.. Now hitting it with the upgrade | 15:23 |
matbu | ccamacho: hehe welcome in upgrade :) | 15:24 |
*** jeckersb is now known as jeckersb_gone | 15:25 | |
*** pradk has joined #tripleo | 15:27 | |
*** dhill__ has quit IRC | 15:27 | |
*** dhill_ has joined #tripleo | 15:27 | |
*** limao has quit IRC | 15:27 | |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: WIP: Prepare build-test role for usage in upstream https://review.openstack.org/404823 | 15:28 |
tbarron | jprovazn: as we discussed, https://bugs.launchpad.net/tripleo/+bug/1646147 is all yours | 15:28 |
openstack | Launchpad bug 1646147 in tripleo "deploy of manila with external ceph fails to update ceph.conf" [Undecided,New] | 15:28 |
*** jeckersb_gone is now known as jeckersb | 15:28 | |
*** Guest69699 is now known as assassin | 15:29 | |
*** liverpooler has joined #tripleo | 15:30 | |
*** jlinkes_ has quit IRC | 15:30 | |
*** jlinkes_ has joined #tripleo | 15:30 | |
*** ayoung has quit IRC | 15:30 | |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 15:30 |
openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: updates: enable ssl https://review.openstack.org/402930 | 15:32 |
openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: updates: fix condition that adds correct DEPLOY_ARGS https://review.openstack.org/403744 | 15:32 |
matbu | EmilienM: ^ first draft | 15:32 |
jprovazn | tbarron, ack, thanks for checking | 15:32 |
matbu | EmilienM: and the switch for project-config: https://review.openstack.org/404812 | 15:32 |
fpan | dprince: ping | 15:33 |
EmilienM | matbu: commented | 15:33 |
EmilienM | matbu: you're super fast :D | 15:33 |
*** pblaho has quit IRC | 15:34 | |
fpan | dprince: would it be ok to backport ping_retry function to stable/mitaka, https://review.openstack.org/#/c/302386/ | 15:34 |
*** ealcaniz has quit IRC | 15:34 | |
EmilienM | matbu: both patches are commented | 15:34 |
matbu | EmilienM: hehe thanks too fast i forgot mysql :) | 15:34 |
*** pblaho has joined #tripleo | 15:34 | |
dprince | fpan: fine with me | 15:35 |
gfidente | tbarron you need to enable the external ceph support too | 15:35 |
gfidente | can I comment on the bug? | 15:35 |
thrash | dprince: on the paths... i'm really just following the lead of others... https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario003-multinode.yaml#L4-L5 | 15:35 |
thrash | dprince: there a reason we shouldn't be? | 15:35 |
*** abregman|mtg is now known as abregman | 15:36 | |
fpan | dprince: cool, thanks. seeing some deployment failures with mitaka because of gw ping | 15:36 |
openstackgerrit | Feng Pan proposed openstack/tripleo-heat-templates: Add ping_retry function https://review.openstack.org/404835 | 15:36 |
dprince | thrash: well, I never execute heat templates from my undercloud | 15:36 |
dprince | thrash: so that patch doesn't exist for me | 15:36 |
dprince | thrash: just means I have to customize those environments if I ever wanted to use them | 15:37 |
thrash | dprince: I don't have a good argument the other way, TBH. | 15:37 |
thrash | :D | 15:37 |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 15:38 |
thrash | So I'm fine with changing it to be relative. | 15:38 |
matbu | EmilienM: ok, updated both, we need to merged project-config before triggering the experimental i guess ? | 15:38 |
EmilienM | matbu: yes | 15:38 |
thrash | dprince: and yet you +2 this: https://review.openstack.org/#/c/404743/ :D | 15:39 |
thrash | lol | 15:39 |
dprince | thrash: yeah, that one slipped through the cracks man :) | 15:39 |
matbu | EmilienM: k, i'm curious to see it running | 15:39 |
thrash | hehe | 15:39 |
thrash | dprince: I'll change the zaqar one. | 15:39 |
dprince | thrash: yep, may as well change them all at once | 15:39 |
EmilienM | matbu: me too | 15:40 |
thrash | dprince: I'll do a follow-on for the scenario002 | 15:40 |
EmilienM | amoralej, shardy: can we close https://bugs.launchpad.net/tripleo/+bug/1638908 now? | 15:40 |
openstack | Launchpad bug 1638908 in tripleo-quickstart "Overcloud deployment fails in minimal configuration with ('Connection aborted.', BadStatusLine("''",))" [Undecided,In progress] - Assigned to Alfredo Moralejo (amoralej) | 15:40 |
dprince | thrash: anyways, I'm glad to see this stuff getting extra tested so I'm an easy +2 besides this issue | 15:40 |
thrash | I mean 003 | 15:40 |
fultonj | If I create a custom role, then should I be able to delete node in a deployment with that role using `openstack overcloud node delete --stack $ID ... -r ... $node_id` ? | 15:40 |
fultonj | shardy: ^ ? | 15:40 |
shardy | fultonj: as far as I know yes, but you shouldn't need the -r | 15:41 |
shardy | fultonj: disclaimer, I've not tested it, bugs are possible | 15:41 |
openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Add Zaqar to scenario002 https://review.openstack.org/404737 | 15:41 |
fultonj | shardy: when i use the -r i get openstack overcloud node delete: error: unrecognized arguments: -r | 15:41 |
shardy | fultonj: yep, that's because you don't need to pass the roles_data to that command | 15:42 |
shardy | it's only used for the plan create or deploy commands | 15:42 |
fultonj | shardy: so I take it that your recomendation to do the node delete without the -r but to include all of the other templates with -e. | 15:42 |
shardy | fultonj: I don't think you even need those tbh | 15:42 |
shardy | just the stack and node id | 15:42 |
mwhahaha | hey folks, could I get additional reviews on https://review.openstack.org/#/c/383950/ as it's been sitting for some time | 15:42 |
shardy | IIRC the docs need updating as it's no longer needed to pass all the -e arguments | 15:43 |
*** zoli|trng is now known as zoli | 15:43 | |
*** zoli is now known as zoliXXL | 15:43 | |
shardy | mwhahaha: ack, will do | 15:44 |
fultonj | shardy: ok, the methods i mentioned so far failed, so my next step is to test with `openstack overcloud node delete --stack $ID $node_id` | 15:44 |
shardy | fultonj: ack, IIRC that's all I passed last time I tested this | 15:44 |
shardy | but I've not tried with custom roles | 15:45 |
openstackgerrit | yolanda.robla proposed openstack-infra/tripleo-ci: Include logs in the undercloud's home directory https://review.openstack.org/404752 | 15:45 |
openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Make scenario template paths relative https://review.openstack.org/404840 | 15:45 |
fultonj | shardy: i was able to delete the custom FooNode simply by reducing the FooNodeCount | 15:45 |
thrash | dprince: ^^^ | 15:45 |
fultonj | shardy: is this a side-effect and not a recommended method? | 15:45 |
thrash | fultonj: yes | 15:45 |
thrash | fultonj: if you went from 2 -> 1, for instance, it would pick a random node. Probably not what you want. | 15:46 |
jaosorior | yolanda: sorry, I was on a meeting. So we're sure that the user is jenkins then? | 15:46 |
shardy | thrash: actually it will pick the oldest node first IIRC | 15:46 |
thrash | shardy: practically random. :) | 15:46 |
yolanda | i just amended with that , to see if that worked | 15:46 |
shardy | so it may do what you want, but it won't use RemovalPolicies, so when you scale up you'll get another (potentially different) "2" node | 15:47 |
thrash | shardy: that makes sense, but no one should really count on that behaviour | 15:47 |
dprince | thrash: +2 | 15:47 |
openstackgerrit | Merged openstack/tripleo-specs: Changed the home-page link https://review.openstack.org/382591 | 15:47 |
shardy | the advantage of node delete is that you never get a "2" node again, so on scale up you'd get "3" instead | 15:47 |
shardy | thrash: yeah - well heat has done that for some time but I agree it's probably a bad thing to rely on | 15:47 |
openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras: Add role extras-common https://review.openstack.org/404842 | 15:47 |
shardy | thrash: there's also now the option to mark a resource unhealthy, which is a different way to make it get deleted on update | 15:48 |
shardy | so we have a few options here | 15:48 |
fultonj | shardy: how many of those options are in Newton? | 15:49 |
yolanda | jaosorior, cross fingers | 15:50 |
shardy | fultonj: all of them | 15:50 |
*** rhallisey has quit IRC | 15:51 | |
shardy | fultonj: if node delete doesn't work though, that's a bug so I'd say stick with that and lets figure out what's wrong | 15:51 |
thrash | shardy: +1 | 15:51 |
fultonj | shardy: thrash ok, thanks. i will test node delete as we discussed and file a bug if necessary | 15:51 |
shardy | fultonj: ack, sounds good, thanks! | 15:52 |
*** chandankumar is now known as chandankumar|off | 15:52 | |
*** rhallisey has joined #tripleo | 15:52 | |
*** ealcaniz has joined #tripleo | 15:52 | |
*** abregman has quit IRC | 15:53 | |
*** mburned_out is now known as mburned | 15:54 | |
*** abregman has joined #tripleo | 15:55 | |
*** Vijayendra has quit IRC | 15:56 | |
*** mburned is now known as mburned_out | 15:57 | |
*** mburned_out is now known as mburned | 15:58 | |
*** jaosorior has quit IRC | 15:58 | |
openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras: ssl: set IPv6 as certificate CN when necessary https://review.openstack.org/402152 | 15:58 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Apply os-net-config with a script instead of element https://review.openstack.org/271450 | 15:58 |
*** mburned is now known as mburned_out | 15:59 | |
openstackgerrit | Merged openstack/tripleo-ui: Parameters listing updates https://review.openstack.org/403726 | 15:59 |
openstackgerrit | Merged openstack/tripleo-ui: Show team and repo badges on README https://review.openstack.org/403194 | 15:59 |
openstackgerrit | Merged openstack-infra/tripleo-ci: Fix tripleo quickstart role names https://review.openstack.org/403843 | 16:00 |
tbarron | gfidente: ack, please do comment | 16:00 |
openstackgerrit | Merged openstack/tripleo-ui: Relax JSON input validation https://review.openstack.org/403634 | 16:00 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Test encrypted volumes in scenario002 https://review.openstack.org/404223 | 16:00 |
*** dsneddon has joined #tripleo | 16:01 | |
openstackgerrit | Merged openstack/tripleo-common: Add panko to generated passwords https://review.openstack.org/403766 | 16:01 |
*** mburned_out is now known as mburned | 16:02 | |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: scenario001: deploy Ceph https://review.openstack.org/404462 | 16:02 |
*** rcernin has quit IRC | 16:02 | |
*** chlong has quit IRC | 16:03 | |
*** mburned is now known as mburned_out | 16:03 | |
amoralej | EmilienM, about https://bugs.launchpad.net/tripleo/+bug/1638908, https://review.openstack.org/399619 is not merged yet and https://review.openstack.org/#/c/402081/ is not in any tagged release | 16:04 |
openstack | Launchpad bug 1638908 in tripleo-quickstart "Overcloud deployment fails in minimal configuration with ('Connection aborted.', BadStatusLine("''",))" [Undecided,In progress] - Assigned to Alfredo Moralejo (amoralej) | 16:04 |
amoralej | so we couldn't test it yet | 16:04 |
EmilienM | amoralej: ack | 16:04 |
openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: deployment updates for containerized compute https://review.openstack.org/400986 | 16:06 |
*** mburned_out is now known as mburned | 16:06 | |
gfidente | tbarron though we're working with jprovazn to have ceph-mds https://review.openstack.org/#/q/status:open+topic:bug/1644784 so we won't need extceph for manila | 16:10 |
tbarron | gfidente: some customers will want to deploy that way though. The deploy of mds is much appreciated though!! - for one thing it will enable much easier testing. | 16:11 |
openstackgerrit | Merged openstack/tripleo-heat-templates: get-occ-config.sh replace deprecated heat commands https://review.openstack.org/403455 | 16:11 |
gfidente | tbarron ack | 16:12 |
openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Environment is a dict, which can't use 'extend' https://review.openstack.org/404853 | 16:12 |
gfidente | tbarron, EmilienM is setting up some scenarios so that we test the overcloud with different configurations | 16:12 |
*** ebarrera has quit IRC | 16:12 | |
gfidente | maybe we can make some space for manila/cephfs somewhere in the future there | 16:13 |
openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: TEST: DO NOT MERGE - Verbose output for image building https://review.openstack.org/404751 | 16:13 |
EmilienM | gfidente: yes ! | 16:13 |
tbarron | gfidente: yeah, i kinda promised EmilienM that we'd help on that | 16:13 |
gfidente | ok so at least we know what we need to do :) | 16:13 |
gfidente | :P | 16:13 |
*** dsneddon_ has joined #tripleo | 16:14 | |
*** ealcaniz has quit IRC | 16:15 | |
openstackgerrit | Merged openstack/tripleo-puppet-elements: Add ceph-mds to the overcloud-controller element https://review.openstack.org/403636 | 16:16 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Add Mistral to scenario003 https://review.openstack.org/404743 | 16:16 |
openstackgerrit | Merged openstack/tripleo-incubator: Changed the home-page link https://review.openstack.org/382602 | 16:16 |
openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras: Add overcloud delete role https://review.openstack.org/402228 | 16:17 |
*** bana_k has joined #tripleo | 16:17 | |
*** chlong has joined #tripleo | 16:18 | |
*** leanderthal is now known as leanderthal|afk | 16:18 | |
*** dsneddon_ has quit IRC | 16:19 | |
*** rajinir has joined #tripleo | 16:19 | |
openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras: prep-network: add support to IPv6 topology https://review.openstack.org/400837 | 16:20 |
dprince | shardy: CI failure on this one says 'OSError: [Errno 8] Exec format error': https://review.openstack.org/#/c/400252/ | 16:21 |
dprince | shardy: wierd thing is I've tried it locally several times and it works fine for me... | 16:22 |
dprince | shardy: I'm wondering if one of the env variables is getting overwritten by something else | 16:22 |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Hiera optimization: use a new hiera hook https://review.openstack.org/384757 | 16:23 |
*** yamahata has joined #tripleo | 16:25 | |
openstackgerrit | Merged openstack/tripleo-puppet-elements: Show team and repo badges on README https://review.openstack.org/403192 | 16:26 |
*** dmarlin has joined #tripleo | 16:27 | |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints https://review.openstack.org/378040 | 16:28 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add feature testing matrix to readme https://review.openstack.org/399269 | 16:28 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add support for testing predictable placement https://review.openstack.org/378014 | 16:28 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test hostname map https://review.openstack.org/378017 | 16:28 |
openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart: Undercloud: added idempotency and sanity checks https://review.openstack.org/402218 | 16:30 |
shardy | dprince: Hmm, which job are you looking at? The multinode one is failing to start mariadb? | 16:30 |
shardy | I guess that's a side-effect but I don't see the error you mention | 16:31 |
dprince | shardy: I was looking at ovb | 16:31 |
dprince | shardy: yeah, mariadb likely fails due to hosts not getting written perhaps | 16:31 |
*** aufi has quit IRC | 16:31 | |
dprince | shardy: http://logs.openstack.org/52/400252/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/842f2ae/logs/overcloud-controller-0/var/log/messages | 16:32 |
dprince | shardy: The 'Exec format' error shows there... | 16:32 |
dprince | shardy: the script as a shebang on it and all. so I'm a bit perplexed by this one | 16:32 |
*** yprokule_ has joined #tripleo | 16:33 | |
*** yprokule has quit IRC | 16:34 | |
*** yprokule_ is now known as yprokule | 16:34 | |
openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Add environment parameter to collect-logs.sh https://review.openstack.org/404863 | 16:39 |
slagle | shardy: i think i may have figured out my "access forbidden issue". access_allowed_handlers is empty in stack.py, and afaict, only register_access_allowed_handler populates that from nova/server.py, which obviously won't happen in this case | 16:40 |
yolanda | ugh, so i rechecked jaosorio patchsed, changing from /home/stack to /home/jenkins, and now i get echo 'ERROR: the main setup script run by this job failed - exit code: 137' | 16:40 |
yolanda | no more detailed error | 16:40 |
yolanda | sshnaidm, ^ | 16:40 |
sshnaidm | yolanda, looking.. | 16:40 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Implement scenario004 with Ceph Rados Gateway scenario https://review.openstack.org/404458 | 16:41 |
sshnaidm | yolanda, where is the logs? | 16:41 |
yolanda | http://logs.openstack.org/52/404752/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9482243/ | 16:42 |
openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras: Remove packages in the requirements file so they are reinstalled. https://review.openstack.org/404867 | 16:42 |
sshnaidm | yolanda, it's infra problem, not related to patch | 16:42 |
sshnaidm | yolanda, I mean it's ovb infra problem :) | 16:42 |
yolanda | ok... | 16:43 |
yolanda | known issue? | 16:43 |
shardy | dprince: Hmm, strange - can you check the script in /usr/lib/heat-config/heat-config-script? | 16:43 |
sshnaidm | yolanda, it means that ovb stack hasn't been created, need to check why | 16:43 |
shardy | it should still be there, perhaps it's mangled somehow? | 16:43 |
sshnaidm | yolanda, I think you can recheck | 16:43 |
*** jkilpatr has quit IRC | 16:43 | |
shardy | slagle: aha, right, let's think of the best way around that one | 16:44 |
*** bana_k has quit IRC | 16:44 | |
dprince | shardy: yep. I'll add debugging there I guess since I'm not hitting it | 16:44 |
shardy | dprince: Oh, yeah, sorry forgot you couldn't reproduce | 16:44 |
yolanda | ok | 16:44 |
openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Add Zaqar to scenario002 https://review.openstack.org/404737 | 16:45 |
*** jcoufal_ has quit IRC | 16:47 | |
shardy | slagle: I guess the options are patch heat to add a conditional disabling access_allowed in stack.py, or add another server subclass in tripleo-common which overrides the methods that actually interact with nova | 16:47 |
shardy | e.g we could register OS::TripleO::DummyServer based on a subclass in tripleo-common, similar to the one we have now which ignores updates | 16:48 |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Configure /etc/hosts via os-collect-config script https://review.openstack.org/400252 | 16:48 |
ccamacho | matbu here are the issues that Im hitting with the upgrades https://etherpad.openstack.org/p/upstream-newton-to-ocata-upgrades | 16:48 |
slagle | shardy: would the swift temp url transport see this same behavior? | 16:48 |
slagle | maybe i should just try that first | 16:48 |
shardy | slagle: No I don't think it would | 16:48 |
slagle | k, will try. that might be the easiest thing for now | 16:49 |
shardy | kk | 16:49 |
*** ebarrera has joined #tripleo | 16:49 | |
matbu | ccamacho: k i'm hitting the same on neutron: https://paste.fedoraproject.org/494211/05245981/ | 16:50 |
matbu | ccamacho: lets keep this etherpad to track all things | 16:50 |
*** trown is now known as trown|lunch | 16:52 | |
matbu | ccamacho: ha k lol, this is really the same stack trace that i have pasted ^ :) sorry | 16:52 |
mwhahaha | the python-networking-cisco module is broken | 16:53 |
mwhahaha | so you can work around it by removing it before running | 16:53 |
mwhahaha | there's a bug that amoralej created but it's just sitting | 16:53 |
mwhahaha | https://bugs.launchpad.net/networking-cisco/+bug/1641311 | 16:53 |
openstack | Launchpad bug 1641311 in networking-cisco "neutron-db-manage fails after https://review.openstack.org/#/c/394201/" [High,Confirmed] | 16:53 |
mwhahaha | matbu: ccamacho -^ | 16:54 |
matbu | mwhahaha: ha great thanks | 16:54 |
ccamacho | matbu at least the issue is reproducible xD | 16:54 |
mwhahaha | https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_image.py#L193 | 16:54 |
mwhahaha | because we force install it -^ | 16:54 |
*** abregman has quit IRC | 16:54 | |
mwhahaha | i wasted half a day on it yesterday | 16:55 |
* mwhahaha cries | 16:55 | |
*** tosky has quit IRC | 16:57 | |
* ccamacho same man :( http://img06.deviantart.net/d392/i/2012/190/9/4/francella_meme___oh_god_just_kill_me_right_now_by_foreverzerodragon-d56mytj.jpg | 16:58 | |
amoralej | mwhahaha, i thought cisco pkg was remove from tripleo images in https://review.openstack.org/#/c/397959/ | 16:58 |
*** lmiccini has quit IRC | 16:58 | |
mwhahaha | not if you build via openstack overcloud image | 16:59 |
mwhahaha | see https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_image.py#L193 | 16:59 |
amoralej | too many ways of building images.... | 16:59 |
*** jkilpatr has joined #tripleo | 16:59 | |
*** tosky has joined #tripleo | 16:59 | |
*** ccamacho is now known as ccamacho|afk | 16:59 | |
mwhahaha | i don't think oooq uses that, i think it uses the other method | 16:59 |
mwhahaha | soooo yea | 16:59 |
amoralej | yes | 16:59 |
amoralej | that's how it has been passing rdo-ci since then | 17:00 |
mwhahaha | so we need to also remove it from overcloud_image.py if that's the correct method forward | 17:00 |
amoralej | unless until cisco maintainers fix it | 17:01 |
mwhahaha | i don't even know what it's for | 17:01 |
*** panda is now known as panda|bbl | 17:01 | |
amoralej | i don't think is a good idea to install all packages in overcloud image, even if we are not actually testing it | 17:01 |
*** mcornea has quit IRC | 17:01 | |
amoralej | that should be in a cisco CI | 17:01 |
mwhahaha | but we do need a better way to define those packages outside of .py and a .yaml | 17:01 |
amoralej | yes | 17:01 |
*** achadha has joined #tripleo | 17:02 | |
*** achadha has quit IRC | 17:02 | |
*** TSCHAK has joined #tripleo | 17:03 | |
*** achadha has joined #tripleo | 17:03 | |
*** bana_k has joined #tripleo | 17:04 | |
openstackgerrit | Merged openstack/puppet-tripleo: Use FQDNs for the services' RabbitMQ configuration https://review.openstack.org/403518 | 17:05 |
openstackgerrit | Merged openstack/puppet-tripleo: Use FQDNs for RabbitMQ's cluster configuration https://review.openstack.org/403540 | 17:06 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Remove yum-plugin-priorities hack https://review.openstack.org/397331 | 17:08 |
*** sanagikoki has joined #tripleo | 17:08 | |
*** dsariel has quit IRC | 17:09 | |
*** b00tcat has quit IRC | 17:11 | |
openstackgerrit | Merged openstack/os-collect-config: Remove discover from test-requirements https://review.openstack.org/345852 | 17:12 |
*** iranzo has quit IRC | 17:13 | |
openstackgerrit | Merged openstack/tripleo-image-elements: Include lsb_release in the hosts element since it's used there. https://review.openstack.org/404088 | 17:14 |
*** zoliXXL is now known as zoli|gone | 17:14 | |
*** numans has quit IRC | 17:17 | |
*** rhallisey has quit IRC | 17:18 | |
*** nyechiel has quit IRC | 17:18 | |
*** rhallisey has joined #tripleo | 17:18 | |
*** chlong has quit IRC | 17:21 | |
*** pcaruana has quit IRC | 17:23 | |
*** yprokule has quit IRC | 17:24 | |
*** jlinkes_ has quit IRC | 17:24 | |
*** bana_k has quit IRC | 17:24 | |
openstackgerrit | Tomas Sedovic proposed openstack/tripleo-docs: Add tripleo-validations user documentation https://review.openstack.org/396197 | 17:25 |
openstackgerrit | Miles Gould proposed openstack/diskimage-builder: Update hpssacli to ssacli in proliant-tools element https://review.openstack.org/396504 | 17:26 |
openstackgerrit | Miles Gould proposed openstack/diskimage-builder: Allow installing proliant tools from an OS package https://review.openstack.org/344214 | 17:26 |
*** florianf has quit IRC | 17:28 | |
*** jcoufal has joined #tripleo | 17:28 | |
openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient: Remove python-networking-cisco from image build https://review.openstack.org/404890 | 17:29 |
*** rodrigods is now known as rodrigods_ | 17:31 | |
*** chlong has joined #tripleo | 17:34 | |
*** shardy has quit IRC | 17:35 | |
*** saibarspeis has joined #tripleo | 17:35 | |
*** penick has joined #tripleo | 17:36 | |
*** tremble has quit IRC | 17:37 | |
*** dtantsur is now known as dtantsur|afk | 17:37 | |
*** udesale has quit IRC | 17:38 | |
openstackgerrit | Koki Sanagi proposed openstack/tripleo-heat-templates: Add THT for networking-fujitsu https://review.openstack.org/404893 | 17:38 |
*** tremble has joined #tripleo | 17:39 | |
*** saibarspeis has quit IRC | 17:40 | |
EmilienM | gfidente: sounds like it worked, now I have to figure Gnocchi | 17:42 |
EmilienM | http://logs.openstack.org/62/404462/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/fcc2a9e/console.html#_2016-11-30_17_01_42_907071 | 17:42 |
EmilienM | I'll look after lunch | 17:42 |
*** achadha has quit IRC | 17:42 | |
*** milan has quit IRC | 17:44 | |
*** myoung is now known as myoung|lunch | 17:46 | |
beagles | pradk, is it a "thing" for the ceilometer meter and event dispatcher to be different values? | 17:46 |
beagles | pradk, ie. likely to happen, valid config, etc. | 17:46 |
pradk | beagles, yes | 17:47 |
beagles | pradk, ok cool thanks | 17:47 |
*** myoung|lunch is now known as myoung | 17:47 | |
*** derekh has quit IRC | 17:47 | |
*** lucasagomes is now known as lucas-afk | 17:48 | |
openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Added CeilometerEventDispatcher heat parameter https://review.openstack.org/396938 | 17:48 |
openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient: Remove python-networking-cisco from image build https://review.openstack.org/404890 | 17:48 |
*** myoung is now known as myoung|lunch | 17:49 | |
*** yamahata has quit IRC | 17:51 | |
openstackgerrit | Koki Sanagi proposed openstack/tripleo-heat-templates: Add THT for networking-fujitsu https://review.openstack.org/404897 | 17:54 |
openstackgerrit | Merged openstack/os-apply-config: Add __ne__ built-in function https://review.openstack.org/387107 | 17:54 |
*** ctayal has joined #tripleo | 17:55 | |
*** dciabrin is now known as dciabrin|away | 17:55 | |
*** nyechiel has joined #tripleo | 17:59 | |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Configure /etc/hosts via os-collect-config script https://review.openstack.org/400252 | 18:02 |
*** dsneddon_ has joined #tripleo | 18:02 | |
openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: TEST: DO NOT MERGE - Verbose output for image building https://review.openstack.org/404751 | 18:03 |
beagles | pradk, lol good thing you're watching :) | 18:04 |
pradk | :) | 18:04 |
beagles | (re: gnocchi events mitaka) | 18:04 |
gfidente | EmilienM cool | 18:04 |
*** numans has joined #tripleo | 18:04 | |
pradk | EmilienM, so i have a question on gnocchi on undercloud | 18:04 |
pradk | should we just default the backend to file instead of swift | 18:04 |
pradk | EmilienM, to keep the foot print smaller? | 18:05 |
*** penick has quit IRC | 18:05 | |
pradk | considering the issues we saw with gnocchi + swift on overcloud .. i'm wondering if thats worth it on undercloud | 18:05 |
*** sudipto_ has quit IRC | 18:06 | |
*** sudipto has quit IRC | 18:06 | |
*** dsneddon_ has quit IRC | 18:07 | |
pradk | swift on its own had to be tweaked quite a bit to handle gnocchi | 18:07 |
*** tremble has quit IRC | 18:10 | |
*** penick has joined #tripleo | 18:11 | |
EmilienM | pradk: file is good as we don't have HA on undercloud now | 18:11 |
gfidente | pradk actually while you're around :) | 18:12 |
pradk | EmilienM, yea i agree | 18:12 |
gfidente | this job is failing when creating the pingtest stack http://logs.openstack.org/62/404462/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/fcc2a9e/console.html | 18:12 |
pradk | gfidente, looking | 18:12 |
*** dmacpher is now known as dmacpher-afk | 18:12 | |
gfidente | the gnocchi logs should be http://logs.openstack.org/62/404462/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/fcc2a9e/logs/subnode-2/var/log/gnocchi/ | 18:12 |
gfidente | can you spot what went wrong? I don't see any error in the logs | 18:12 |
EmilienM | resources.gnocchi_res_alarm: Resource 0476a3a2-b765-48e9-89e3-fb4b882c1b0b does not exist (HTTP 404) | 18:13 |
EmilienM | yeah I'm also looking :/ | 18:13 |
EmilienM | we see the error in Aodh too http://logs.openstack.org/62/404462/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/fcc2a9e/logs/subnode-2/var/log/aodh/app.txt.gz#_2016-11-30_17_01_36_205 | 18:13 |
*** achadha has joined #tripleo | 18:14 | |
gfidente | ah aodh | 18:14 |
gfidente | thanks | 18:14 |
pradk | looks like the resource doesnt exists | 18:14 |
pradk | 0476a3a2-b765-48e9-89e3-fb4b882c1b0b | 18:14 |
pradk | yea gnocchi alarms are a type of aodh alarms | 18:14 |
gfidente | so aodh is just a consumer there? | 18:15 |
EmilienM | pradk: note: the pingtest without ceph | 18:16 |
EmilienM | so it works fine with file backend I guess | 18:16 |
pradk | gfidente, alarm is created in aodh and evaluated based on data it gathers from gnocchi | 18:16 |
*** ayoung has joined #tripleo | 18:16 | |
*** achadha_ has joined #tripleo | 18:16 | |
pradk | hmm so just with ceph? | 18:16 |
pradk | lemme poke around | 18:17 |
*** achadha has quit IRC | 18:17 | |
*** achadha_ has quit IRC | 18:17 | |
EmilienM | pradk: yes, see https://review.openstack.org/#/c/404462/3/ci/environments/scenario001-multinode.yaml | 18:17 |
matbu | ccamacho|afk: hey man Stack overcloud UPDATE_COMPLETE :) I was going to drink a beer and then ... (but tbh, i'm not sure it's really a working upgrade) | 18:17 |
EmilienM | pradk: GnocchiBackend: 'file' | 18:17 |
ccamacho|afk | matbu!!! http://paste.openstack.org/show/590995/ | 18:17 |
ccamacho|afk | fuck yeah!!! | 18:17 |
*** achadha has joined #tripleo | 18:17 | |
ccamacho|afk | ajajja | 18:17 |
EmilienM | lol | 18:17 |
ccamacho|afk | I mean, there are some services not starting but the approach works, tomorrow Ill glue all the N -> O and add your ansible module xD | 18:18 |
EmilienM | gfidente: we can start without gnocchi/rbd now | 18:20 |
*** jkilpatr has quit IRC | 18:21 | |
gfidente | EmilienM yes I guess that is fine | 18:23 |
gfidente | though the ovb job should be using it for gnocchi too | 18:23 |
gfidente | I actually wasn't even able to find the error causing the resource to fail | 18:24 |
gfidente | not sure if pradk found anything | 18:24 |
gfidente | but I would try just one last recheck before giving up | 18:24 |
*** jkilpatr has joined #tripleo | 18:24 | |
pradk | gfidente, nothing standing out yet .. did gnocchi-upgrade run with or without skip-storage.. do we know? | 18:25 |
EmilienM | gfidente: are you sure ovb job runs gnocchi with rbd backend? | 18:25 |
EmilienM | gfidente: you don't run any gnocchi pingtest for ovb :) | 18:26 |
EmilienM | so we can't certify it works | 18:26 |
gfidente | ah that's not good | 18:26 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Implement scenario004 with Ceph Rados Gateway scenario https://review.openstack.org/404458 | 18:26 |
gfidente | as that means we've a problem in the stable release | 18:27 |
gfidente | :( | 18:27 |
Guest68279 | mwhahaha: heya - could you share the data for determining the # of workers? didn't know if there was a link to this work. | 18:27 |
*** Guest68279 is now known as rook | 18:27 | |
EmilienM | gfidente: I don't know. pradk: did you test gnocchi+rbd outside CI? | 18:28 |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Prepare build-test role for usage in upstream https://review.openstack.org/404823 | 18:28 |
mwhahaha | rook: so the numbers were primarily chosen based on feed back from other groups, for example https://review.openstack.org/#/c/297342/ has a discussion in the reviews on optimal numbers for keystone | 18:28 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Fix unbound variable in standalone ping test https://review.openstack.org/404914 | 18:28 |
rook | yup mwhahaha akrzos is on my team... we were talking about this today. | 18:29 |
rook | Alex Krzos akrzos who started that discussion | 18:29 |
gfidente | EmilienM but pingtest_template seems empty to me | 18:30 |
mwhahaha | but that being said, i had similar stuff over in fuel land, let me get bugs | 18:30 |
pradk | EmilienM, gfidente, http://logs.openstack.org/62/404462/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/fcc2a9e/console.html#_2016-11-30_17_01_38_956299 | 18:30 |
gfidente | so I assume it's using the standard pingtest template for the scenario too, no? | 18:30 |
mwhahaha | rook: https://review.openstack.org/#/c/311079/ | 18:30 |
*** bana_k has joined #tripleo | 18:31 | |
mwhahaha | rook: so overall, the 'defaults' from python have generally been bad. the puppet is an attempt to adjust down the defaults. but like i keep saying they defaults are not optimized. never are so if there is tuning to be had it should be done on a case by case basis and we need to provide a better way for users to consume tuned numbers | 18:31 |
EmilienM | pradk: http://logs.openstack.org/62/404462/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/fcc2a9e/console.html#_2016-11-30_17_01_41_692200 | 18:31 |
*** arxcruz has quit IRC | 18:32 | |
mwhahaha | rook: so my recommendation is to stop obessing about os_workers unless you have another formula that works across the board that will address basic memory consumption and conneciton limit issues for all services and focus on specific tunings that could be consumed via THT | 18:32 |
pradk | EmilienM, yea so the resource doesnt exist in gnocchi.. my suspision is we need to initialize storage and we arent | 18:32 |
rook | mwhahaha i agree - I just thought there was data backing the decision vs. just discussions. | 18:32 |
pradk | EmilienM, does this work with swift? | 18:32 |
gfidente | EmilienM oh no I see the variable is set based on multinode env | 18:32 |
gfidente | cool | 18:33 |
mwhahaha | rook: we had raw numbers for fuel but I no longer have them so we attempted to take the work started over there and apply it upstream | 18:33 |
rook | mwhahaha it is less about obessing(sp?) and more about being able to discuss data, which seems lacking. | 18:33 |
EmilienM | pradk: I didn't check | 18:33 |
*** arxcruz has joined #tripleo | 18:33 | |
mwhahaha | the data was not published in a public forum so I can't provide the history | 18:33 |
rook | mwhahaha ok, cool... if there was a method to the testing, that would be great... | 18:34 |
gfidente | pradk how are we supposed to initialize storage? | 18:34 |
*** yamahata has joined #tripleo | 18:34 | |
*** ebarrera has quit IRC | 18:34 | |
gfidente | pradk fwiw I think you can download the gnocchi.conf with the ceph settings here http://logs.openstack.org/62/404462/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/fcc2a9e/logs/subnode-2/etc/gnocchi/gnocchi.conf.txt.gz | 18:35 |
pradk | gfidente, when gnocchi-upgrade run it does it unless we pass skip-storage.. but my swift works fine, so i would expect ceph to follow that | 18:35 |
*** kbyrne has quit IRC | 18:35 | |
pradk | gfidente, looking at conf, one sec | 18:35 |
gfidente | pradk sounds like we need to track this down with a bug | 18:36 |
gfidente | maybe for the scenario we can start with file | 18:36 |
gfidente | wdyt? | 18:36 |
*** jpena is now known as jpena|off | 18:37 | |
pradk | gfidente, yea sounds good to me.. i'll do a deploy with ceph tonight and see if i can reproduce it locally | 18:37 |
pradk | gfidente, this is working in newton for sure as we did a validation with qe last week and gnocchi alarms work | 18:38 |
*** kbyrne has joined #tripleo | 18:38 | |
pradk | gfidente, assume there is no way to hop on this ci node and check whats in gnocchi? | 18:39 |
*** trown|lunch is now known as trown | 18:39 | |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Don't run on 127.0.0.* hosts https://review.openstack.org/404920 | 18:44 |
*** myoung|lunch is now known as myoung | 18:45 | |
*** amoralej is now known as amoralej|off | 18:47 | |
*** rodrigods_ is now known as rodrigods | 18:49 | |
*** fzdarsky is now known as fzdarsky|afk | 18:50 | |
openstackgerrit | Merged openstack/tripleo-specs: Enable Oslo.Messaging Dual Backends https://review.openstack.org/396740 | 18:54 |
thrash | yolanda: any idea what the error was with image building in the first place? Was it on a specific patch? | 18:55 |
thrash | yolanda: looks like it's working in general on the ovb-ha job. | 18:55 |
*** tosky has quit IRC | 18:56 | |
*** penick has quit IRC | 18:57 | |
openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Support multiple meter dispatchers in ceilometer config https://review.openstack.org/404924 | 19:01 |
*** penick has joined #tripleo | 19:01 | |
yolanda | thrash, this one for example https://review.openstack.org/#/c/404161/ | 19:02 |
openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud: Add gnocchi support on undercloud https://review.openstack.org/392992 | 19:04 |
*** pkovar has quit IRC | 19:04 | |
*** egafford has quit IRC | 19:05 | |
*** cylopez has quit IRC | 19:05 | |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Hiera optimization: use a new hiera hook https://review.openstack.org/384757 | 19:06 |
dprince | trown: mind giving this a quick review https://review.rdoproject.org/r/#/c/3947/1 | 19:08 |
trown | dprince: sure | 19:08 |
EmilienM | dprince: +2 | 19:09 |
dprince | EmilienM: ack, thanks. This will help unblock me a bit | 19:09 |
dprince | trown: while you are at it this would be handy to have too https://review.openstack.org/#/c/392615/ | 19:10 |
dprince | trown: passing CI... | 19:10 |
trown | hehe... "There is a new hiera hook in town" | 19:10 |
*** saneax-_-|AFK is now known as saneax | 19:12 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Add Neutron network type and VLAN ranges to network-environment.yaml https://review.openstack.org/392325 | 19:15 |
openstackgerrit | Charlie Llewellyn proposed openstack/tripleo-heat-templates: Add retry to RHEL registration https://review.openstack.org/386529 | 19:15 |
*** snecklifter has joined #tripleo | 19:17 | |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Configure /etc/hosts via os-collect-config script https://review.openstack.org/400252 | 19:24 |
*** SpamapS has quit IRC | 19:26 | |
*** dciabrin|away has quit IRC | 19:27 | |
*** dciabrin|away has joined #tripleo | 19:28 | |
dprince | trown: https://review.openstack.org/#/c/392559/ please? | 19:28 |
dprince | oh, there is a fixup in the commit message | 19:29 |
openstackgerrit | Dan Prince proposed openstack/tripleo-puppet-elements: Move nova.conf truncation into separate element https://review.openstack.org/392559 | 19:29 |
dprince | beagles, EmilienM: again ^^^ just a commit msg update | 19:29 |
*** akrivoka has quit IRC | 19:30 | |
bnemec | Can we merge https://review.openstack.org/#/c/403744/ ? | 19:34 |
trown | dprince: how does that install.d bit work today? | 19:34 |
bnemec | It passed all the jobs on the previous patch set and the only change was the commit message. | 19:34 |
dprince | bnemec: +A | 19:35 |
bnemec | dprince: Thanks | 19:35 |
trown | dprince: nevermind read up to sbaker's comment | 19:35 |
trown | dprince: makes sense now | 19:35 |
dprince | trown: ck | 19:36 |
dprince | a-ck | 19:36 |
trown | dprince: do we want to remove os-refresh-config from element-deps then? | 19:36 |
openstackgerrit | Koki Sanagi proposed openstack/puppet-tripleo: Add networking-fujitsu support https://review.openstack.org/404938 | 19:37 |
dprince | trown: we want to move away from installing os-refresh-config scripts via the "magic" os-refresh-config directory | 19:38 |
dprince | trown: we'll still use os-refresh-config and all, but we'll directly install the scripts via install.d package stuff | 19:38 |
dprince | trown: all of this is moving us away from image-element crufty-ness | 19:38 |
dprince | trown: and eventually even instack, etc. | 19:39 |
*** jprovazn has quit IRC | 19:39 | |
dprince | if I don't get any reviews on https://review.openstack.org/#/c/342486/ soon I'm just going to land it myself | 19:41 |
*** SpamapS has joined #tripleo | 19:41 | |
dprince | almost 5 months of no core reviews and I'm not breaking anything | 19:42 |
trown | dprince: ok that part makes sense, so eventually we would get rid of the install.d part in the os-refresh-config element? and/or move it out of t-i-e? | 19:42 |
dprince | trown: I think eventually the idea is that we'll install os-refresh-config (the package) but not require anything in tripleo-image-elements | 19:43 |
dprince | trown: right now we rely on the tripleo-image-elements os-refresh-config element to move element scripts around for us | 19:43 |
trown | dprince: ah ok, then we can remove it from element-deps | 19:43 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Remove yum-plugin-priorities hack https://review.openstack.org/397331 | 19:43 |
dprince | trown: eventually, yes | 19:44 |
trown | got it | 19:44 |
*** anshul has joined #tripleo | 19:46 | |
openstackgerrit | Merged openstack/tripleo-quickstart: main script: Use zuul cloner if present to fetch an pip install extras https://review.openstack.org/403776 | 19:51 |
*** achadha_ has joined #tripleo | 19:51 | |
openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras: overcloud-deploy: add variable for hypervisor_wait https://review.openstack.org/404943 | 19:52 |
*** anshul has quit IRC | 19:53 | |
*** achadha has quit IRC | 19:54 | |
openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart-extras: Add introspection with retries option https://review.openstack.org/403677 | 19:57 |
gfidente | EmilienM mind if I give a recheck there, just to make sure? | 20:07 |
EmilienM | gfidente: where? | 20:07 |
gfidente | scenario | 20:07 |
*** achadha has joined #tripleo | 20:08 | |
*** achadha has quit IRC | 20:08 | |
EmilienM | gfidente: sure thing | 20:08 |
*** achadha has joined #tripleo | 20:09 | |
stevebaker | morning | 20:09 |
*** achadha_ has quit IRC | 20:09 | |
EmilienM | stevebaker: o/ | 20:09 |
EmilienM | gfidente: otherwise I'll use file backend for gnocchi | 20:09 |
openstackgerrit | Merged openstack/tripleo-quickstart: Always upload overcloud image when has been customized https://review.openstack.org/396849 | 20:09 |
*** achadha_ has joined #tripleo | 20:10 | |
stevebaker | trown, dprince: I'm slowly working towards all orc scripts being installed by RPMs, not elements - this is the next step https://review.openstack.org/#/c/386326/ | 20:10 |
*** dsariel has joined #tripleo | 20:11 | |
stevebaker | trown: and anything left over can be installed by some install.d shell | 20:11 |
dprince | stevebaker: ack, I like it | 20:11 |
matbu | ccamacho|afk: great :) | 20:12 |
gfidente | EmilienM yes if it fails again | 20:13 |
*** achadha has quit IRC | 20:13 | |
gfidente | dprince trying, https://review.openstack.org/#/c/342486/5/os_collect_config/zaqar.py | 20:13 |
EmilienM | gfidente: ack | 20:13 |
gfidente | on line118 the timeout is the ttl set on line 90 ? | 20:13 |
dprince | gfidente: no, that is the queue timeout. Socket timeout would be separate I think | 20:14 |
*** achadha_ has quit IRC | 20:15 | |
gfidente | oh it's the socket timeot | 20:15 |
dprince | gfidente: sorry, I may hve mis-read your question. The timeout on line 90 is just the queue timeout | 20:16 |
gfidente | dprince ok I was trying to understand what happens after the timeout | 20:16 |
dprince | gfidente: which relates to how long zaqar will subscribe you to that queue | 20:16 |
dprince | gfidente: if it times out os-collect-config runs again. Totally safe I think since we always re-check the message queue again on each os-collect-config iteration | 20:17 |
gfidente | so do I get it right this will rise an ex | 20:17 |
gfidente | exit and rerun | 20:17 |
dprince | gfidente: exactly | 20:17 |
gfidente | ah cool, thanks | 20:18 |
openstackgerrit | Koki Sanagi proposed openstack/puppet-tripleo: Add networking-fujitsu support to puppet-tripleo https://review.openstack.org/404938 | 20:21 |
*** panda|bbl is now known as panda | 20:21 | |
*** hjensas has joined #tripleo | 20:22 | |
*** hjensas has quit IRC | 20:22 | |
*** hjensas has joined #tripleo | 20:22 | |
slagle | stevebaker: hey, i'm trying to move away from having to using admin creds when using the deploued-server template | 20:22 |
slagle | assuming POLL_TEMP_URL is the software config transport...can I query heat to tell me what the temp url actually is? | 20:23 |
slagle | i dont see it in the resource metadata | 20:23 |
*** nyechiel has quit IRC | 20:24 | |
stevebaker | slagle: yeah, I've been thinking about this too. I think the only option is to implement something properly in heat, something like a OS::Heat::DeployedServer which is a drop-in replacement for OS::Nova::Server | 20:26 |
*** paramite has quit IRC | 20:26 | |
slagle | stevebaker: ok, yea that might be the only option i guess | 20:26 |
stevebaker | slagle: there may be another way, now that I actually have deployed-server working for me I'll have a proper look | 20:27 |
slagle | earlier i tried to use POLL_SERVER_HEAT, but discovered that there were no access_allowed_handlers registered so occ was getting a forbidden error | 20:27 |
openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Use transient hostname for deployed servers https://review.openstack.org/404954 | 20:27 |
slagle | turns out those get registered in nova/server.py in heat | 20:28 |
*** jcoufal has quit IRC | 20:28 | |
dprince | slagle, stevebaker: I might prefer an in-heat version of deployed-server I think | 20:28 |
slagle | so now i was trying to use POLL_TEMP_URL..but getting stuck again | 20:28 |
stevebaker | yeah, there is a lot of logic in nova/server.py to build the occ bootstrap data, that needs to be exposed some other way | 20:28 |
dprince | slagle, stevebaker: I tried to switch you over to apply-config (which uses proper signalling) and hit a race condition in the implementation I think https://review.openstack.org/#/c/396171/ | 20:29 |
stevebaker | slagle: any chance undercloud can have an ironic *and* fake drivers set up now? | 20:29 |
* dprince was considering trying no signalling for this | 20:29 | |
slagle | stevebaker: you mean mixing the 2? | 20:29 |
stevebaker | slagle: yeah | 20:30 |
slagle | not sure if that would work with deployed-server | 20:30 |
*** saneax is now known as saneax-_-|AFK | 20:30 | |
stevebaker | you wouldn't need deployed server at all if some of your servers were fake nova, just something equivalent to get-occ-config.sh | 20:31 |
stevebaker | dprince: what is racing do you know? | 20:31 |
dprince | stevebaker: If I try to rely on heat-config signals instead of 99-refresh-config it hangs | 20:32 |
dprince | stevebaker: the signel is never recieved and it hangs | 20:32 |
slagle | stevebaker: yea, that would probably work | 20:32 |
openstackgerrit | Merged openstack/tripleo-puppet-elements: Disable old hiera.yaml o-a-c script https://review.openstack.org/392615 | 20:32 |
*** ohamada has quit IRC | 20:32 | |
stevebaker | dprince: hmm, now that I actually have a working environment, I'll try it out | 20:33 |
stevebaker | slagle: so is nova capable of having different drivers on one host now? | 20:34 |
dprince | stevebaker: I was guessing that no_signal might fix it perhaps. hacky though | 20:34 |
slagle | stevebaker: not that i know, that would have to be on the ironic side | 20:34 |
slagle | ironic configures the driver per node, and has a fake driver | 20:34 |
dprince | slagle: I personally would prefer the in-heat option as relying on Ironic,Nova is a whole bunch of extra requirements | 20:35 |
stevebaker | dprince: remember that execution order of hook scripts is determined by the name property of the deployment resource. If you're not setting a name explicitly then you can't be sure of execution order | 20:36 |
*** dbecker has quit IRC | 20:36 | |
dprince | slagle: it wouldn't be ideal for the undercloud installation case at all which also would rely on a deployed server mechanism | 20:36 |
slagle | dprince: right..both are kind of viable options depending on the use case | 20:36 |
slagle | if you want a "heat only" solution, deployed-server is for that | 20:36 |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-docs: Document how to quiesce and delete a cephstorage node https://review.openstack.org/404434 | 20:36 |
dprince | stevebaker: so set name on the InstanceIdDeployment then? | 20:37 |
stevebaker | yes, a heat-only solution would be good | 20:37 |
*** rcernin has joined #tripleo | 20:37 | |
dprince | stevebaker: I haven't tried that. I did try adding the explicit depends on to the HostEntryDeployment. But that seemed to have no effect | 20:38 |
stevebaker | dprince: its just something to keep in mind - it may have no bearing to this situation | 20:38 |
slagle | stevebaker: would it make sense to refactor the nova/server.py class into a base class with all the nova api calls in a nova specific subclass in heat? | 20:38 |
dprince | stevebaker: it is related to the fact that all of the heat "hooks" run at os-refresh-config 50 or so | 20:38 |
dprince | stevebaker: so the metadata shows up much later than it did before | 20:38 |
slagle | stevebaker: i can start looking into that if you think it makes sense | 20:38 |
stevebaker | slagle: yeah, or a subclass of Server which makes no nova calls | 20:39 |
slagle | ok, will have a look | 20:39 |
*** ctayal has quit IRC | 20:40 | |
*** rodrigods is now known as rodrigods_ | 20:41 | |
stevebaker | slagle: actually, I think break out the occ logic into utility functions and have a minimal OS::Heat::DeployedServer resource which exposes the resulting deployment data. If the user needs something which behaves like a OS::Nova::Server resource they can to a deployed-server style template with the parameters/outputs needed to pretend to be their OS::Nova::Server | 20:42 |
*** egafford has joined #tripleo | 20:42 | |
dprince | slagle: you are tring this to avoid the admin credentials for CI jobs then? | 20:43 |
dprince | slagle: I like this idea, just trying to understand your use case... | 20:43 |
slagle | dprince: more or less, yea | 20:43 |
slagle | getting away from admin creds was just the first thing i tried to tackle to make progress towards a true split stack | 20:44 |
slagle | i thought it would be easy'ish :) | 20:44 |
slagle | but my end goal overall is split stack | 20:45 |
slagle | and trying to make deployed-server more fit for production | 20:45 |
dprince | slagle: well, FWIW the undercloud installer case I'm using it just fine without get-occ-data.sh and I consider that production enough | 20:46 |
dprince | slagle: but I have a different mechanism in place for the stack-id etc. | 20:47 |
slagle | dprince: i havent looked at that in a little while, but how are you configuring occ? | 20:47 |
stevebaker | slagle: is there any CI job which is OVB with deployed-server? | 20:47 |
slagle | stevebaker: no | 20:48 |
*** achadha has joined #tripleo | 20:48 | |
stevebaker | slagle: how about a CI job with deployed-server? | 20:48 |
slagle | stevebaker: yes, the multinode jobs | 20:48 |
dprince | slagle: I exec it manually rather than have it run via systemd. And I create the stack each time, so the stack is essentially ephemeral | 20:48 |
dprince | slagle: you could do the same thing with and Ansible o-c-c runner | 20:49 |
dprince | slagle: and then you'd have a split stack I think | 20:49 |
slagle | stevebaker: nonha-multinode, and all the scenario jobs | 20:49 |
stevebaker | slagle: ok, my dev env is now OVB+deployed-server - maybe this can become more common in the future | 20:49 |
dprince | slagle: line 67 here https://review.openstack.org/#/c/351351/7/tripleoclient/v1/undercloud_deploy.py | 20:50 |
dprince | slagle: I've got a much newer version locally but still polishing it up a bit | 20:51 |
slagle | dprince: yea, was just looking at that | 20:51 |
slagle | i think using admin is fine there for the undercloud install | 20:51 |
slagle | but i dont want to have to do that on overcloud nodes | 20:52 |
*** arxcruz has quit IRC | 20:52 | |
dprince | slagle: right, well if the stack is ephemeral I think we could probably configure heat with some other user too | 20:52 |
dprince | slagle: you just need a user you can pre-create so you can trick out the signalling | 20:52 |
stevebaker | slagle: there are resources to create keystone users - if there was just a way to allow that user to call heat on that stack | 20:53 |
*** liverpooler has quit IRC | 20:54 | |
*** nyechiel has joined #tripleo | 20:58 | |
openstackgerrit | Merged openstack-infra/tripleo-ci: updates: fix condition that adds correct DEPLOY_ARGS https://review.openstack.org/403744 | 21:00 |
*** ctayal has joined #tripleo | 21:00 | |
*** ctayal has quit IRC | 21:00 | |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Make pep8 task run template generation https://review.openstack.org/394415 | 21:02 |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add local template generation tox task https://review.openstack.org/394410 | 21:02 |
openstackgerrit | Christian Schwede proposed openstack/puppet-tripleo: Include swift::storage::loopbacks class https://review.openstack.org/404967 | 21:03 |
*** dprince has quit IRC | 21:03 | |
*** gfidente has quit IRC | 21:07 | |
*** jayg is now known as jayg|g0n3 | 21:09 | |
*** ctayal has joined #tripleo | 21:10 | |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Always use cached images in updates job https://review.openstack.org/386713 | 21:12 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Temporarily disable ceph in the updates job https://review.openstack.org/397288 | 21:12 |
*** ctayal has quit IRC | 21:13 | |
*** amoralej|off is now known as amoralej | 21:21 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Implement scenario004 with Ceph Rados Gateway scenario https://review.openstack.org/404458 | 21:21 |
openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras: Use generic cloud image for undercloud https://review.openstack.org/401548 | 21:23 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: scenario001: deploy Ceph https://review.openstack.org/404462 | 21:24 |
openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Environment is a dict, which can't use 'extend' https://review.openstack.org/404853 | 21:25 |
*** owalsh has quit IRC | 21:32 | |
*** owalsh_ has joined #tripleo | 21:33 | |
*** ctayal has joined #tripleo | 21:36 | |
*** owalsh__ has joined #tripleo | 21:37 | |
*** jkilpatr has quit IRC | 21:38 | |
*** dsneddon_ has joined #tripleo | 21:39 | |
*** owalsh__ is now known as owalsh | 21:39 | |
*** owalsh_ has quit IRC | 21:39 | |
*** rhallisey has quit IRC | 21:42 | |
*** dsneddon_ has quit IRC | 21:43 | |
openstackgerrit | Merged openstack/tripleo-common: Build overcloud with overcloud-agent element https://review.openstack.org/386326 | 21:48 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints https://review.openstack.org/378040 | 21:48 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add feature testing matrix to readme https://review.openstack.org/399269 | 21:48 |
openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud: Add gnocchi support on undercloud https://review.openstack.org/392992 | 21:49 |
*** jtomasek has quit IRC | 21:51 | |
*** dtrainor has quit IRC | 21:52 | |
*** ctayal has quit IRC | 21:54 | |
*** cdearborn has joined #tripleo | 21:59 | |
*** jrist has quit IRC | 22:01 | |
*** [1]cdearborn has quit IRC | 22:01 | |
openstackgerrit | Merged openstack/tripleo-puppet-elements: Move nova.conf truncation into separate element https://review.openstack.org/392559 | 22:01 |
*** lblanchard has quit IRC | 22:01 | |
*** jrist has joined #tripleo | 22:02 | |
*** jrist has joined #tripleo | 22:02 | |
*** ctayal has joined #tripleo | 22:05 | |
*** jrist has quit IRC | 22:07 | |
*** dbecker has joined #tripleo | 22:08 | |
*** trown is now known as trown|outtypewww | 22:10 | |
*** bfournie has quit IRC | 22:10 | |
*** bfournie has joined #tripleo | 22:11 | |
*** bfournie has quit IRC | 22:11 | |
*** penick has quit IRC | 22:12 | |
*** pblaho` has joined #tripleo | 22:14 | |
*** bfournie has joined #tripleo | 22:16 | |
*** pblaho has quit IRC | 22:16 | |
*** ctayal has quit IRC | 22:18 | |
*** rasca has quit IRC | 22:20 | |
openstackgerrit | Steve Baker proposed openstack/tripleo-common: Represent dib elements as a list https://review.openstack.org/402249 | 22:21 |
*** TSCHAK has quit IRC | 22:21 | |
*** amoralej is now known as amoralej|off | 22:22 | |
*** lblanchard has joined #tripleo | 22:23 | |
*** Goneri has quit IRC | 22:25 | |
*** jeckersb is now known as jeckersb_gone | 22:31 | |
openstackgerrit | Dan Prince proposed openstack/tripleo-common: Drop use of hosts element. https://review.openstack.org/400253 | 22:34 |
*** ctayal has joined #tripleo | 22:35 | |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Deployed server: switch to apply-config hook https://review.openstack.org/396171 | 22:36 |
openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient: Add overcloud support report command https://review.openstack.org/405001 | 22:36 |
openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient: Add overcloud support report command https://review.openstack.org/405001 | 22:37 |
*** fultonj has quit IRC | 22:38 | |
*** mnaser has quit IRC | 22:41 | |
*** jrist has joined #tripleo | 22:42 | |
*** lblanchard has quit IRC | 22:42 | |
openstackgerrit | Alex Schultz proposed openstack/tripleo-common: Add support workflows https://review.openstack.org/405002 | 22:43 |
*** cdearborn has quit IRC | 22:48 | |
*** mnaser has joined #tripleo | 22:50 | |
*** achadha_ has joined #tripleo | 22:57 | |
*** noslzzp has quit IRC | 22:57 | |
*** achadha has quit IRC | 22:57 | |
*** athomas has quit IRC | 22:58 | |
*** flepied has quit IRC | 23:01 | |
*** achadha_ has quit IRC | 23:01 | |
*** dbecker has quit IRC | 23:02 | |
*** nyechiel has quit IRC | 23:02 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Implement scenario004 with Ceph Rados Gateway scenario https://review.openstack.org/404458 | 23:12 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/os-collect-config: Updated from global requirements https://review.openstack.org/400907 | 23:13 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/400776 | 23:16 |
*** panda is now known as panda|Zz | 23:16 | |
*** chlong has quit IRC | 23:22 | |
*** ctayal has quit IRC | 23:24 | |
*** pmannidi has joined #tripleo | 23:25 | |
openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras: Use generic cloud image for undercloud https://review.openstack.org/401548 | 23:25 |
*** rcernin has quit IRC | 23:26 | |
*** dsneddon_ has joined #tripleo | 23:27 | |
*** jcoufal has joined #tripleo | 23:29 | |
*** dsneddon_ has quit IRC | 23:31 | |
*** apetrich has quit IRC | 23:32 | |
*** apetrich has joined #tripleo | 23:32 | |
*** penick has joined #tripleo | 23:38 | |
*** Goneri has joined #tripleo | 23:40 | |
*** penick_ has joined #tripleo | 23:42 | |
*** pradk has quit IRC | 23:43 | |
*** penick has quit IRC | 23:43 | |
*** penick_ is now known as penick | 23:43 | |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Hiera optimization: use a new hiera hook https://review.openstack.org/384757 | 23:45 |
*** jkilpatr has joined #tripleo | 23:46 | |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Make pep8 task run template generation https://review.openstack.org/394415 | 23:47 |
*** ctayal has joined #tripleo | 23:51 | |
openstackgerrit | Merged openstack/tripleo-common: Environment is a dict, which can't use 'extend' https://review.openstack.org/404853 | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!