| *** akuznetsov has joined #tripleo | 00:10 | |
| *** akuznetsov has quit IRC | 00:16 | |
| *** rlandy is now known as rlandy|bbl | 00:25 | |
| *** lblanchard has joined #tripleo | 00:45 | |
| *** limao has joined #tripleo | 00:52 | |
| *** dhill_ has quit IRC | 00:54 | |
| *** bana_k has quit IRC | 01:06 | |
| *** akuznetsov has joined #tripleo | 01:11 | |
| *** akuznetsov has quit IRC | 01:15 | |
| *** yamahata has quit IRC | 01:15 | |
| *** bkopilov has quit IRC | 01:26 | |
| *** cmyster has quit IRC | 01:26 | |
| *** saneax is now known as saneax-_-|AFK | 01:28 | |
| *** cmyster has joined #tripleo | 01:29 | |
| openstackgerrit | zhangyanxian proposed openstack/tripleo-image-elements: Fix typos in rootwrap.conf https://review.openstack.org/373922 | 01:32 |
|---|---|---|
| *** bkopilov has joined #tripleo | 01:32 | |
| *** dmacpher-afk has quit IRC | 01:44 | |
| *** alop has joined #tripleo | 02:00 | |
| *** mburned is now known as mburned_out | 02:08 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Re-enable temprevert/cherry-pick/pin functionality https://review.openstack.org/370961 | 02:08 |
| *** akuznetsov has joined #tripleo | 02:12 | |
| *** akuznetsov has quit IRC | 02:16 | |
| *** rlandy|bbl is now known as rlandy | 02:38 | |
| *** alop has quit IRC | 02:58 | |
| *** david-lyle has quit IRC | 03:03 | |
| *** rlandy has quit IRC | 03:07 | |
| *** dmacpher has joined #tripleo | 03:11 | |
| *** akuznetsov has joined #tripleo | 03:13 | |
| *** akuznetsov has quit IRC | 03:17 | |
| *** ebalduf has joined #tripleo | 03:25 | |
| *** akshai has joined #tripleo | 03:33 | |
| *** coolsvap has joined #tripleo | 03:36 | |
| *** akshai has quit IRC | 03:37 | |
| *** cmyster has quit IRC | 03:40 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore https://review.openstack.org/365449 | 03:40 |
| *** rwsu has joined #tripleo | 03:43 | |
| *** cmyster has joined #tripleo | 03:48 | |
| *** akuznetsov has joined #tripleo | 04:13 | |
| *** kberger has quit IRC | 04:15 | |
| *** kberger has joined #tripleo | 04:16 | |
| *** akuznetsov has quit IRC | 04:18 | |
| *** michchap has quit IRC | 04:20 | |
| *** yolanda has quit IRC | 04:20 | |
| *** yamahata has joined #tripleo | 04:37 | |
| *** kberger has quit IRC | 04:45 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Remove excessive output when configuring nodes https://review.openstack.org/372477 | 04:46 |
| *** kberger has joined #tripleo | 04:46 | |
| *** bana_k has joined #tripleo | 04:52 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Remove openstackclient imports in the new parameters command https://review.openstack.org/372621 | 04:55 |
| *** sshnaidm|afk is now known as sshnaidm | 05:03 | |
| *** saneax-_-|AFK is now known as saneax | 05:03 | |
| sshnaidm | morning | 05:03 |
| *** jaosorior has joined #tripleo | 05:07 | |
| *** akuznetsov has joined #tripleo | 05:14 | |
| *** akuznetsov has quit IRC | 05:18 | |
| *** ianw has quit IRC | 05:23 | |
| *** ianw has joined #tripleo | 05:27 | |
| *** mcornea has joined #tripleo | 05:30 | |
| *** tzumainn has quit IRC | 05:42 | |
| *** rajinir has quit IRC | 05:45 | |
| *** bana_k has quit IRC | 05:46 | |
| *** florianf has joined #tripleo | 05:51 | |
| *** florianf has quit IRC | 05:51 | |
| *** limao has quit IRC | 05:52 | |
| *** florianf has joined #tripleo | 05:54 | |
| cmyster | morning | 05:56 |
| *** apetrich has quit IRC | 06:01 | |
| *** apetrich has joined #tripleo | 06:02 | |
| *** rcernin has joined #tripleo | 06:07 | |
| *** ccamacho has joined #tripleo | 06:09 | |
| *** pgadiya has joined #tripleo | 06:09 | |
| *** akuznetsov has joined #tripleo | 06:15 | |
| *** rasca has joined #tripleo | 06:15 | |
| *** akuznetsov has quit IRC | 06:20 | |
| *** limao has joined #tripleo | 06:20 | |
| *** shardy has joined #tripleo | 06:23 | |
| *** oshvartz has joined #tripleo | 06:23 | |
| *** nyechiel has joined #tripleo | 06:23 | |
| *** bana_k has joined #tripleo | 06:24 | |
| *** jprovazn has joined #tripleo | 06:25 | |
| *** jbadiapa has joined #tripleo | 06:34 | |
| bandini | morning | 06:34 |
| bandini | matbu: ever seen this one? http://paste.openstack.org/show/582501/ | 06:34 |
| bandini | Property KeystoneCredential0 not assigned when doing upgrades | 06:35 |
| *** aufi has joined #tripleo | 06:35 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Add missing unit tests for the 'configure' workflows https://review.openstack.org/369332 | 06:35 |
| jaosorior | bandini: that property was introduced very recently to puppet-keystone | 06:37 |
| *** aufi has quit IRC | 06:38 | |
| *** aufi has joined #tripleo | 06:38 | |
| bandini | jaosorior: thanks. This might be a problem, because we this is during upgrade where we have new tht but the older puppet modules on the overcloud (it is the init step of the upgrade) | 06:39 |
| * bandini looks | 06:39 | |
| matbu | bandini: /me looks | 06:39 |
| ccamacho | morning guys | 06:39 |
| matbu | bandini: nop not yet :) | 06:40 |
| bandini | matbu: be quick in hitting it, otherwise I feel lonely :D | 06:40 |
| jaosorior | shardy: hey dude are you around yet> | 06:40 |
| social | moin | 06:40 |
| jaosorior | shardy: I'm adding a script to an openstack service. But since the script has not been packages it fails when I do --delorean-build. How do I address that? | 06:41 |
| matbu | bandini: lol yes | 06:43 |
| *** apetrich has quit IRC | 06:44 | |
| *** apetrich has joined #tripleo | 06:46 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use the passed in workflow when creating or updating a plan https://review.openstack.org/374201 | 06:50 |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config: WIP: Handle deployment update for DPDK nic changes https://review.openstack.org/373680 | 06:50 |
| *** bana_k has quit IRC | 06:52 | |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config: WIP: Handle deployment update for DPDK nic changes https://review.openstack.org/373680 | 06:52 |
| openstackgerrit | Merged openstack/tripleo-ui: Retrieve zaqar websocket url from keystone https://review.openstack.org/372499 | 06:55 |
| *** limao has quit IRC | 06:58 | |
| shardy | jaosorior: Hi - you need to submit a patch to get the script added to RDO packaging AFAIK | 07:03 |
| shardy | I think the repo where the specs are maintained moved a while back, so I'd ask the latest process in #rdo | 07:03 |
| *** limao has joined #tripleo | 07:04 | |
| matbu | bandini: is there a fix for "Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml" ? | 07:05 |
| jaosorior | shardy: but I'm still developing that script. I'm trying to test the deployment and the script at the same time :/ | 07:05 |
| bandini | matbu: there is a workaround: pushd <tht-directory>; swift download overcloud; popd | 07:05 |
| bandini | it got me past the issue | 07:05 |
| shardy | matbu: see https://bugs.launchpad.net/tripleo/+bug/1624727 | 07:05 |
| openstack | Launchpad bug 1624727 in tripleo "Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml" [Critical,Triaged] | 07:05 |
| matbu | shardy: bandini thx | 07:06 |
| shardy | matbu: can you please add details of how you're reproducing to the bug | 07:06 |
| shardy | I still cannot reproduce the issue locally, but am aiming to figure out a fix when I can | 07:06 |
| * shardy will probably have to rebuild his undercloud | 07:06 | |
| matbu | shardy: i'm wondering, i used to copy the tht dir in $HOME | 07:07 |
| shardy | jaosorior: you can test locally with a locally built package or DeployArtifacts, but in the gate, we test with packages | 07:07 |
| shardy | so I'm not sure what to tell you | 07:07 |
| shardy | add a placeholder script, add it to the package, then iterate on it? | 07:07 |
| shardy | or prove it locally then add it to the package | 07:07 |
| matbu | shardy: but it's producible in the upgrade M to N workflow | 07:08 |
| shardy | matbu: are you sure you have the latest tripleo-common? | 07:08 |
| jaosorior | shardy: yeah, I'm talking about local testing | 07:08 |
| shardy | there was a bug specific to updating the plan which d0ugal fixed recently | 07:08 |
| shardy | matbu: but thanks, I'll try again w/upgrade/update and see if I can reproduce | 07:08 |
| * d0ugal hopes he fixed them all | 07:08 | |
| jaosorior | shardy: not sure where the spec is for the package in the tripleo.sh --delorean-build workflow | 07:08 |
| matbu | shardy: i deploy a "current-passed-ci" , so something probably not really up to date | 07:09 |
| shardy | matbu: ack, sounds like you may not have the fixes for update yet then | 07:09 |
| shardy | matbu: can you confirm the tripleo-common version? | 07:09 |
| shardy | (and tripleoclient) | 07:09 |
| shardy | d0ugal can then help figure out if you have the needed fixes (sounds like probably not) | 07:10 |
| matbu | shardy: python-tripleoclient-5.1.1-0.20160920215837.fdbb7be.el7.centos.noarch openstack-tripleo-common-5.1.1-0.20160920134327.2d87e96.el7.centos.noarch | 07:10 |
| *** chem has joined #tripleo | 07:11 | |
| *** zoli_gone-proxy is now known as zoliXXL | 07:11 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Fix the default plan creation https://review.openstack.org/371347 | 07:12 |
| *** zoliXXL is now known as zoli|wfh | 07:12 | |
| openstackgerrit | Dougal Matthews proposed openstack/instack-undercloud: Ensure that the default plan was created successfully https://review.openstack.org/373446 | 07:15 |
| shardy | jaosorior: you can build the package directly via dlrn in dev mode, then iterate on the spec file locally I think | 07:16 |
| *** akuznetsov has joined #tripleo | 07:16 | |
| shardy | . ~/tripleo/delorean/venv/activate && dlrn --config-file projects.ini --dev --package-name openstack-keystone | 07:16 |
| shardy | for example | 07:16 |
| shardy | you can see where it's pulling down the packaging stuff from | 07:16 |
| b00tcat | Hi, can someone review this? https://review.openstack.org/#/c/373352/ :-) it's a biggie though | 07:17 |
| shardy | b00tcat: will do, thanks for the update :) | 07:18 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names https://review.openstack.org/366529 | 07:20 |
| *** akuznetsov has quit IRC | 07:20 | |
| *** ebarrera has joined #tripleo | 07:23 | |
| *** jlinkes has joined #tripleo | 07:23 | |
| *** zoli|wfh is now known as zoli_gone-proxy | 07:24 | |
| *** zoli_gone-proxy is now known as zoliXXL | 07:30 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the unused service_host arg from node registration https://review.openstack.org/326036 | 07:32 |
| *** jpena|off is now known as jpena | 07:34 | |
| *** chem has quit IRC | 07:35 | |
| *** chem has joined #tripleo | 07:35 | |
| *** jpich has joined #tripleo | 07:39 | |
| *** abehl has joined #tripleo | 07:41 | |
| *** akuznetsov has joined #tripleo | 07:50 | |
| *** apetrich has quit IRC | 07:50 | |
| *** apetrich has joined #tripleo | 07:52 | |
| *** akuznetsov has quit IRC | 07:55 | |
| *** ohamada has joined #tripleo | 07:55 | |
| *** akuznetsov has joined #tripleo | 07:55 | |
| *** panda|zZ is now known as panda | 07:56 | |
| *** dsariel has joined #tripleo | 07:59 | |
| *** oshvartz has quit IRC | 08:02 | |
| *** athomas has joined #tripleo | 08:03 | |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient: Upgrade needs to create the keystone credential https://review.openstack.org/374600 | 08:04 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient: Upgrade needs to create the keystone credential https://review.openstack.org/374600 | 08:04 |
| *** rasca has quit IRC | 08:07 | |
| *** rasca has joined #tripleo | 08:08 | |
| dsariel | hi, is there a way to install controller, compute and ceph on the same node (like packstack all-in-one)? | 08:08 |
| shardy | dsariel: Yes, you can add the compute & ceph OSD services to the ControllerServices list | 08:10 |
| shardy | dsariel: e.g see: | 08:10 |
| shardy | https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/multinode.yaml#L6 | 08:10 |
| shardy | http://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html | 08:11 |
| dsariel | shardy, awesome! thanks a lot :-) | 08:12 |
| shardy | dsariel: np | 08:12 |
| shardy | dsariel: we're testing the controller+compute single node setup in CI, but not with the ceph OSD co-located | 08:13 |
| shardy | in theory it should work tho, let us know how you get on :) | 08:13 |
| sshnaidm | slagle, hi | 08:18 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Return the result of create_plan in create_deployment_plan workflow https://review.openstack.org/371348 | 08:18 |
| d0ugal | sshnaidm: I'd guess slagle wont be awake for a few hours yet - unless he is in a different timezone than normal. | 08:20 |
| sshnaidm | d0ugal, ok :) | 08:20 |
| *** dmacpher has quit IRC | 08:22 | |
| *** dsneddon has quit IRC | 08:24 | |
| *** tremble has joined #tripleo | 08:37 | |
| *** dsneddon has joined #tripleo | 08:38 | |
| *** derekh has joined #tripleo | 08:41 | |
| *** zoliXXL is now known as zoli|wfh | 08:41 | |
| jaosorior | anybody with a running deployment that can answer something quick for me? | 08:44 |
| *** pkovar has joined #tripleo | 08:46 | |
| *** pkovar has quit IRC | 08:47 | |
| *** pkovar has joined #tripleo | 08:48 | |
| jistr | jaosorior: undercloud or overcloud? i have only undercloud atm | 08:48 |
| *** akuznetsov has quit IRC | 08:48 | |
| *** akuznetsov has joined #tripleo | 08:48 | |
| *** akuznetsov has quit IRC | 08:49 | |
| jaosorior | jistr: undercloud is fine | 08:50 |
| jaosorior | jistr: do you have glance-api-paste.ini in /etc/glance ? | 08:51 |
| jistr | jaosorior: hmm no it's not there | 08:51 |
| jaosorior | jistr: where is glance getting that config from? | 08:52 |
| jistr | hmm i think there are defaults somewhere | 08:52 |
| jaosorior | Jokke_: are you around? | 08:52 |
| *** hewbrocca-afk is now known as hewbrocca | 08:52 | |
| *** akuznetsov has joined #tripleo | 08:53 | |
| mcornea | jistr: jaosorior I think this is the default: /usr/share/glance/glance-api-dist-paste.ini | 08:55 |
| jistr | mcornea: ah yea, thanks! | 08:55 |
| *** akuznetsov has quit IRC | 08:57 | |
| jaosorior | mcornea: thanks | 08:58 |
| jaosorior | jistr, mcornea: Do you have any idea how we point to that in puppet? can't seem to find it | 08:59 |
| jistr | i don't think we point to that in puppet, those are not meant for editing afaik... i'm not sure how are they applied in glance though, e.g. if they are merged with the content in /etc, or if they are only used if there's not a matching "overriding file" in /etc | 09:00 |
| *** rasca has quit IRC | 09:10 | |
| *** dsneddon has quit IRC | 09:10 | |
| *** milan has joined #tripleo | 09:10 | |
| *** dsneddon has joined #tripleo | 09:11 | |
| *** rasca has joined #tripleo | 09:14 | |
| *** yamahata has quit IRC | 09:17 | |
| *** akuznetsov has joined #tripleo | 09:19 | |
| *** akuznetsov has quit IRC | 09:23 | |
| *** stendulker has joined #tripleo | 09:27 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 09:28 |
| *** milan is now known as milan|f00d | 09:28 | |
| *** oshvartz has joined #tripleo | 09:29 | |
| *** chem has quit IRC | 09:37 | |
| *** chem has joined #tripleo | 09:38 | |
| *** chem has quit IRC | 09:39 | |
| *** chem has joined #tripleo | 09:39 | |
| *** andrey-mp has joined #tripleo | 09:39 | |
| tbarron | marios: i updated https://review.openstack.org/#/c/358525 with results of my cephfs/manila overcloud deploy attempt | 09:39 |
| andrey-mp | Hi! anybody knows where to get list of all nodes in post hook on cotroller? | 09:40 |
| tbarron | marios: /etc/manila/manila.conf is getting edited as we expect and systemd controlled services (api, scheduler) are started but not pcs controlled share service, perhaps b/c the deploy itself is failing earlier, on the db sync: error running 'manila manage-db sync'. | 09:41 |
| tbarron | marios: that db sync code looks to be in manila-puppet ? merged a long time ago. | 09:42 |
| shardy | andrey-mp: do you need the ips/names running a specific service, or really all nodes? | 09:42 |
| andrey-mp | shardy: names of nodes | 09:42 |
| shardy | andrey-mp: the hosts file contains all the nodes, and we write hiera with $service_node_names for each service | 09:43 |
| andrey-mp | shardy: then i can parse name and do specific steps for each | 09:43 |
| tbarron | marios: one thing I notice is that cinder runs it (on the lead controller) when api service is started, but manila is waiting to run the db sync when scheduler service is started | 09:43 |
| marios | tbarron: great thanks for update ... I don't think I'll have time to look at that ftr... I believe Jokke_ is driving that | 09:43 |
| tbarron | marios: the cinder way makes sense intuitively since api service starts first and it interacts with DB | 09:43 |
| panda | 2016-09-22 09:03:25Z [overcloud-Controller-cuwh4w2oe7qy-0-6ohgoxw2xx56]: CREATE_FAILED Engine went down during stack CREATE | 09:43 |
| panda | wut ? | 09:43 |
| tbarron | marios: sure, i will ping Jokke_ | 09:44 |
| shardy | panda: did you run out of RAM? | 09:44 |
| tbarron | Jokke_: updates on cephfs backend testing for manila in backlog above ^^^^^^ | 09:44 |
| shardy | the OOM killer often decides heat-engine is a good candidate for killing when that happens | 09:44 |
| panda | shardy: yep, dmesg is full of OOM ... :( | 09:44 |
| marios | tbarron: thanks - but getting the info on the review would be useful regardless, and then you can just point at it, instead of repeating yourself :) | 09:44 |
| panda | shardy: I thought 32G were enough for ha | 09:44 |
| *** florianf has quit IRC | 09:45 | |
| shardy | panda: how much memory does your undercloud have? | 09:45 |
| tbarron | marios: yeah, i've got the info in the review | 09:45 |
| marios | tbarron: right thanks just looked, i will try and cycle back to the review later and see if i can spot something | 09:45 |
| shardy | you could add some swap if it's nearly enough | 09:45 |
| panda | shardy: 8G | 09:45 |
| tbarron | marios: on the review that's outstanding even if the issue is elsewhere | 09:45 |
| marios | tbarron: so you only get this trying manila-cephfs backend? | 09:45 |
| shardy | panda: I run a 32G dev box, and IMO it's not enough to do a 3 controller HA deployment with compute | 09:46 |
| shardy | you might be better doing a single controller with pacemaker enabled | 09:46 |
| tbarron | marios: no, i see it with netapp too, just had been hitting blockeers that kept the deploy from getting this far before | 09:46 |
| tbarron | marios: i'll put a note there | 09:46 |
| andrey-mp | shardy: thanks. hosts file realy contains all nodes. but service_node_names I can't find in my installation (I installed Mitaka version...) | 09:46 |
| marios | tbarron: i see so then i will try harder to have a look today :) | 09:46 |
| panda | shardy: that's not what I need to reproduce ... :( | 09:47 |
| tbarron | marios: indicating that the issue is not cephfs specific | 09:47 |
| shardy | andrey-mp: ah, yeah the node_names thing is new for Newton | 09:47 |
| panda | shardy: but thanks for the advices | 09:47 |
| andrey-mp | shrady: ok, thank you | 09:47 |
| shardy | panda: you might want to consider disabling some services in the overcloud deployment then | 09:47 |
| shardy | so you can do a minimal 3 node HA deployment | 09:47 |
| panda | shardy: how ? | 09:48 |
| *** florianf has joined #tripleo | 09:49 | |
| shardy | http://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html | 09:49 |
| tbarron | marios: i may play around with manila-puppet a bit to make the manila stuff look more like cinder unless you think that's a dead end approach | 09:49 |
| panda | shardy: wonderful, thanks! | 09:49 |
| marios | tbarron: i think it would be easier to fix the problem in the tht/puppet-tripleo (s/easier/faster) than trying to fix something in puppet-manila... unless it really is broken. so the answer may be both, tht/tripleo in the shorter term (like rc2 fix ) | 09:50 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Sync worker-config.yaml with low-memory-usage.yaml https://review.openstack.org/374660 | 09:50 |
| tbarron | marios: k, good advice! | 09:50 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Make sure major upgrade script fails. https://review.openstack.org/366623 | 09:52 |
| tbarron | marios: the only reason i was looking at puppet-manila was that i only saw the DB sync code there, not in THT, but I'll look again | 09:52 |
| *** tosky has joined #tripleo | 09:57 | |
| panda | I'm also seeing these in today's ha periodic job Error: /Stage[main]/Swift::Storage::Account/Swift::Storage::Generic[account]/Swift::Service[swift-account-replicator]/Service[swift-account-replicator]: Cannot allocate memory - fork(2)m | 10:00 |
| panda | memory requirements are generally increasing ? | 10:01 |
| *** andrey-mp has quit IRC | 10:01 | |
| shardy | panda: folks keep integrating new services, and quite a few ended up enabled by default | 10:03 |
| shardy | so, yes, memory requirements have crept up over the last couple of cycles | 10:04 |
| shardy | happily we now have a way for folks to easily disable stuff they don't need | 10:04 |
| shardy | if it's in a CI job we may need to investigate further tho, as we shouldn't have suddenly needed more memory very recently | 10:05 |
| panda | shardy: is there something we can disable in the ha periodic job ? There coverage should be considered too. | 10:06 |
| shardy | panda: hard question to answer, I would assume with the new HA lite architecture it's less critical we have full coverage of those services not managed by pacemaker | 10:08 |
| shardy | and/or we can have a mixture of scenarios which achieve full coverage | 10:08 |
| shardy | that said, if the job was working, and now it's not, there may be a regression somewhere | 10:08 |
| *** andrey-mp has joined #tripleo | 10:08 | |
| shardy | I'd suggest doing a local HA deployment and looking at where the memory is going before stuff gets killed | 10:09 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Display error message when socket is closed https://review.openstack.org/374669 | 10:13 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Provide more information when 'node provide' fails https://review.openstack.org/374670 | 10:13 |
| *** jprovazn has quit IRC | 10:15 | |
| panda | shardy: that is launch a deploy and look at ps continuously until we get OOM or the fork error, or configure dstat to give per-process statistics (don't know if it's possible) | 10:17 |
| shardy | panda: your choice, we run dstat in CI, but I tend to use top or htop locally | 10:19 |
| shardy | speaking of which gnocchi-statsd is hogging all my CPU on an otherwise idle overcloud :( | 10:19 |
| jaosorior | ayoung: let me know when you're online. I need some httpd help | 10:21 |
| *** limao has quit IRC | 10:21 | |
| *** andrey-mp has quit IRC | 10:21 | |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1626473 | 10:22 |
| openstack | Launchpad bug 1626473 in tripleo "gnocchi is eating all my CPU :(" [High,Triaged] | 10:22 |
| *** akuznetsov has joined #tripleo | 10:22 | |
| shardy | be interested to see if anyone else can reproduce that | 10:22 |
| shardy | pradk: ^^ FYI | 10:22 |
| *** dtantsur|afk is now known as dtantsur | 10:26 | |
| *** akuznetsov has quit IRC | 10:27 | |
| *** fzdarsky has joined #tripleo | 10:32 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Configure heat engine to not use convergence https://review.openstack.org/333890 | 10:34 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Stop using deprecated network range https://review.openstack.org/343443 | 10:34 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Stacks and Resources data storing in app state https://review.openstack.org/374227 | 10:37 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Add settings to general config https://review.openstack.org/371144 | 10:38 |
| *** mburned_out is now known as mburned | 10:41 | |
| bandini | marios: might want to keep an eye on this one https://bugs.launchpad.net/tripleo/+bug/1626452 | 10:42 |
| openstack | Launchpad bug 1626452 in tripleo "M/N upgrades - Error: Could not find class ::tripleo::trusted_cas" [Critical,New] | 10:42 |
| *** dprince has joined #tripleo | 10:42 | |
| marios | bandini: thanks | 10:43 |
| marios | bandini:ouch that one sounds messy. | 10:43 |
| bandini | marios: yeah I am a little concerned tbh. will keep trying until I get at least the init step working | 10:44 |
| marios | bandini: so we may need to OS::Heat::None all the services... | 10:44 |
| bandini | am afraid so, unless we can come up with a better plan | 10:44 |
| marios | bandini: 'new' i mean... yeah... ok, well thanks for the heads up for now | 10:45 |
| bandini | np ;) | 10:45 |
| marios | bandini: about that other one... i am guessing obviously yes we want the fsid to stay the same @ https://review.openstack.org/#/c/374600/2 | 10:46 |
| marios | bandini: so it may/should be its own review, esp if it becomes more involved to make it so | 10:47 |
| *** hjensas has joined #tripleo | 10:47 | |
| *** kberger has quit IRC | 10:47 | |
| bandini | marios: let's see. if it is simple enough we might want to keep it in a single review otherwise we can split, sure | 10:48 |
| bandini | for now I am doing just workaround until I at least get the init step working | 10:48 |
| *** kberger has joined #tripleo | 10:48 | |
| bandini | then I will start working on proper patches for all the issues | 10:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/373722 | 10:48 |
| marios | bandini: thanks... seems your about a day ahead of the osp10 packages atm :) we started hitting the things you complained about yesterday morning | 10:48 |
| marios | bandini: so far nothing _too_ nasty... even that OS::Heat::None one in the worst case is just a list in the env file | 10:49 |
| bandini | marios: yeah let's cross fingers. I will join the call today (sorry about yesterday was a bit poorly) | 10:50 |
| marios | bandini: scrum at ... right was going to say, welcome to join, i will harrass you later too | 10:50 |
| bandini | aye ;) | 10:51 |
| openstackgerrit | Merged openstack/python-tripleoclient: Use the passed in workflow when creating or updating a plan https://review.openstack.org/374201 | 10:53 |
| *** ohamada has quit IRC | 10:55 | |
| *** ohamada has joined #tripleo | 10:55 | |
| panda | when is newton release deadline ? | 10:55 |
| *** hjensas has quit IRC | 10:57 | |
| *** hjensas has joined #tripleo | 10:59 | |
| *** hjensas has quit IRC | 10:59 | |
| *** hjensas has joined #tripleo | 10:59 | |
| panda | maybe we can just bump undercloud memory to 12G temporarily until release .. | 11:00 |
| *** pkovar has quit IRC | 11:01 | |
| panda | sshnaidm: ^ | 11:01 |
| panda | and then take countermeasures | 11:02 |
| *** thrash|g0ne is now known as thrash | 11:02 | |
| *** fzdarsky_ has joined #tripleo | 11:06 | |
| *** pkovar has joined #tripleo | 11:06 | |
| *** hjensas has quit IRC | 11:07 | |
| sshnaidm | panda, I don't know, to focus on memory leaking in heat is also the option | 11:08 |
| panda | sshnaidm: I don't know how much time do we have to do this ... for example I cant get result on the ipv6 ha job right now, and it should be tested before relase | 11:08 |
| *** dprince has quit IRC | 11:10 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Add metricd workers support in gnocchi https://review.openstack.org/374704 | 11:11 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Setting to 1 GnocchiMetricdWorkers https://review.openstack.org/374709 | 11:14 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 11:15 |
| *** jprovazn has joined #tripleo | 11:15 | |
| *** fzdarsky_ has quit IRC | 11:17 | |
| *** fzdarsky has quit IRC | 11:17 | |
| *** fzdarsky has joined #tripleo | 11:17 | |
| *** lucas-afk is now known as lucasagomes | 11:18 | |
| sshnaidm | panda, I think it's better to discuss it here | 11:19 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Add metricd workers support in gnocchi https://review.openstack.org/374704 | 11:19 |
| *** jistr is now known as jistr|mtg | 11:20 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Setting to 1 GnocchiMetricdWorkers https://review.openstack.org/374709 | 11:21 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 11:21 |
| *** akuznetsov has joined #tripleo | 11:23 | |
| *** lblanchard has quit IRC | 11:23 | |
| *** stendulker has quit IRC | 11:24 | |
| shardy | panda: https://releases.openstack.org/newton/schedule.html | 11:25 |
| shardy | panda: we're aiming to cut RC2 (and branch stable/newton) next week, the the final release will hopefully be aligned with the main newton release during w/c 3rd October | 11:27 |
| shardy | we are following the cycle-trailing model, so it's permitted to declare our final release later for some repos, but it'd be best if we can release very close to the time of the rest of OpenStack | 11:27 |
| *** akuznetsov has quit IRC | 11:28 | |
| panda | sshnaidm: I don't think we can solve the memory crysis in one week efficiently for everyone ... let's hope it has been a one time problem, if not it's probably best to increase memory for this last week | 11:35 |
| panda | shardy: thanks. | 11:35 |
| openstackgerrit | Merged openstack/tripleo-common: Return the result of create_plan in create_deployment_plan workflow https://review.openstack.org/371348 | 11:38 |
| *** bfournie has quit IRC | 11:42 | |
| *** pkovar has quit IRC | 11:47 | |
| *** ccamacho is now known as ccamacho|lunch | 11:47 | |
| *** coolsvap has quit IRC | 11:52 | |
| *** jpena is now known as jpena|lunch | 11:53 | |
| *** zigo has quit IRC | 11:54 | |
| *** zigo has joined #tripleo | 11:58 | |
| *** zigo is now known as Guest83601 | 11:59 | |
| *** trown|outtypewww is now known as trown | 12:03 | |
| *** Guest83601 has quit IRC | 12:03 | |
| *** pkovar has joined #tripleo | 12:05 | |
| *** jayg|g0n3 is now known as jayg | 12:09 | |
| *** mbozhenko has joined #tripleo | 12:10 | |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient: Upgrade needs to create the keystone credential https://review.openstack.org/374600 | 12:11 |
| *** cdearborn has joined #tripleo | 12:11 | |
| *** zigo_ has joined #tripleo | 12:12 | |
| mbozhenko | Hello all. I want to start to work with TripleO and contribute to it. Can anyone kick me into some sort of quick start doc for setting up dev env? | 12:12 |
| honza | jtomasek: do you know if there is a way to get all of the resources for a stack in one API call including all the attributes? i don't think there is and it's making my code ugly :( | 12:15 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Stacks and Resources data storing in app state https://review.openstack.org/374227 | 12:16 |
| *** zigo_ has quit IRC | 12:17 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Switch default image location back to CentOS CDN https://review.openstack.org/374753 | 12:17 |
| *** zigo_ has joined #tripleo | 12:18 | |
| *** rhallisey has joined #tripleo | 12:19 | |
| *** milan|f00d is now known as milan | 12:20 | |
| *** rodrigods has quit IRC | 12:24 | |
| *** rodrigods has joined #tripleo | 12:24 | |
| *** akuznetsov has joined #tripleo | 12:25 | |
| bandini | marios: do you have any tips as to how I can find all the services affected by https://bugs.launchpad.net/tripleo/+bug/1626452, maybe I should just noop them all in the init step? | 12:27 |
| openstack | Launchpad bug 1626452 in tripleo "M/N upgrades - Error: Could not find class ::tripleo::trusted_cas" [Critical,New] | 12:27 |
| shardy | mbozhenko: Hi, welcome! | 12:27 |
| shardy | mbozhenko: We have some docs here http://docs.openstack.org/developer/tripleo-docs/ | 12:27 |
| bandini | marios: so far I got three (snmp, trusted_cas and libvirt) | 12:27 |
| shardy | mbozhenko: if you want a quick summary of how I set up my environment (uses a script from our CI to automate a few steps from the docs), see here: | 12:27 |
| shardy | http://paste.fedoraproject.org/432545/47454728/ | 12:28 |
| shardy | mbozhenko: ideally you need a machine with >= 32G ram to test the virt setup I describe there | 12:28 |
| *** florianf has quit IRC | 12:29 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names https://review.openstack.org/366529 | 12:29 |
| shardy | mbozhenko: also there is the tripleo-quickstart tool, which is an ansible approach to bootstrapping a tripleo environment | 12:29 |
| shardy | https://github.com/openstack/tripleo-quickstart/ | 12:29 |
| *** akuznetsov has quit IRC | 12:30 | |
| *** bfournie has joined #tripleo | 12:35 | |
| jtomasek | honza: there is not, how many of the resources do you actually need? | 12:35 |
| jtomasek | honza: I thought, you need just a specific one | 12:35 |
| *** pkovar has quit IRC | 12:36 | |
| honza | jtomasek: just one, code is not that bad --- there just needs to be some logic in the reducer to prevent overwriting | 12:36 |
| openstackgerrit | Chris Jones proposed openstack/tripleo-quickstart: Add ssh option IdentitiesOnly. https://review.openstack.org/374769 | 12:36 |
| *** jistr|mtg is now known as jistr | 12:36 | |
| *** gfidente has joined #tripleo | 12:37 | |
| jtomasek | honza: you dispatch the action for that single resource and in reducer you can update that specific resource with additional attributes | 12:37 |
| jtomasek | honza: what do you mean by overwriting? | 12:37 |
| honza | jtomasek: yep, but that API call usually comes back before the fetchResources one so fetchResources overwrites your data | 12:38 |
| honza | jtomasek: no big deal | 12:38 |
| honza | race conditions ftw | 12:38 |
| *** jeckersb_gone is now known as jeckersb | 12:38 | |
| jtomasek | honza: I see | 12:39 |
| marios | bandini: so noop all in the init should be ok because we don't run the postconfig ... *i think* ... not sure how we'd come up with a list of new services... thinking | 12:43 |
| marios | bandini: i mean i know we don't run the postconfig cos we noop it, but i mean i think it should be ok | 12:43 |
| marios | bandini: i.e wont delete all the things/services | 12:44 |
| marios | bandini: we could compare /var/lib/tripleo/installed-packages/overcloud_controller_pacemaker1 2 3 etc on the nodes for list of thigns already there | 12:45 |
| *** pkovar has joined #tripleo | 12:45 | |
| marios | bandini: but noop all sounds better if it works for now | 12:45 |
| bandini | marios: yeah I just did a noop for all of them and I have the init step terminating successfully | 12:45 |
| marios | bandini: cool. do you still possess an overcloud? with things running on it? | 12:45 |
| bandini | marios: looking right now ;) | 12:46 |
| bandini | marios: yep still functional | 12:46 |
| marios | bandini: thanks bandini even more beer for you | 12:46 |
| *** andrey-mp has joined #tripleo | 12:47 | |
| mbozhenko | shardy: thank you! | 12:48 |
| *** zigo_ has quit IRC | 12:48 | |
| marios | bandini: but for converge we would need a list of things not there already | 12:49 |
| marios | bandini: so we will still have to solve that | 12:49 |
| mbozhenko | shardy: I will try and let you know on the outcomes | 12:49 |
| bandini | marios: but during converge the puppet-tripleo rpms are the newton ones so it should not matter, right? | 12:50 |
| *** rlandy has joined #tripleo | 12:50 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: Noop all the TripleO::Services during the major-upgrade-pacemaker-init step https://review.openstack.org/374788 | 12:50 |
| marios | bandini: reading bug again, i thought it was a case of new services getting started ... ah no it is missing puppet-tripleo being pulled in by the new service templates | 12:51 |
| *** zigo_ has joined #tripleo | 12:51 | |
| marios | bandini: and they aren't nooped./.. yeah fine so when we have the puppet-tripleo should be good | 12:51 |
| shardy | bandini: Question re ^^ | 12:51 |
| shardy | bandini: why noop the services, vs the deployment applying them? | 12:51 |
| shardy | the actual services don't do anything except contain data | 12:51 |
| marios | shardy: this is just init step that fails and it fails because the puppet-tripleo isn't there yet, in this step of the upgrade | 12:52 |
| shardy | probably you want to noop the deployment steps in puppet/post.yaml (or rather that entire stack) | 12:52 |
| marios | shardy: so the tht is referencing classses in puppet-tripleo that aren't there | 12:52 |
| shardy | marios: but those resources don't apply puppet | 12:52 |
| shardy | they just contain some text | 12:52 |
| b00tcat | Is there any way to re-run *only* the puppet code after having provisioned a heat stack with tripleo? provisioning the overcloud takes a huge amount of time, and if I have -let's say- a syntax error it'd be a pita ^^" | 12:52 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.j2.yaml#L455 | 12:53 |
| marios | shardy: bandini so am not clear why are we getting the error for the missing class then still | 12:53 |
| shardy | marios: Can we instead Noop OS::TripleO::PostDeploySteps? | 12:53 |
| marios | shardy: we already do | 12:53 |
| shardy | hmm | 12:53 |
| *** Goneri has joined #tripleo | 12:53 | |
| bandini | shardy: so we already noop some stuff https://github.com/openstack/tripleo-heat-templates/blob/master/environments/major-upgrade-pacemaker-init.yaml | 12:54 |
| marios | shardy: i mean we oh i mean we noop the ControllerPostDeployment... all nodes | 12:54 |
| shardy | marios: that may be the issue, we've reworked how *PostDeployment works | 12:54 |
| bandini | shardy: so you think just nooping OS::TripleO::PostDeploySteps will do? | 12:54 |
| marios | https://github.com/openstack/tripleo-heat-templates/blob/master/environments/major-upgrade-pacemaker-init.yaml like this shardy | 12:54 |
| shardy | so you may need to noop the PostDeploySteps instead | 12:54 |
| marios | shardy: right sounds like it may be it :) | 12:54 |
| shardy | marios: sec, let me show you the patch | 12:54 |
| jistr | hehe i was just looking into it :) | 12:55 |
| jistr | indeed | 12:55 |
| marios | bandini: so it sounds like it was actually trying to deploy the services | 12:55 |
| jistr | yea | 12:55 |
| marios | bandini: cos our 'don't run the post deploy config puppet stuff' isn't working now | 12:55 |
| jistr | i have a patch actually :) | 12:55 |
| shardy | https://review.openstack.org/#/c/365763/14/overcloud.yaml | 12:55 |
| marios | jistr: cool link | 12:55 |
| jistr | marios: lemme write a commit message :) | 12:56 |
| shardy | marios: that's probably your issue, nooping PostDeployment stuff won't do anything anymore | 12:56 |
| marios | jistr: haha :D | 12:56 |
| bandini | jistr: feel free to slap it on top of https://review.openstack.org/#/c/374788/ | 12:56 |
| marios | shardy: fantastic thanks shardy bandini jistr | 12:56 |
| bandini | thanks shardy jistr | 12:56 |
| shardy | np! :) | 12:56 |
| gfidente | ccamacho|lunch, thanks for the ext4/liberty comments! | 12:57 |
| gfidente | thanks a lot | 12:57 |
| *** david-lyle has joined #tripleo | 12:58 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: No-op Puppet for upgrades/migrations according to composable roles https://review.openstack.org/374791 | 12:58 |
| jistr | marios bandini ^^ | 12:58 |
| gfidente | I am still unsure at this point why what seems a grub issue should not be seen with newer versions of openstack | 12:58 |
| *** andrey-mp has left #tripleo | 12:58 | |
| jistr | meh copy pasta | 12:58 |
| gfidente | but I think we spent enough time and resources on it, given it's going EOL | 12:58 |
| EmilienM | panda: something must goes wrong in ipv6 experimental job, it's timeouting all the time :( | 12:59 |
| EmilienM | bnemec: were you lucky in your tests? | 12:59 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: No-op Puppet for upgrades/migrations according to composable roles https://review.openstack.org/374791 | 12:59 |
| marios | jistr: thanks | 12:59 |
| matbu | bandini: cool for https://review.openstack.org/#/c/374788 did you test it ? | 12:59 |
| jistr | marios, bandini ^^ more like that | 12:59 |
| *** ccamacho|lunch is now known as ccamacho | 12:59 | |
| jaosorior | EmilienM: have you ever manipulated HTTP headers with httpd? | 12:59 |
| matbu | bandini: and does the # close bug works better ? ;) | 12:59 |
| *** cylopez has joined #tripleo | 12:59 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4 https://review.openstack.org/371209 | 12:59 |
| marios | jistr: heh, i went looking for puppet/post.yaml i thought it was a special noop template :) | 13:00 |
| bandini | matbu: I tested it but we're going for jistr review here https://review.openstack.org/374791 | 13:00 |
| EmilienM | jaosorior: with apache you mean? | 13:00 |
| jistr | marios: yea just my copypasta :)) | 13:00 |
| ccamacho | gfidente np man, just bad I dont have something to fix it :( | 13:00 |
| jaosorior | yeah | 13:00 |
| *** jpena|lunch is now known as jpena | 13:00 | |
| EmilienM | jaosorior: you need a l7 proxy | 13:01 |
| EmilienM | mod_headers does it in apache | 13:01 |
| matbu | bandini: jistr ha cool :) | 13:01 |
| EmilienM | it's well documented here: http://httpd.apache.org/docs/2.0/mod/mod_headers.html | 13:01 |
| jaosorior | EmilienM: yeah, I've been checking that out | 13:02 |
| EmilienM | jaosorior: I used it in the past yes | 13:02 |
| EmilienM | jaosorior: let me know if you need more help | 13:02 |
| panda | EmilienM: this morning even periodic jobs had memory problems during tests. I tried to reproduce the build locally, but apparently 8G for the undercloud are not enough anymore, and I had OOM killer shut down heat-engine | 13:02 |
| jaosorior | EmilienM: but haven't figured out how to get apache to accept underscores in HTTP headers in an elegant way | 13:02 |
| jaosorior | the only way I've found out is to do a bunch of setenvif with each header, and then add it with mod_headers | 13:02 |
| jistr | bandini, marios, matbu: i'm trying to think about some possible drawbacks, hopefully there wouldn't be any, as the puppet only gets data generated from the service chains etc., but in itself it doesn't produce any data to be consumed elsewhere, so i hope it would work the same way as we had for K/L and L/M, simply not running the puppet, but still producing the Hiera etc. Hopefully that's ok even now that we | 13:03 |
| jistr | use composability. | 13:03 |
| jaosorior | EmilienM: basically I'm trying to get Glance to work with httpd... but glance uses a bunch of headers that contain underscores | 13:03 |
| jaosorior | EmilienM: so apache filters those | 13:03 |
| jaosorior | EmilienM: the only documented way to fix that is http://httpd.apache.org/docs/trunk/env.html#fixheader which I would have to do for each header...and I was looking for a more compact solution | 13:03 |
| EmilienM | panda: ok | 13:03 |
| marios | jistr: it looks much simpler... if the end result is same (none of the post-config puppet run) it should be fine. So on converge we will end up with any new services that weren't default/existing in 9 as well | 13:03 |
| bandini | jistr: it is certainly fine for the init step, I will give feedback as soon as I manage to get past init ;) | 13:04 |
| marios | jistr: (as side note) | 13:04 |
| EmilienM | jaosorior: the modea headers can help you to do that | 13:04 |
| matbu | bandini: jistr marios yep me too, testing the tripleoclient fix right now | 13:04 |
| jaosorior | EmilienM: yeah... I've been trying for 3 hours now. Haven't gotten mod_headers to do what I need | 13:05 |
| marios | jistr: yeah this is what shardy was saying earlier... the tht will contain/define the new services, all of them, including ones we didn't have before. but it is just the config so as long as that isn't passed to puppet to run it should be fine | 13:05 |
| panda | EmilienM: we were discussing about bumpping temporarily the undercloud memory for tripleo-ci to 12G until release, and then change the jobs configuration, but we'lre trying to understand if it's only a peek of memory or it's a permanent requirement | 13:05 |
| jistr | marios: yea hopefully :) Btw any idea what services are the ones that are different between M and N? i saw a BZ email recently where tosky brought up that Sahara actually was default before but isn't anymore, and that it will probably cause some issues... | 13:06 |
| tosky | jistr: iirc Sahara is the only one which was enabled by default and not core, but maybe there was nother | 13:07 |
| marios | jistr: no this is what bandini and i were discussing earlier, when we thought we should noop individual 'new' services. we could compare the service chains/enabled_services with the lists from /var/lib/tripleo/installed-packages/overcloud_controller_pacemakerX | 13:07 |
| jistr | hmm right | 13:08 |
| *** jcoufal has joined #tripleo | 13:08 | |
| *** coolsvap has joined #tripleo | 13:08 | |
| *** zoli|wfh is now known as zoli|lunch | 13:09 | |
| marios | jistr: but perhaps we don't need to do that. i mean we let the config deploy 'stock' newton new services and all. or if we need to we can noop things i guess | 13:09 |
| marios | jistr: i mean on converge | 13:09 |
| jistr | marios: yea deploying new could hopefully be ok, it's maybe potential removing that we might need to pay more attention to | 13:10 |
| jistr | it's probably a bit complicated due to the fact that some people might want to remove Sahara while others would like to keep it... | 13:10 |
| tosky | marios: also, you may want to apply the new configuration before restarting the services | 13:11 |
| marios | jistr: keeping it *should* be ok, assuming we can just make it one of the enabled services | 13:12 |
| marios | tosky: do you mean, with sahara from M, if you want to upgrade to N with sahara, yu also need som more config? like a migration? | 13:12 |
| *** jaosorior has quit IRC | 13:12 | |
| marios | tosky: yes i see the note in the issue | 13:13 |
| *** jaosorior has joined #tripleo | 13:13 | |
| tosky | marios: if you deployed M, you have a certain set of plugins in sahara.conf (explicitely written down by TripleO); if you upgrade, even if you include the environment file, it seems that the new configuration is applied after the restart, not before | 13:13 |
| marios | jistr: tosky seems we are already tracking this on the lifecycle readme issues | 13:13 |
| tosky | and the restart fails because the set of available plugins in N is different (one is lacking) | 13:13 |
| tosky | marios: partially, it does not cover the case "we want to remove it" and apparently even the "yes, please keep it" case is buggy | 13:14 |
| marios | tosky: right this is what omri was hitting earlier i think (the plugin issue https://bugs.launchpad.net/tripleo/+bug/1615056 ) | 13:15 |
| openstack | Launchpad bug 1615056 in tripleo "M/N upgrade sahara-api fails to start." [Undecided,Fix released] - Assigned to Emilien Macchi (emilienm) | 13:15 |
| *** akuznetsov has joined #tripleo | 13:15 | |
| tosky | marios: ah, a manual fix | 13:15 |
| tosky | a bit... hackish | 13:15 |
| tosky | but I see | 13:15 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Switch default image location back to CentOS CDN https://review.openstack.org/374753 | 13:15 |
| marios | tosky: no no we won't go with a manual fix, | 13:16 |
| marios | tosky: i am just saying we are tracking this and we still need to work it out | 13:16 |
| tosky | marios: sorry, bad working on my side; the call to crudini is "manual" compared to the proper "converge the configuration to this point" provided by puppet | 13:16 |
| EmilienM | matbu: https://review.openstack.org/#/c/374600/ | 13:17 |
| marios | tosky: oh to remove the plugin you mean | 13:17 |
| EmilienM | why do you patch stable/newton? | 13:17 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/373722 | 13:17 |
| tosky | marios: yes | 13:17 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Add swap to the undercloud when using an overcloud image https://review.openstack.org/374809 | 13:18 |
| marios | tosky: well it has to happen during the controller upgrade step otherwise we can't bring up sahara-api | 13:19 |
| *** akuznetsov has quit IRC | 13:19 | |
| *** kjw3 has joined #tripleo | 13:19 | |
| matbu | EmilienM: well, cause i hit the issue, but yes i should patch master | 13:19 |
| marios | tosky: converge won't happen until after controllers, then computes and ceph nodes updated so sahara-api would be down that whole time | 13:19 |
| tosky | marios: right; but then why isn't the new configuration applied directly before restartin the services? I mean, the configuration coming from hiera? | 13:20 |
| marios | tosky: if mean if we just let it happen on converge as defined by the newton templates | 13:20 |
| marios | tosky: we don't want to apply the config at that point we need that to happen everywhere at once, on converge | 13:20 |
| *** pgadiya has quit IRC | 13:21 | |
| tosky | marios: and then you restart everything again? | 13:21 |
| marios | tosky: e.g. what if your config changes passwords? if we ran config on controller upgrade then we'd get the new passwords only on controllers | 13:21 |
| tosky | uhm, I see (I think) | 13:21 |
| tosky | so with that crudini line the case "please keep Sahara" should be fixed | 13:22 |
| marios | tosky: yeah there is also a controlled service restart after converge | 13:22 |
| marios | tosky: so it should solve the case of yeah keep sahara | 13:22 |
| tosky | still the case where Sahara needs to be removed should be handled somehow | 13:22 |
| marios | tosky: yes, we are tracking it so it will get done at some point. I mean, if it is OS::Heat::None now it should mean that nothing will be done to it so we may need to manually remove during one of the upgrade steps, probably the controller upgrade | 13:23 |
| tosky | marios: or at least disable it | 13:24 |
| marios | tosky: i mean nothing will be done to the existing installation | 13:24 |
| rhallisey | d0ugal, hey Dougal, is there a mistral workflow in place that will trigger updates? | 13:24 |
| rhallisey | or is the workflow to deploy again | 13:25 |
| shardy | rhallisey: the workflow is to run the deploy workflow again I think, with some tweaks to how we prepare/update the plan in tripleoclient | 13:26 |
| shardy | we update the stack instead of creating it | 13:26 |
| *** fultonj has joined #tripleo | 13:26 | |
| rhallisey | shardy, should we have an update workflow instead? | 13:27 |
| rhallisey | I do think it's similar to deploy, but the process itself is a series of steps | 13:28 |
| rhallisey | there is some variation | 13:28 |
| shardy | rhallisey: Yeah, there was some discussion about this yesterday | 13:28 |
| shardy | e.g how do we maintain the openstack overcloud update command | 13:28 |
| shardy | that probably will need a different workflow, as you say due to the breakpoints etc | 13:28 |
| shardy | I'm not sure on the status of that, but I have a feeling we have a gap there which needs to be fixed asap | 13:29 |
| shardy | rhallisey: so, we have updates (just update configuration or scale out) == deploy workflow | 13:29 |
| shardy | the update (package update for applying errata) == update workflow | 13:29 |
| rhallisey | from what I can tell, the current update is a redeloy | 13:29 |
| shardy | rhallisey: there's a bunch of stuff in there related to breakpoints which doesn't exist in the overcloud deploy path | 13:30 |
| shardy | and it changes UpdateIdentifier which triggers yum update | 13:30 |
| *** ayoung has quit IRC | 13:30 | |
| shardy | but the actual interaction with heat to start the update is the same as a deploy I guess | 13:30 |
| rhallisey | yea I agree, I'm just saying the 'old' update path isn't hooked into the new mistra; + swift workflow | 13:30 |
| social | shardy: slagle: how are composable roles going to update? what'll replace overcloud-without-mergepy.yaml ? | 13:30 |
| social | ah | 13:30 |
| * social just reads backlog | 13:31 | |
| shardy | rhallisey: Yeah, we either need to live with tripleoclient's existing functionality there | 13:31 |
| shardy | (which means updates won't work via the UI) | 13:31 |
| shardy | or wire it in via mistral | 13:31 |
| shardy | you're right | 13:31 |
| shardy | social: there's an overcloud.j2.yaml, which is rendered in a mistral action during plan creation | 13:31 |
| rhallisey | ok let's go with mistral then | 13:31 |
| shardy | social: you can see the rendered version via swift download overcloud overcloud.yaml | 13:32 |
| EmilienM | please add your rc2 patches on this gerrit topic: https://review.openstack.org/#/q/topic:tripleo/rc2 | 13:32 |
| shardy | overcloud-without-mergepy.yaml no longer exists, and it's been deprecated for several years | 13:32 |
| shardy | we've got a spurious error coming from tripleoclient tho which needs to be fixed | 13:32 |
| rhallisey | overcloud-without-mergepy is still a constant O.o | 13:33 |
| shardy | rhallisey: yeah, we need to remove it | 13:33 |
| rhallisey | shardy, ok will do. | 13:33 |
| shardy | d0ugal: was going to do it I think | 13:33 |
| rhallisey | gotcha | 13:33 |
| *** ramishra has quit IRC | 13:34 | |
| *** akuznetsov has joined #tripleo | 13:35 | |
| social | shardy: and it's not only a constat but it's used during stack update | 13:35 |
| rhallisey | I think upgrade references too | 13:35 |
| *** ramishra has joined #tripleo | 13:35 | |
| shardy | Yeah, those are bugs in the client | 13:35 |
| shardy | because we don't test those paths in CI | 13:35 |
| shardy | https://review.openstack.org/#/c/365735/ | 13:36 |
| shardy | I fixed it there for deploy, but missed that we use it for update | 13:36 |
| shardy | so we'll need to do a similar fix | 13:36 |
| shardy | to read the file from the plan instead of the local disk | 13:36 |
| *** myoung|gone is now known as myoung | 13:37 | |
| *** pkovar has quit IRC | 13:37 | |
| honza | jtomasek: looks like we'll also need rbrady's password patch before we can wrap this up | 13:37 |
| rbrady | honza, jtomasek: I'm currently trying to figure out why it doesn't pass CI. will keep you updated | 13:38 |
| honza | rbrady: thanks! | 13:38 |
| rhallisey | shardy, cool thanks! | 13:38 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Teardown libvirt pool: fix pool file removal https://review.openstack.org/374408 | 13:38 |
| jtomasek | honza: you can set the passwords manually through GUI, it is slightly hard to find them though | 13:38 |
| jtomasek | honza: most of them are in controller services | 13:38 |
| social | shardy: how this will work if I have newton undercloud and want to update mitaka? | 13:39 |
| rhallisey | social, good question | 13:39 |
| honza | jtomasek: my original patch made a call to /stack/<name>/environment to get the password but it looks like it'll be available in environmentConfiguration | 13:40 |
| bandini | gfidente: is it correct to assume that CephClusterFSID must change during an upgrade? | 13:41 |
| jtomasek | honza: I think it is safer to get it from Heat, rather than from plan configuration | 13:42 |
| shardy | social: if you're updating to a new release, the first step will be to upgrade the undercloud, so you'll have the new templates and mistral pieces to render the new overcloud.yaml | 13:42 |
| honza | jtomasek: ok | 13:42 |
| shardy | the rendered file will just get passed into heat as if it was on the local disk | 13:42 |
| shardy | but we're still working through a few client issues around this atm | 13:42 |
| d0ugal | rhallisey: Yeah, so this was on my TODO list but I've not started it yet. | 13:42 |
| social | shardy: I think the requirement is to update old overcloud | 13:42 |
| social | eg scale/downscale and system updates | 13:42 |
| rhallisey | d0ugal, gotcha. No worries | 13:42 |
| gfidente | bandini, no it does not have to | 13:42 |
| shardy | social: Sure, then you will point at an old version of tripleo-heat-templates | 13:43 |
| shardy | which will have the old overcloud.yaml in it | 13:43 |
| bandini | gfidente: ack thanks | 13:43 |
| shardy | social: in that case, the j2 rendering in mistral does nothing | 13:43 |
| shardy | but the deployment flow works the same | 13:43 |
| d0ugal | rhallisey: two related bugs: https://bugs.launchpad.net/tripleo/+bug/1614928 and https://bugs.launchpad.net/tripleo/+bug/1626128 | 13:43 |
| openstack | Launchpad bug 1614928 in tripleo "openstack overcloud update stack should be powered by a mistral workflow" [High,Triaged] | 13:43 |
| openstack | Launchpad bug 1626128 in tripleo "openstack overcloud update stack is broken" [Critical,Triaged] | 13:43 |
| rhallisey | d0ugal, ya I also reported one too: https://bugs.launchpad.net/tripleo/+bug/1623978 | 13:44 |
| openstack | Launchpad bug 1623978 in tripleo "overcloud update fails because of missing template in swift" [Undecided,New] | 13:44 |
| d0ugal | rhallisey: hah :) | 13:44 |
| rhallisey | d0ugal, :) | 13:45 |
| ccamacho | guys is there any doc re how to use/test TripleO UI ? just curious as never used it before and there are some bugs related | 13:45 |
| shardy | jtomasek: ^^ | 13:46 |
| *** tzumainn has joined #tripleo | 13:46 | |
| *** dsneddon has quit IRC | 13:47 | |
| jtomasek | ccamacho: I've been using this tool up to now https://github.com/flofuchs/o3-virt-setup | 13:49 |
| jtomasek | ccamacho: although it should be much easier to install GUI as it is going to get installed as part of undercloud. mandre has last puppet patch pending to achieve that afaik | 13:50 |
| *** ramishra has quit IRC | 13:50 | |
| *** pkovar has joined #tripleo | 13:51 | |
| panda | sshnaidm: still happening on the gates http://logs.openstack.org/92/363592/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-newton/adba039/logs/postci.txt.gz | 13:52 |
| mandre | jtomasek: that last patch was merged yesterday | 13:52 |
| jpich | jtomasek, ccamacho: That patch merged earlier today :) https://review.openstack.org/#/c/363167/ | 13:52 |
| jpich | yestoday | 13:52 |
| panda | sshnaidm: master worked, newton failed. | 13:52 |
| jtomasek | mandre, jpich: thanks, great! | 13:52 |
| panda | sshnaidm: out of memory | 13:52 |
| social | rhallisey: for now I'm testing update with manualy provided overcloud.yaml from swift | 13:53 |
| social | rhallisey: I kinda expect it to break more :) | 13:53 |
| jtomasek | ccamacho: in any case, installing GUI manually is also possible by following https://github.com/openstack/tripleo-ui/blob/master/README.md | 13:54 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Separate Template Processing From Create/Update Plan https://review.openstack.org/370868 | 13:54 |
| jtomasek | ccamacho: it is slightly out of date - no validations api and tripleo-api is needed any more, so GUI setup is basically just: clone GUI repo, set cors for services, do endpoints tunnels, and run GUI using 'npm install and npm start' | 13:55 |
| *** tzumainn has quit IRC | 13:55 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Deprecate the NeutronL3HA parameter https://review.openstack.org/374835 | 13:55 |
| *** tzumainn has joined #tripleo | 13:55 | |
| sshnaidm | panda, yeah, I see | 13:56 |
| sshnaidm | panda, more and more such errors in last days | 13:56 |
| panda | sshnaidm: we're doomed. | 13:56 |
| sshnaidm | panda, can you please start a ML thread? flavor update is 5 min change, but I think we need a consensus about it | 13:59 |
| tosky | panda: always | 13:59 |
| panda | sshnaidm: a bug is not enough ? | 14:00 |
| ccamacho | jtomasek awesome, just taking notes to trying to test it. | 14:01 |
| sshnaidm | panda, no, nobody looks at them | 14:01 |
| panda | sshnaidm: lol, ok. | 14:01 |
| rhallisey | social, ya wfm | 14:01 |
| sshnaidm | panda, sad-but-true | 14:01 |
| panda | sshnaidm: ok, on openstack-dev with tags TripleO and CI | 14:02 |
| sshnaidm | panda, yep, thanks, let's start the flame | 14:03 |
| shardy | perhaps try to avoid using "we're doomed" as the subject line ;) | 14:03 |
| panda | tosky: Snape style answer | 14:03 |
| panda | shardy: we're f$^$#d ? | 14:03 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Stop plan creation when container exists https://review.openstack.org/369623 | 14:04 |
| *** paramite has joined #tripleo | 14:04 | |
| jpich | d0ugal: It felt too weird also updating someone else's name in a TODO() in https://review.openstack.org/#/c/369623/2 so I didn't do it, sorry!! | 14:06 |
| d0ugal | jpich: haha, I didn't expect you to. It was just an observation of failure | 14:06 |
| *** ramishra has joined #tripleo | 14:06 | |
| jpich | d0ugal: You'll just have to fix the actual TODO in order to make it disappear... ;) | 14:07 |
| d0ugal | jpich: that TODO isn't for me! It's for a dmatthews, no idea who that is. | 14:08 |
| jpich | d0ugal: lol | 14:08 |
| *** akuznetsov has quit IRC | 14:09 | |
| *** florianf has joined #tripleo | 14:12 | |
| EmilienM | panda: still timeouting :( | 14:13 |
| EmilienM | interesting regular HA job doesn't timeout much comparing to ipv6 | 14:13 |
| *** mbozhenko has quit IRC | 14:16 | |
| *** rajinir has joined #tripleo | 14:17 | |
| panda | EmilienM: but they fail for memory error ... I don't know if it's related. I will launch again a local test, I added some swap to undercloud so maybe I'll be able to complete deploy and see what happens | 14:18 |
| panda | EmilienM: where do you see the time out ? | 14:19 |
| d0ugal | EmilienM: Do we use https://github.com/openstack/puppet-mistral? | 14:19 |
| *** saneax is now known as saneax-_-|AFK | 14:20 | |
| *** snecklifter has joined #tripleo | 14:21 | |
| snecklifter | hi, is there a reason why os-net-config is determined to dhcp an interface despite me using use_dhcp: false | 14:21 |
| snecklifter | driving me nuts | 14:21 |
| social | rhallisey: is it doing anything? I don't think it's doing anything | 14:22 |
| snecklifter | this is on mitaka | 14:22 |
| weshay | sshnaidm, can you please add the latest swift error from the ha periodic job to the etherpad.. http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/9d0c76f/logs/overcloud-controller-0/var/log/messages | 14:23 |
| shardy | d0ugal: yes we use it to configure mistral on the undercloud and overcloud (when the composable mistral patch lands) | 14:24 |
| *** athomas has quit IRC | 14:24 | |
| sshnaidm | weshay, I reran it and it passed http://logs.openstack.org/15/359215/12/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/35feaca/ | 14:25 |
| d0ugal | shardy: I am trying to understand the log files - I don't think we log everything from Mistral | 14:26 |
| sshnaidm | weshay, I'm not sure it's swift issue, seems like it's memory issue again | 14:26 |
| d0ugal | shardy: I think we should have a log file for each of these: https://github.com/openstack/puppet-mistral/tree/master/templates | 14:27 |
| panda | EmilienM: now I see it ... | 14:29 |
| EmilienM | d0ugal: yes sir | 14:29 |
| *** jaosorior has quit IRC | 14:29 | |
| d0ugal | EmilienM: Do you know where the mistral log file is configured? I'm trying to track that down. | 14:32 |
| EmilienM | d0ugal: https://github.com/openstack/puppet-mistral/blob/master/manifests/init.pp#L234 | 14:34 |
| EmilienM | check https://github.com/openstack/puppet-mistral/blob/master/manifests/init.pp#L236 | 14:35 |
| EmilienM | is debug setup on the undercloud manifest? | 14:35 |
| EmilienM | https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L487 | 14:35 |
| EmilienM | I don't see mistral::debug::true | 14:35 |
| EmilienM | so no logs | 14:35 |
| honza | jtomasek: looks like i messed up your patch :( | 14:36 |
| jpich | d0ugal: May I pick your brain about deprecation warnings for the new node management commands? ( https://bugs.launchpad.net/tripleo/+bug/1595205 ) | 14:36 |
| openstack | Launchpad bug 1595205 in tripleo "[tripleoclient] Additional "overcloud node" commands" [High,Fix released] - Assigned to Julie Pichon (jpichon) | 14:36 |
| d0ugal | EmilienM: hrm, there is one log file /var/log/mistral/mistral-server.log | 14:36 |
| EmilienM | d0ugal: good so debug is true by default | 14:37 |
| *** athomas has joined #tripleo | 14:37 | |
| d0ugal | EmilienM: but I would like to make sure we are logging everything and have three logs - api, engine and executor. | 14:37 |
| EmilienM | indeed, it's weird we don't have 3 logs | 14:37 |
| honza | jtomasek: i failed to rebase properly; this is the most frustrating part of gerrit :( | 14:37 |
| EmilienM | it looks like a packaging thing | 14:37 |
| d0ugal | EmilienM: and no, we don't have debug logging - it is set to INFO | 14:37 |
| d0ugal | jpich: Sure! | 14:37 |
| EmilienM | d0ugal: sounds like it's managed by packaging script | 14:38 |
| jpich | d0ugal: In the end only introspect and provide got a deprecation warning. At this point I feel like it'd be better to remove the warnings and add them back during the next release, rather than deprecate the other commands because the docs, CI, etc are still using/referencing them... I know you thought it'd be better to do it anyway to encourage people to actually use them and update the docs, etc though :/ | 14:39 |
| EmilienM | https://github.com/rdo-packages/mistral-distgit/blob/rpm-master/openstack-mistral-api.service | 14:39 |
| EmilienM | so it takes default in mistral.conf | 14:39 |
| jpich | d0ugal: There's also kind of a similar discussion going on for https://review.openstack.org/#/c/337676/ | 14:39 |
| d0ugal | EmilienM: aha, thanks! | 14:40 |
| EmilienM | d0ugal: wait, I didn't help yet | 14:40 |
| EmilienM | d0ugal: something is weird here | 14:41 |
| d0ugal | EmilienM: That much is expected :) | 14:41 |
| d0ugal | jpich: Yeah, I don't have a strong view. | 14:41 |
| d0ugal | jpich: I just know the sooner we mark them as deprecated the sooner we can actually delete them | 14:42 |
| EmilienM | d0ugal: to me, it sounds like we run mistral-server process and by default mistral create mistral-server.log file | 14:42 |
| d0ugal | jpich: but the actual code overhead is fairly small | 14:42 |
| EmilienM | d0ugal: do you see engine/api/... in the mistral-server.log? | 14:42 |
| EmilienM | if yes, it makes all sense | 14:42 |
| social | rhallisey: you are testing on RDO newton/master? | 14:42 |
| rhallisey | social, yes | 14:43 |
| d0ugal | EmilienM: Yeah, it seems they all log to the same file | 14:43 |
| rhallisey | newton | 14:43 |
| EmilienM | d0ugal: it's a bug in msitral I think | 14:43 |
| social | rhallisey: do you have issues with nova-compute during deploy? | 14:43 |
| EmilienM | https://github.com/rdo-packages/mistral-distgit/blob/rpm-master/openstack-mistral-executor.service#L8 | 14:43 |
| d0ugal | EmilienM: (which then means I don't know what happened to the logging I added on my local checkout of mistral :/) | 14:43 |
| EmilienM | mistral should create a logfile per "server" option given | 14:43 |
| jpich | d0ugal: Right. So I'm gonna remove the deprecation warnings for now, and I'll propose a patch to add them back to the relevant commands early in Ocata, together with some docs updates | 14:43 |
| d0ugal | EmilienM: Right, so mistral is ignoring that. I'll take a look. | 14:43 |
| social | rhallisey: I have issue in oslo.messaging that I'll probably have to track down | 14:43 |
| jpich | d0ugal: And take it from there. Thanks! | 14:44 |
| *** chem` has joined #tripleo | 14:44 | |
| d0ugal | jpich: Sounds good! | 14:44 |
| EmilienM | d0ugal: that's the root cause | 14:44 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Stacks and Resources data storing in app state https://review.openstack.org/374227 | 14:44 |
| *** chem has quit IRC | 14:45 | |
| rhallisey | social, no, I've been able to deploy ok | 14:45 |
| rhallisey | last time I did it was yerterday afternoon | 14:45 |
| social | rhallisey: yes the issue is actually race, if compute deploys later than controllers it should work fine | 14:46 |
| rhallisey | interesting | 14:46 |
| social | I'll give it another run today to debug/fix it | 14:46 |
| rhallisey | social, what error do you see? Is it a timeout? | 14:47 |
| rhallisey | message timeout? | 14:47 |
| social | rhallisey: 90% it's this https://bugs.launchpad.net/oslo.messaging/+bug/1581148 | 14:47 |
| openstack | Launchpad bug 1581148 in oslo.messaging "Constant exceptions "NotFound: Basic.consume: (404) NOT_FOUND - no queue abc in vhost '/'" in log" [Undecided,Fix released] - Assigned to Kirill Bespalov (k-besplv) | 14:47 |
| rhallisey | social, a few times I've had the heatclient return | 14:48 |
| rhallisey | : message timeout <hash> | 14:48 |
| rhallisey | that could be it | 14:49 |
| *** ayoung has joined #tripleo | 14:49 | |
| social | I have reproducer now so I only need to apply patch update images and rerund deploy | 14:49 |
| *** zoli|lunch is now known as zoli|wfh | 14:50 | |
| d0ugal | EmilienM: so, it looks like we can set the log file by passing --log-file | 14:51 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Update gnocchi database during M/N upgrade. https://review.openstack.org/374884 | 14:51 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move keystone::auth into service_config_settings https://review.openstack.org/370573 | 14:51 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Tolerate missing keys from role_data in service templates https://review.openstack.org/374237 | 14:51 |
| EmilienM | d0ugal: patch the distgit :) | 14:51 |
| EmilienM | https://github.com/rdo-packages/mistral-distgit | 14:51 |
| EmilienM | and problem solved! | 14:51 |
| d0ugal | EmilienM: I can't find where the default comes from, maybe oslo.config uses the process name? | 14:51 |
| d0ugal | EmilienM: will do :) | 14:51 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Update gnocchi database during M/N upgrade. https://review.openstack.org/374884 | 14:54 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient: Keystone credentials needs to be set with the overcloud password https://review.openstack.org/374892 | 14:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move keystone::auth into service_config_settings https://review.openstack.org/370573 | 14:58 |
| *** akuznetsov has joined #tripleo | 14:59 | |
| *** rcernin has quit IRC | 15:07 | |
| EmilienM | d0ugal: yes oslo config I think | 15:08 |
| b00tcat | when deploying an overcloud using `openstack overcloud deploy --templates`, what's the default value for TEMPLATES? | 15:08 |
| b00tcat | on `/usr/share/tripleo/templates/` I can only find 2 XMLs | 15:08 |
| b00tcat | :/ | 15:08 |
| *** pkovar has quit IRC | 15:09 | |
| shardy | b00tcat: /usr/share/openstack-tripleo-heat-templates/ | 15:10 |
| *** ebarrera has quit IRC | 15:11 | |
| shardy | we should output that in the help text | 15:11 |
| shardy | (but we don't) | 15:11 |
| *** yamahata has joined #tripleo | 15:11 | |
| *** zigo_ is now known as zigo | 15:14 | |
| b00tcat | shardy: thanks shardy | 15:15 |
| b00tcat | oops, too much shardy | 15:15 |
| jtomasek | honza: checking | 15:15 |
| *** dhill_ has joined #tripleo | 15:17 | |
| jtomasek | honza: it is no big deal, just a few bits, I am going to send a new update | 15:17 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Stacks and Resources data storing in app state https://review.openstack.org/374227 | 15:20 |
| *** yolanda has joined #tripleo | 15:20 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Neutron metadata agent worker count fix https://review.openstack.org/374915 | 15:21 |
| *** tremble has quit IRC | 15:21 | |
| *** jcoufal has quit IRC | 15:21 | |
| hewbrocca | shardy: apropos of nothing at all, are there unit tests for the composable services stuff? | 15:22 |
| shardy | hewbrocca: No, we need to add CI scenarios which cover both composable services and custom roles | 15:23 |
| hewbrocca | thanks | 15:23 |
| shardy | hewbrocca: we're somewhat using composable services for the multinode job tho | 15:23 |
| shardy | because we use it to do an all-in-one deployment | 15:23 |
| hewbrocca | sure | 15:23 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Swift puppet-tripleo to use puppet-openstack_spec_helper https://review.openstack.org/374916 | 15:23 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper https://review.openstack.org/374916 | 15:24 |
| EmilienM | hewbrocca: i'm working o nit | 15:24 |
| EmilienM | shardy: ^ | 15:24 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Use the proper private keys for ssh config file https://review.openstack.org/370919 | 15:24 |
| hewbrocca | EmilienM: excellent | 15:24 |
| EmilienM | puppet unit tests? | 15:24 |
| shardy | EmilienM: Ah, I was assuming CI coverage vs unit tests for puppet-tripleo | 15:25 |
| EmilienM | I'm working on both scenarios and unit tests actually, and mwhahaha also started to add unit tests | 15:25 |
| shardy | I know we're improving things which is great | 15:25 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: When deploy finishes, show overcloud info https://review.openstack.org/370765 | 15:25 |
| EmilienM | shardy, hewbrocca: it's documented here: https://github.com/openstack-infra/tripleo-ci#service-testing-matrix | 15:25 |
| EmilienM | and we're working on adding more services but we wait for the release | 15:25 |
| *** dsneddon has joined #tripleo | 15:26 | |
| jtomasek | honza: I've just updated both patches | 15:26 |
| jtomasek | honza: here is the diff of the changes I did to your patch: https://review.openstack.org/#/c/370765/6..7 | 15:26 |
| *** jcoufal has joined #tripleo | 15:27 | |
| jtomasek | honza: I am going to review it tomorrow. In addition, I am working on DeploymentDetail component now, which is going to render DeploymentConfirmation, DeploymentProgress, DeploymentSuccess, DeploymentFailure components depending what state the deployment is in | 15:27 |
| *** dmacpher has joined #tripleo | 15:29 | |
| *** lucasagomes is now known as lucas-hungry | 15:30 | |
| *** jcoufal_ has joined #tripleo | 15:31 | |
| *** panda is now known as panda|break | 15:31 | |
| *** pkovar has joined #tripleo | 15:32 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Always configure ipv6 address with net-iso https://review.openstack.org/374922 | 15:32 |
| *** jcoufal has quit IRC | 15:33 | |
| *** aufi has quit IRC | 15:35 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper https://review.openstack.org/374916 | 15:35 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient: Keystone credentials needs to be set with the overcloud password https://review.openstack.org/374892 | 15:35 |
| honza | jtomasek: if anyone could explain to me how gerrit works, that would be great :) | 15:37 |
| honza | jtomasek: thanks | 15:37 |
| *** ccamacho is now known as ccamacho|afk | 15:39 | |
| ccamacho|afk | honza http://docs.openstack.org/infra/manual/developers.html | 15:40 |
| *** akuznetsov has quit IRC | 15:40 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use low-memory-usage.yaml in ci https://review.openstack.org/374931 | 15:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove the get_hiera_key function https://review.openstack.org/367367 | 15:40 |
| honza | ccamacho|afk: yes, yes, yes, but it never does what i want, it has a mind of its own :) | 15:41 |
| * hewbrocca thinks gerrit is a regression | 15:41 | |
| gfidente | ping tbarron | 15:42 |
| *** dsariel has quit IRC | 15:45 | |
| *** ebarrera has joined #tripleo | 15:45 | |
| tbarron | gfidente: pong | 15:46 |
| gfidente | tbarron, wonder if you still have the environment where you tested https://review.openstack.org/#/c/358525 ? | 15:46 |
| gfidente | I would like to look into what is failing, seems like we could see what failed from os-collect-config logs | 15:47 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper https://review.openstack.org/374916 | 15:48 |
| EmilienM | bnemec, panda|break: I think swift/ipv6 is fixed | 15:48 |
| EmilienM | memcache_servers = [fd00:fd00:fd00:2000::18]:11211,[fd00:fd00:fd00:2000::14]:11211,[fd00:fd00:fd00:2000::1c]:11211 | 15:48 |
| EmilienM | I saw it in CI | 15:48 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use low-memory-usage.yaml in ci https://review.openstack.org/374931 | 15:48 |
| tbarron | gfidente: yeah, I'll pm you access if you'd like | 15:49 |
| *** abehl has quit IRC | 15:50 | |
| gfidente | tbarron, ah nice, yes we could do tmux | 15:50 |
| tbarron | gfidente: i left it in that state | 15:50 |
| bnemec | EmilienM: Okay, I had no luck yesterday. | 15:50 |
| *** Ryjedo has quit IRC | 15:50 | |
| gfidente | Jokke_, ^^ | 15:50 |
| EmilienM | bnemec: http://logs.openstack.org/74/363674/25/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ipv6/92d07bf/logs/ | 15:50 |
| EmilienM | logs are here, if you look into swift config, we're good now | 15:50 |
| tbarron | gfidente: it's a beaker box, i asked for extended reservation and so far no one has taken it back :) | 15:50 |
| EmilienM | but now it timeouts on something I haven't figured yet | 15:50 |
| gfidente | tbarron++ | 15:51 |
| *** kjw3 has quit IRC | 15:51 | |
| bnemec | EmilienM: Yeah, that does look right. | 15:51 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Remove deprecation warning for bulk introspection https://review.openstack.org/374935 | 15:51 |
| d0ugal | When are we releasing RC2 again? | 15:51 |
| EmilienM | d0ugal: next week | 15:51 |
| d0ugal | damn :) | 15:52 |
| EmilienM | d0ugal: maximum Thursday I think | 15:52 |
| EmilienM | if shardy is ok | 15:52 |
| EmilienM | and we're branching puppet modules next week | 15:52 |
| bandini | marios: https://bugs.launchpad.net/tripleo/+bug/1626628 | 15:54 |
| openstack | Launchpad bug 1626628 in tripleo "M/N Upgrade - major-upgrade-pacemaker times out" [Critical,New] | 15:54 |
| bandini | I have the system live in case | 15:54 |
| bandini | bbiab supper time | 15:54 |
| EmilienM | let's make puppet-tripleo great again | 15:55 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper https://review.openstack.org/374916 | 15:56 |
| marios | bandini: thanks very much /me eod will have a look tomorrow | 15:57 |
| beagles | +1 on that | 15:57 |
| beagles | re: puppet-tripleo that is | 15:57 |
| jistr | chem`: hi, i see you reported the gnocchi not starting bug, i think we're just missing: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf in the upgrade | 15:57 |
| jrist | jtomasek: do you see what I'm saying with my -1 on https://review.openstack.org/#/c/374227/6/src/js/constants/StacksConstants.js ? | 15:58 |
| *** dmacpher is now known as dmacpher-afk | 15:59 | |
| shardy | EmilienM: +1 on next week for RC2 | 16:00 |
| jistr | chem`: commented on the upstream bug with a code pointer too | 16:00 |
| EmilienM | so we see the timeout http://logs.openstack.org/74/363674/25/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ipv6/92d07bf/logs/undercloud/var/log/heat/heat-api.txt.gz#_2016-09-22_14_11_55_271 | 16:00 |
| jrist | what day for rc2? | 16:00 |
| EmilienM | jrist: I'll do it on Thursday | 16:00 |
| EmilienM | at evening | 16:00 |
| EmilienM | so we have 5 days | 16:01 |
| d0ugal | Panic! | 16:01 |
| jrist | thx | 16:01 |
| EmilienM | well no | 16:01 |
| EmilienM | we'll have stable/newton in place | 16:01 |
| d0ugal | :) | 16:01 |
| EmilienM | and we'll be able to backport bugfixes | 16:01 |
| shardy | https://releases.openstack.org/newton/schedule.html | 16:01 |
| shardy | Yeah we still have the possibility of making some fixes before declaring the release final | 16:02 |
| d0ugal | So we technically have until summit for bug fixing? | 16:02 |
| EmilienM | shardy: do you have 2 min to help in looking why we timeout in ipv6 jobs? | 16:02 |
| EmilienM | shardy: I have found the timeout in heat api logs but not the actual resource | 16:02 |
| shardy | d0ugal: bug fixes can always be backported, but we're aiming to declare the release final very close to the main Newton release | 16:02 |
| shardy | which is w/c 3rd October | 16:02 |
| shardy | so 2 weeks | 16:02 |
| d0ugal | shardy: k, thanks. | 16:02 |
| shardy | EmilienM: sure | 16:03 |
| shardy | EmilienM: FYI I think one of the reasons we're using more memory is more heavy usage of yaql in the templates | 16:04 |
| EmilienM | ok | 16:04 |
| shardy | it may be we can optimize that, but probably not before the release | 16:04 |
| EmilienM | I wasn't sure it was related | 16:04 |
| hewbrocca | o NOES the yaql parser is a memory hog | 16:04 |
| shardy | I'd like to convert a lot of the yaql calls to native heat functions, which should be a lot less expensive | 16:04 |
| *** myoung is now known as myoung|biab | 16:04 | |
| shardy | let(root => $) -> $.data.map.items().where($[0] in $root.data.services).select($[1]).reduce($1.mergeWith($2), {}) | 16:05 |
| *** jpich has quit IRC | 16:05 | |
| shardy | how could that not eat memory? :D | 16:05 |
| EmilienM | sounds like using python in heat would help | 16:05 |
| shardy | EmilienM: yeah, that's what I'd like to do | 16:05 |
| EmilienM | shardy: do we have WIP in CI to reduce failures? | 16:06 |
| EmilienM | I saw a patch from bnemec to use jistr's template for low memory setups | 16:06 |
| jistr | EmilienM: it will not work though, i'm just reviewing it | 16:06 |
| EmilienM | but iiuc we're already using low worker # | 16:06 |
| bnemec | EmilienM: That's basically a noop though. | 16:06 |
| bnemec | Yeah | 16:06 |
| EmilienM | yeah | 16:06 |
| bnemec | Yeah! | 16:06 |
| bnemec | :-) | 16:06 |
| jrist | shardy or EmilienM - how do I connect two series together? i.e. rc1 -> rc2 in launchpad | 16:07 |
| shardy | EmilienM: panda|break started a ML thread, we have to decide if we tune for lower memory usage or accept an even bigger undercloud | 16:07 |
| shardy | jrist: wha? | 16:07 |
| shardy | jrist: not sure what you mean | 16:07 |
| jrist | you have newton-1 -> newton-2 ... https://launchpad.net/tripleo | 16:07 |
| *** masco has joined #tripleo | 16:07 | |
| *** b00tcat has quit IRC | 16:07 | |
| jtomasek | jrist: hmm, I don't | 16:08 |
| jrist | is it based on branch? | 16:08 |
| jrist | jtomasek: the inconsistency in the sentences | 16:08 |
| bnemec | Gah, that password prompt bug is still there. | 16:08 |
| shardy | https://launchpad.net/tripleo/newton | 16:08 |
| * bnemec rages | 16:08 | |
| EmilienM | bnemec: yes | 16:09 |
| shardy | jrist: sorry, I still don't follow - I just created the milestones in LP, and we targetted stuff to them | 16:09 |
| EmilienM | :( | 16:09 |
| shardy | jrist: they're not linked, other than by being part of the same series, e.g newton | 16:09 |
| jrist | shardy: oh | 16:09 |
| EmilienM | shardy: how we could tune more? | 16:09 |
| jrist | let me see, thanks shardy | 16:09 |
| jtomasek | jrist: I don't see any comment there. I extracted those from DeploymentStatus component. I am ok with changing them as you like, just let me know what to change | 16:10 |
| shardy | EmilienM: I suspect we could tune rabbit and mysql better for memory constrained systems | 16:10 |
| shardy | they're the two biggest memory users after heat-engine IME | 16:10 |
| jrist | shardy: I think I'm doing something wrong https://launchpad.net/tripleo-ui | 16:10 |
| jrist | jtomasek: https://review.openstack.org/#/c/374227/1/src/js/constants/StacksConstants.js | 16:11 |
| shardy | jrist: you've created multiple series instead of milestones against one newton series | 16:11 |
| jtomasek | jrist: oh, I see it now, sorry.... I am blind | 16:11 |
| jrist | shardy: but don't bother looking since we're going to tripleo proper | 16:11 |
| mwhahaha | shardy: no it's all heat engine by a factor of like 4 | 16:11 |
| *** jlinkes has quit IRC | 16:11 | |
| mwhahaha | mysql/rabbit only use like 500M max | 16:11 |
| d0ugal | Fun, the tripleoclient unit tests are broken. | 16:11 |
| mwhahaha | combined | 16:11 |
| jrist | shardy: so series should be newton | 16:11 |
| jrist | and milestones should be in the series | 16:11 |
| shardy | jrist: yes | 16:12 |
| shardy | mwhahaha: Ok, in my previous testing all three were taking over 1G | 16:13 |
| shardy | probably need to look at it again | 16:13 |
| mwhahaha | shardy: i've got a basic deploy going and heat-engine is near 2g | 16:13 |
| mwhahaha | sorry 3G math is hard | 16:14 |
| *** egafford has quit IRC | 16:15 | |
| shardy | mwhahaha: Ok, we may have an issue then, I reported some heat bugs earlier in the cycle and we got the memory usage down way below that | 16:16 |
| shardy | at least for small developer deployments | 16:16 |
| shardy | http://people.redhat.com/~shardy/heat/plots/heat_before_after_filesfix_plot.png | 16:16 |
| mwhahaha | yea it's way more than that now | 16:17 |
| shardy | that was a 2 node nonha deployment before/after the fixes | 16:17 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Move inclusion of ::manila::db::mysql in manila/api profile https://review.openstack.org/374961 | 16:17 |
| *** masco has quit IRC | 16:18 | |
| *** dtantsur is now known as dtantsur|afk | 16:19 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use low-memory-usage.yaml in ci https://review.openstack.org/374931 | 16:20 |
| mwhahaha | we also may want to put a cap on workers for the services | 16:20 |
| *** rcernin has joined #tripleo | 16:20 | |
| weshay | panda|break, fyi https://review.openstack.org/#/c/374922/ | 16:21 |
| shardy | mwhahaha: Yeah, we already do for the overcloud in CI, but perhaps we can reduce them further on the undercloud | 16:21 |
| *** rbrady is now known as rbrady|afk | 16:23 | |
| *** fultonj has quit IRC | 16:24 | |
| ayoung | jrist, where do I find the files that describe how to configure Keystone for the undercloud? | 16:26 |
| ayoung | I know where in the puppet module to look, but not what calls that | 16:26 |
| panda|break | bnemec: you think your v6 environments may be used to test ipv6 locally ? my loca test hangs at NetworkDeployment | 16:26 |
| mwhahaha | ayoung: https://github.com/openstack/instack-undercloud/tree/master/elements/puppet-stack-config | 16:27 |
| jrist | ayoung: did you mean jistr? | 16:27 |
| *** fultonj has joined #tripleo | 16:27 | |
| ayoung | jrist, I means mwhahaha obviously | 16:27 |
| bnemec | panda|break: Make sure network-environment.yaml actually matches your configuration. | 16:27 |
| mwhahaha | :D | 16:27 |
| bnemec | panda|break: Usually when that happens to me it's because I used a non-default cidr on the undercloud and forgot to change network-environment.yaml to match. | 16:28 |
| ayoung | I really just asked someone I thoght might be able to point me in the right direction, or point me at someone that could point me, or .... | 16:28 |
| mwhahaha | ayoung: i was in there looking for something else, thought i'd share | 16:28 |
| *** pkovar has quit IRC | 16:28 | |
| panda|break | weshay: thanks, I also wonder why the NETISO_V4/6 condition exist | 16:28 |
| panda|break | bnemec: generally or on vlan10 ? | 16:28 |
| bnemec | panda|break: Oh, and that will only work if your local environment is ovb. You may need to change the nic-configs to match in other environments. | 16:29 |
| ayoung | mwhahaha, is that instack? | 16:29 |
| shardy | mwhahaha: Ok, I can confirm your observations, heat is using a huge amount more memory for me too | 16:29 |
| mwhahaha | ayoung: yea openstack undercloud install calls instack-undercloud-setup or whatever it's called | 16:29 |
| panda|break | bnemec: no, libvirt ... | 16:29 |
| ayoung | ok | 16:30 |
| bnemec | panda|break: Okay, then those templates probably won't work. Try https://github.com/openstack/tripleo-heat-templates/blob/master/environments/net-multiple-nics-v6.yaml instead. | 16:30 |
| panda|break | bnemec: replace both with this one ? | 16:33 |
| *** hewbrocca is now known as hewbrocca-afk | 16:34 | |
| bnemec | panda|break: Not network-isolation-v6.yaml. You'll still need that one. | 16:34 |
| panda|break | bnemec: great, thanks. | 16:34 |
| *** lucas-hungry is now known as lucasagomes | 16:35 | |
| bnemec | panda|break: And you may still need some of network-environment.yaml. In CI we used http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/test-environments/net-iso.yaml | 16:35 |
| bnemec | panda|break: Although looking now, I think those are all the default, so I'm not sure why we did that. :-/ | 16:35 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Change the level of mocking for the wait_for_stack_ready test https://review.openstack.org/374968 | 16:37 |
| d0ugal | ^ That fixes the tripleoclient tests, reviews please! | 16:38 |
| panda|break | bnemec: usually that means that they converged gradually :) | 16:38 |
| d0ugal | (and by fixes, I mean they are slightly less terrible, I'd like to rm -rf them in Ocata and start over) | 16:38 |
| *** mcornea has quit IRC | 16:39 | |
| *** derekh has quit IRC | 16:40 | |
| *** zoli|wfh is now known as zoli|gone | 16:41 | |
| *** zigo has quit IRC | 16:41 | |
| *** zoli|gone is now known as zoli_gone-proxy | 16:41 | |
| *** cylopez has quit IRC | 16:42 | |
| *** bana_k has joined #tripleo | 16:42 | |
| *** pkovar has joined #tripleo | 16:43 | |
| bnemec | d0ugal: Reviewed, but I think you changed the test's behavior. | 16:43 |
| d0ugal | bnemec: Yeah, I stopped it testing heatclient? maybe I missed something else. | 16:44 |
| shardy | panda|break: can you link a failing CI job with OOM (or tell me the heat version from a failing job)? | 16:45 |
| bnemec | d0ugal: See my inline comment. I explained how the test flow changed. | 16:45 |
| d0ugal | k, thanks | 16:45 |
| shardy | it looks like we promoted yesterday, but I just want to ensure we're running the latest heat | 16:45 |
| *** nyechiel has quit IRC | 16:45 | |
| ccamacho|afk | shardy https://review.openstack.org/#/c/374660/ | 16:46 |
| bnemec | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_name:%20*tripleo-ci*%20AND%20build_status:%20FAILURE%20AND%20message:%20%5C%22503%20Service%20Unavailable%5C%22 | 16:46 |
| ccamacho|afk | ^OOM | 16:46 |
| bnemec | ^OOM query | 16:46 |
| bnemec | IRC race condition. :-) | 16:46 |
| bnemec | Logstash appears to have quit indexing at 19:00 two days ago though. | 16:47 |
| bnemec | It claims there are no OOM messages since then, which is definitely bogus. | 16:47 |
| shardy | Do we aggregate the dstat data anywhere? | 16:49 |
| shardy | I know there's Dan's graphite server, but that seems to only contain timing stats | 16:49 |
| shardy | it'd be super useful to identify when the memory usage went up significantly | 16:50 |
| *** zigo has joined #tripleo | 16:51 | |
| *** zigo is now known as Guest18656 | 16:52 | |
| panda|break | shardy: the OOM was in a local job I started this morning, and I overwrote it. I relaunched with the same version, yum info reports this release: 0.20160921092642.38a4afa.el7.centos, on the second run, with swap on, memory used on swap is 800M | 16:52 |
| *** ohamada has quit IRC | 16:54 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs: Add IP Assignment to node_placement https://review.openstack.org/374977 | 16:54 |
| *** Guest18656 has quit IRC | 16:56 | |
| *** zigo_ has joined #tripleo | 16:59 | |
| gfidente | tbarron, that line in scheduler.pp is defining a refresh | 17:02 |
| gfidente | if db sync happens, -share is notified | 17:02 |
| gfidente | but it does not enforce a dependency on db sync | 17:03 |
| gfidente | it's this guy pulling it in https://github.com/openstack/puppet-manila/blob/master/manifests/api.pp#L185 | 17:03 |
| tbarron | gfidente: i see, so puppet-manila is doing it from api and THT had it from api after all, | 17:04 |
| *** jpena is now known as jpena|off | 17:05 | |
| tbarron | no and puppet-manila had it from api after all, I can't type | 17:05 |
| tbarron | gfidente: ^^ | 17:05 |
| tbarron | gfidente: now i understand why your proposed fix s/b sufficient | 17:05 |
| EmilienM | shardy: we don't have dstat but I can add it | 17:06 |
| EmilienM | shardy: we have it in puppet ci | 17:06 |
| * tbarron is doing too many things at once, you OOO guys don't know what that's like :-P | 17:07 | |
| gfidente | ahahah | 17:07 |
| *** zigo_ has quit IRC | 17:07 | |
| tbarron | gfidente: thanks for all the help, i'll be back in an hour or less | 17:07 |
| EmilienM | mhh we already have WORKSPACE/logs/dstat-csv.log | 17:09 |
| EmilienM | http://logs.openstack.org/74/363674/25/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ipv6/92d07bf/logs/dstat-csv.txt.gz | 17:09 |
| EmilienM | the format is really broken though or I'm dumb? | 17:09 |
| *** zigo_ has joined #tripleo | 17:11 | |
| bnemec | EmilienM: Yeah, but we don't have a way to track changes in dstat output over time. | 17:11 |
| bnemec | I'm also not sure dstat by itself is enough. We need it broken down per-process. | 17:12 |
| bnemec | Unless we want to just assume heat is always the culprit when memory increases. ;-) | 17:12 |
| shardy | well as a first step, just knowing when the entire host memory usage went up would help | 17:12 |
| mwhahaha | have you guys used atop | 17:12 |
| shardy | but yeah, ideally per-process graphs would be great | 17:12 |
| bkero | That's another top-like util with some more verbose defaults, right? | 17:12 |
| EmilienM | bnemec: dstat can track per process | 17:13 |
| shardy | then maybe push all the maximums to the graphite server with the timings or something | 17:13 |
| mwhahaha | it captures proc/cpu/disk over time and provided a replayable file | 17:13 |
| EmilienM | not sure what you meant | 17:13 |
| EmilienM | but let me show how we have dstat in puppet ci | 17:13 |
| mwhahaha | fuel uses it to capture what's happening and it's useful if you want to know where the utilization is occuring | 17:13 |
| EmilienM | http://logs.openstack.org/04/374604/1/check/gate-puppet-openstack-integration-3-scenario003-tempest-centos-7/ca8202f/logs/dstat.txt.gz | 17:13 |
| bnemec | EmilienM: Okay, well _our_ dstat output doesn't appear to. | 17:13 |
| EmilienM | bnemec: I'm proposing a change | 17:13 |
| EmilienM | mwhahaha: looks like a good idea | 17:14 |
| mwhahaha | https://linux.die.net/man/1/atop | 17:14 |
| mwhahaha | it's a little funky in file rotation, but would be super useful for at least ci | 17:14 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: dstat: improve output to track high cpu process https://review.openstack.org/374982 | 17:15 |
| *** rbrady|afk is now known as rbrady | 17:15 | |
| EmilienM | mwhahaha: do you have an example of output? | 17:16 |
| mwhahaha | yea let me go pull a file out of a fuel log dump | 17:16 |
| EmilienM | thanks | 17:16 |
| mwhahaha | it's one of those things like sar where you have to use it to parse the output | 17:16 |
| *** egafford has joined #tripleo | 17:18 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper https://review.openstack.org/374916 | 17:18 |
| *** bana_k has quit IRC | 17:18 | |
| *** florianf has quit IRC | 17:19 | |
| *** social has quit IRC | 17:19 | |
| mwhahaha | so not sure the best way to provide an atop file | 17:20 |
| EmilienM | on the undercloud at least | 17:21 |
| *** ebarrera has quit IRC | 17:21 | |
| EmilienM | like we have in dstat | 17:21 |
| *** trown is now known as trown|lunch | 17:21 | |
| mwhahaha | yea there's an atopsar you can run on the file after the fact | 17:21 |
| mwhahaha | but the true usefulness is running atop after the fact because it's a top like interface that allows you to go back and forth through time in the file | 17:21 |
| mwhahaha | so if you know something happened at like 13:45, you can see what cpu/mem is like and what processes are doing what | 17:22 |
| mwhahaha | but like i said there is also atopsar which lets you dump the same info out for a file, https://linux.die.net/man/1/atopsar | 17:22 |
| mwhahaha | so you can get like the top3 memory consumers over time, http://paste.openstack.org/show/582632/ | 17:23 |
| pradk | can i get some reviews on https://review.openstack.org/#/c/360004/ please | 17:23 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient: Keystone credentials and CephClusterFSID needs to be set with the overcloud password https://review.openstack.org/374892 | 17:24 |
| mwhahaha | or most cpu http://paste.openstack.org/show/582633/ | 17:24 |
| EmilienM | interesting | 17:24 |
| mwhahaha | so if you install it and it's running, we'd just need to capture the atop file after the fact | 17:24 |
| EmilienM | pradk: have you tested it? | 17:25 |
| EmilienM | mwhahaha: a first good step would be to install it, run it and save the file in workspace. is it big? | 17:25 |
| mwhahaha | if you grab https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.smoke_neutron/7920/artifact/logs/7920/fail_error_deploy_neutron_tun-fuel-snapshot-2016-09-22_15-50-45.tar | 17:26 |
| mwhahaha | and extract the logs-fuel.tar.gz | 17:26 |
| mwhahaha | there's an atop file in fuel/var/log/atop | 17:26 |
| mwhahaha | it's about 7M | 17:26 |
| mwhahaha | you can yum install atop and use atop or atopsar on it | 17:26 |
| EmilienM | 2016-09-22 17:25:42.011604 | Gem::RemoteFetcher::UnknownHostError: timed out | 17:26 |
| EmilienM | sigh | 17:27 |
| EmilienM | I'll push again to have a gem mirror in OpenStack Infra | 17:27 |
| EmilienM | for the record, I started this work almost a year ago https://review.openstack.org/#/c/253616/ | 17:27 |
| pradk | EmilienM, in the process, jumping through some workarounds to get upgrade going | 17:30 |
| beagles | pradk: I read it, but sadly I think my upgrade-fu is too shaky to weigh in. | 17:31 |
| pradk | my unercloud upgrade fails .. so looking at workarounds now | 17:31 |
| * beagles is working on that as it is going to be something that is "always going to be there" | 17:31 | |
| pradk | yea | 17:32 |
| shardy | http://people.redhat.com/~shardy/heat/plots/heat_before_after_end_newton.png | 17:34 |
| shardy | ouch | 17:34 |
| shardy | zaneb: ^^ FYI | 17:35 |
| shardy | so we definitely have another heat memory leak :( | 17:35 |
| beagles | shardy: out of curiousity, what is the time scale in? | 17:36 |
| beagles | runs? | 17:36 |
| slagle | beagles: increments of 22 | 17:36 |
| slagle | whatever that means :) | 17:37 |
| beagles | lol | 17:37 |
| shardy | beagles: seconds I think | 17:37 |
| slagle | every 22 of something....we leak memory | 17:37 |
| shardy | I have a script which measures the sum of all heat-engine memory usage every second | 17:37 |
| panda|break | shardy: nice. | 17:37 |
| beagles | ooohh.. okay I thought this was over a larger time scale | 17:37 |
| beagles | nice | 17:37 |
| shardy | beagles: it's one deployment | 17:37 |
| shardy | one 2 node nonha overcloud deployment | 17:37 |
| shardy | the lower lines are the same test done earlier in the cycle before/after some fixes | 17:38 |
| beagles | shardy: right.. got it. The different sets of lines are for different points in the cycle... | 17:38 |
| beagles | yeah | 17:38 |
| beagles | cool | 17:38 |
| shardy | we're now in much worse shape than even before those fixes :( | 17:38 |
| beagles | but sucky | 17:38 |
| shardy | ya | 17:38 |
| *** pradk has quit IRC | 17:39 | |
| *** pradk has joined #tripleo | 17:42 | |
| *** bana_k has joined #tripleo | 17:44 | |
| *** tosky has quit IRC | 17:47 | |
| *** pkovar has quit IRC | 17:48 | |
| EmilienM | shardy: do we know what in Heat could cause it? Is it also related to YAQL work in THT? | 17:50 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper https://review.openstack.org/374916 | 17:51 |
| EmilienM | mwhahaha: it pass my local tests :) ^ | 17:51 |
| d0ugal | bnemec: replied to your comment | 17:51 |
| shardy | EmilienM: Not yet, I suspect it's not related to the yaql work, because those steps show big jumps in memory usage during the deployment, but all of the yaql evaluations happen at the start | 17:52 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: dstat: improve output to track high cpu process https://review.openstack.org/374982 | 17:53 |
| shardy | EmilienM: what we need ideally is some historical data, so we can estimate when the memory usage started to get bad again | 17:53 |
| shardy | alternatively, I'll have to spend a day bisecting with different heat builds and/or memory profiling heat | 17:53 |
| shardy | I've done that before, several times, and it's going to be time consuming :( | 17:53 |
| EmilienM | right | 17:54 |
| EmilienM | I'm wondering if we can automate it | 17:54 |
| EmilienM | using delorean builds | 17:54 |
| shardy | https://bugs.launchpad.net/heat/+bug/1626675 | 17:54 |
| openstack | Launchpad bug 1626675 in heat "Further memory usage issues with big stacks" [Undecided,New] | 17:54 |
| shardy | I raised that, so if anyone collects any data, please add it there | 17:54 |
| EmilienM | like a loop that would deploy tripleo using old delorean builds | 17:54 |
| shardy | EmilienM: Yeah, that would be ideal, I'm sure it can be scripted | 17:54 |
| d0ugal | Any other input on this would be good. The tripleoclient tests are currently broken. https://review.openstack.org/#/c/374968/ | 17:55 |
| shardy | we'd have to back out a couple of tht patches using very new heat features, such as the conditionals one beagles added a few days ago | 17:55 |
| EmilienM | shardy: I even think we don't have to redeploy the whole undercloud | 17:55 |
| shardy | EmilienM: No, we don't, I test locally with delorean builds of heat all the time | 17:55 |
| bnemec | d0ugal: Ah, so part of the problem is that we moved the polling out of tripleoclient | 17:55 |
| shardy | it's just the time to build them, then redeploy the overcloud | 17:56 |
| EmilienM | right | 17:56 |
| EmilienM | Puppet CI noticed some heat timeouts a few weeks ago | 17:56 |
| EmilienM | let me find the history | 17:56 |
| EmilienM | I remember /me complaining about it | 17:56 |
| bnemec | d0ugal: And I see we actually have a test for the create_complete case (although it looks sketchy to me at first glance, but that's a problem for another time). | 17:56 |
| EmilienM | shardy: let me find it | 17:56 |
| EmilienM | it's in http://status.openstack.org/openstack-health/#/ somewhere | 17:57 |
| shardy | https://review.openstack.org/#/c/370467/ is one fix which was mentioned, but we have that | 17:57 |
| shardy | I suspect there's some other circular reference issue that's crept in | 17:57 |
| bnemec | d0ugal: Okay, +2. | 17:58 |
| EmilienM | http://status.openstack.org/openstack-health/#/job/gate-puppet-openstack-integration-3-scenario003-tempest-centos-7?resolutionKey=day | 17:58 |
| EmilienM | tempest.api.orchestration.stacks.test_stacks.StacksTestJSON | 17:59 |
| slagle | EmilienM: did you mean to drop the -csv from the dstat logfile name too? | 17:59 |
| slagle | just noticed | 17:59 |
| EmilienM | slagle: yes | 17:59 |
| EmilienM | the format will be like puppet CI | 17:59 |
| EmilienM | http://logs.openstack.org/04/374604/1/check/gate-puppet-openstack-integration-3-scenario003-tempest-centos-7/ca8202f/logs/dstat.txt.gz | 17:59 |
| EmilienM | it will be mor ehelpful than a csv format | 17:59 |
| EmilienM | let me find in logstash where heat started to be slow in puppet CI | 18:00 |
| slagle | EmilienM: you are still using --output though, so it will still be in csv format | 18:01 |
| EmilienM | damn | 18:01 |
| EmilienM | ok | 18:01 |
| EmilienM | a sec | 18:01 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move keystone::auth into service_config_settings https://review.openstack.org/370573 | 18:01 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: dstat: improve output to track high cpu process https://review.openstack.org/374982 | 18:02 |
| *** shardy has quit IRC | 18:09 | |
| *** dmsimard is now known as dmsimard|afk | 18:10 | |
| EmilienM | goo.gl/5xsr7i | 18:11 |
| EmilienM | that's the query I'm using to find when in puppet CI we had the heat bug | 18:12 |
| EmilienM | and all started on September 12th | 18:12 |
| EmilienM | and the promotion wat on 12th, I'll check timestamps but maybe something in Heat between 9th and 12th broke (9th was the previous promotion) | 18:13 |
| *** myoung|biab is now known as myoung | 18:13 | |
| d0ugal | bnemec: thanks | 18:14 |
| EmilienM | ok logstash is still computing the query and it happenned before | 18:14 |
| *** yamahata has quit IRC | 18:15 | |
| EmilienM | first time on 2016-09-13T21:45:38.058Z | 18:15 |
| EmilienM | so now I'm investigating commits between 12 and 13th september in heat | 18:21 |
| EmilienM | because that's where it started to be unstable for us | 18:21 |
| EmilienM | (in puppet CI) | 18:21 |
| EmilienM | my first feeling is about https://github.com/openstack/heat/commit/f18e57e004e65faf0ed2d043384709007f83b2b0 | 18:22 |
| EmilienM | zaneb: ^ wdyt? | 18:22 |
| *** milan has quit IRC | 18:23 | |
| EmilienM | bnemec: when do you see the timeout thing in logstash for tripleo jobs? | 18:23 |
| EmilienM | bnemec: when does it start i mean | 18:23 |
| *** trown|lunch is now known as trown | 18:24 | |
| *** ayoung has quit IRC | 18:25 | |
| * EmilienM running the query agai for tripleo | 18:25 | |
| EmilienM | bnemec: do you think there is a better way to find this problem rather than message: "503 Service Unavailable" ? | 18:25 |
| pradk | has anyone tried upgrades using upstream repos? | 18:26 |
| pradk | bandini, jistr, ^^? | 18:26 |
| EmilienM | so even in bnemec's query, failures started on September 12th | 18:26 |
| bandini | pradk: I am, currently stuck on the major-upgrade-pacemaker step | 18:27 |
| bandini | pradk: here is my diary https://etherpad.openstack.org/p/tripleo-mitaka-newton-upgrades | 18:28 |
| pradk | bandini, what repos are you using when you switch for undercloud upgrades | 18:28 |
| pradk | bandini, thx i'll check | 18:28 |
| pradk | bandini, so instead of sudo rhos-release -P 10 -r 7.3 .. which upstream repo are you using | 18:29 |
| pradk | bandini, that could be why my upgrade is failing, perhaps i have an older snapshot repo | 18:29 |
| bnemec | EmilienM: Timeout or oom? | 18:29 |
| bandini | pradk: https://paste.fedoraproject.org/432843/74568987/ this is what I am doing | 18:30 |
| bandini | the undercloud upgrade has worked pretty much all the time in the last weeks | 18:30 |
| bandini | pradk: what issue are you seeing? | 18:30 |
| pradk | k lemme try that.. i was getting traceback on some dib | 18:31 |
| pradk | bandini, you still had to turn off openstack-* and neutron-* services before upgrade? | 18:32 |
| bandini | pradk: I do it because I had it in my scripts from the previous upgrade cycle, not sure they are strictly still needed | 18:33 |
| bandini | pradk: let's say they don't hurt ;) | 18:33 |
| EmilienM | bnemec: anything that says heat timeout | 18:33 |
| EmilienM | bnemec: I documented my little research https://bugs.launchpad.net/heat/+bug/1626675/comments/1 | 18:34 |
| openstack | Launchpad bug 1626675 in heat "Further memory usage issues with big stacks" [Undecided,New] | 18:34 |
| EmilienM | therve, zaneb: when you get a moment, please look https://bugs.launchpad.net/heat/+bug/1626675/comments/1 | 18:34 |
| pradk | k cool | 18:34 |
| *** rasca has quit IRC | 18:36 | |
| *** snecklifter has quit IRC | 18:36 | |
| *** fzdarsky has quit IRC | 18:37 | |
| zaneb | EmilienM: why is that patch suspicious? | 18:41 |
| EmilienM | zaneb: because it talks about performance a little | 18:41 |
| EmilienM | zaneb: my only assumption is that something broke us on 11/12/13 th | 18:41 |
| zaneb | right, it should have improved performance if anything | 18:42 |
| *** coolsvap has quit IRC | 18:42 | |
| zaneb | EmilienM: what's the 503 the result of? OOM killer? | 18:43 |
| EmilienM | bnemec ^ | 18:45 |
| openstackgerrit | James Slagle proposed openstack/instack-undercloud: Update default VM memory https://review.openstack.org/375054 | 18:45 |
| *** sshnaidm is now known as sshnaidm|afk | 18:45 | |
| EmilienM | https://github.com/openstack/heat/compare/b9d1e30...a9e9b31 | 18:45 |
| EmilienM | mwhahaha, zaneb^ that's the diff between when it used to work fine and when it broke | 18:45 |
| EmilienM | (still for puppet CI) | 18:46 |
| EmilienM | https://github.com/openstack/heat/commit/873a40851dd7807c6de0ee73affb7af2be875519 is also suspicious | 18:48 |
| openstackgerrit | James Slagle proposed openstack/tripleo-docs: Update minimum specs for virt setup https://review.openstack.org/375057 | 18:48 |
| EmilienM | zaneb: wdyt ^? | 18:48 |
| zaneb | EmilienM: https://github.com/openstack/heat/commit/d79236468931db780fe90e4297f5033fe9db24cf is by far the most suspicious of that lot | 18:49 |
| *** yamahata has joined #tripleo | 18:49 | |
| EmilienM | zaneb: why | 18:50 |
| EmilienM | we don't use it in RDO | 18:50 |
| EmilienM | we use oslo deps from latest tag | 18:50 |
| EmilienM | in uc | 18:50 |
| zaneb | the rest just seems really unlikely | 18:51 |
| EmilienM | ¯\_(ツ)_/¯ | 18:51 |
| mwhahaha | we used the same oslodb | 18:51 |
| zaneb | stuff related to specific resource types that you don't use | 18:51 |
| mwhahaha | in puppet | 18:51 |
| EmilienM | mwhahaha: yes I just checked, that's not oslo db | 18:51 |
| zaneb | reverting a patch from a couple of days before | 18:51 |
| zaneb | stuff related to convergence that doesn't do anything yet, and anyway we don't use convergence in TripleO (although we might in this gate) | 18:52 |
| EmilienM | i don't know if we use it in puppet gate | 18:52 |
| zaneb | it's the default, so if you don't *not* use it then you do | 18:52 |
| EmilienM | so in puppet gate we use it | 18:54 |
| openstackgerrit | Merged openstack/tripleo-docs: Add IP Assignment to node_placement https://review.openstack.org/374977 | 18:55 |
| bnemec | I thought we explicitly turned off convergence on the undercloud? | 18:56 |
| EmilienM | yes we do | 18:57 |
| EmilienM | heat::engine::convergence_engine: false | 18:57 |
| mwhahaha | are we assuming the puppet timeout issue is related to the current memory thing? | 18:59 |
| *** absubram has joined #tripleo | 19:01 | |
| zaneb | good question | 19:01 |
| mwhahaha | cause i don't think they are related :D | 19:02 |
| mwhahaha | i think the puppet one is https://github.com/openstack/heat/commit/873a40851dd7807c6de0ee73affb7af2be875519 | 19:02 |
| EmilienM | indeed | 19:02 |
| *** absubram has quit IRC | 19:03 | |
| zaneb | EmilienM: the puppet one is https://bugs.launchpad.net/heat/+bug/1622979 ? | 19:04 |
| openstack | Launchpad bug 1622979 in heat "Stack DELETE IN_PROGRESS: Unhandled error in asynchronous task" [Undecided,New] | 19:04 |
| *** [1]cdearborn has joined #tripleo | 19:05 | |
| *** absubram has joined #tripleo | 19:06 | |
| EmilienM | zaneb: yes | 19:07 |
| dtrainor | I have a stubborn deployment that seems to be stuck. I don't see any errors on either my 1 controller or 1 compute except for an authentication failure for ro_snmp_user (I don't think that's related, but I'll look in to it later). Here's some output from os-collect-config https://paste.fedoraproject.org/432883/57119314/ | 19:07 |
| zaneb | EmilienM: ok, I just closed that as a duplicate of https://bugs.launchpad.net/heat/+bug/1626173 - it's the exact same stack trace | 19:07 |
| openstack | Launchpad bug 1626173 in heat "stack failed to reach DELETE_COMPLETE status (timeout)" [High,Fix released] - Assigned to Crag Wolfe (cwolfe) | 19:07 |
| *** ayoung has joined #tripleo | 19:14 | |
| *** ayoung has quit IRC | 19:14 | |
| *** cdearborn has quit IRC | 19:15 | |
| *** r-mibu has quit IRC | 19:17 | |
| *** r-mibu has joined #tripleo | 19:17 | |
| openstackgerrit | Chris Jones proposed openstack/tripleo-quickstart: Add ssh option IdentitiesOnly. https://review.openstack.org/374769 | 19:22 |
| *** jprovazn has quit IRC | 19:24 | |
| *** zigo_ is now known as zigo | 19:30 | |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder: Shorten DHCP timeout in dhcp-all-interfaces https://review.openstack.org/375073 | 19:30 |
| *** cdearborn has joined #tripleo | 19:35 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Ceilometer Wsgi Mitaka->Newton upgrades https://review.openstack.org/360004 | 19:42 |
| *** [1]cdearborn has quit IRC | 19:49 | |
| bnemec | That moment when you realize you just wiped the vm that had all of your custom heat templates. :-( | 19:50 |
| *** snecklifter has joined #tripleo | 19:50 | |
| *** ayoung has joined #tripleo | 19:51 | |
| *** jeckersb is now known as jeckersb_gone | 19:51 | |
| *** snecklifter has quit IRC | 19:54 | |
| mwhahaha | oh noes | 19:57 |
| *** ayoung has quit IRC | 19:57 | |
| *** chem` has quit IRC | 19:57 | |
| openstackgerrit | Merged openstack/tripleo-common: Fix the default plan creation https://review.openstack.org/371347 | 20:02 |
| *** chem` has joined #tripleo | 20:09 | |
| EmilienM | slagle: what happenned to gate-tripleo-ci-centos-7-nonha-multinode ? | 20:11 |
| EmilienM | 44 min :-O | 20:11 |
| slagle | i regularly see runs under 45 minutes :) | 20:11 |
| slagle | it's all over the map though | 20:11 |
| EmilienM | so dstat | 20:12 |
| EmilienM | http://logs.openstack.org/82/374982/3/check/gate-tripleo-ci-centos-7-nonha-multinode/7c1a7e9/logs/dstat.txt.gz | 20:12 |
| slagle | faster i have ever seen it run is 39 minutes | 20:12 |
| EmilienM | that's super fast | 20:12 |
| *** chem` has quit IRC | 20:13 | |
| *** nyechiel has joined #tripleo | 20:14 | |
| bnemec | The osic nodes are really fast, from what I can tell. | 20:15 |
| bnemec | On a semi-related note, does anyone know why we still have an ovb-nonha job in the experimental pipline? | 20:16 |
| bnemec | *pipeline even | 20:16 |
| EmilienM | to test nonha on 2 nodes I guess? | 20:17 |
| EmilienM | the multinode job test nonha on a single overcloud node | 20:17 |
| EmilienM | I'm not sure of the benefit to keep the ovb non ha indeed | 20:17 |
| EmilienM | I know it not high priority but here are some easy/small patches to support recent version of puppet: https://review.openstack.org/#/q/topic:tripleo/puppet4 | 20:18 |
| openstackgerrit | Merged openstack/python-tripleoclient: Change the level of mocking for the wait_for_stack_ready test https://review.openstack.org/374968 | 20:18 |
| bnemec | Oh wait, I know this actually. | 20:20 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add myself to the planet https://review.openstack.org/374385 | 20:20 |
| bnemec | I'm pretty sure that's there so we can do check experimental on other projects like heat and ironic. | 20:20 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/373722 | 20:21 |
| EmilienM | bnemec: also | 20:22 |
| EmilienM | bnemec: but I'm not sure the reason of running it in tripleo projects | 20:22 |
| pradk | EmilienM, can we merge this.. https://review.openstack.org/#/c/371591/ | 20:22 |
| bnemec | EmilienM: We've only got one experimental-tripleo queue. We'd have to create a separate one to not run it on tripleo projects. | 20:22 |
| bnemec | Which, meh. | 20:23 |
| bnemec | Experimental jobs get lowest priority on test envs anyway. | 20:23 |
| EmilienM | pradk: done | 20:23 |
| pradk | thx | 20:23 |
| *** panda|break is now known as panda | 20:25 | |
| *** Goneri has quit IRC | 20:28 | |
| *** absubram_ has joined #tripleo | 20:32 | |
| *** bfournie has quit IRC | 20:33 | |
| *** jayg is now known as jayg|g0n3 | 20:33 | |
| *** absubram has quit IRC | 20:33 | |
| *** absubram_ is now known as absubram | 20:33 | |
| *** nyechiel has quit IRC | 20:41 | |
| *** lucasagomes is now known as lucas-afk | 20:42 | |
| *** paramite has quit IRC | 20:43 | |
| *** fpan has quit IRC | 20:46 | |
| *** cylopez has joined #tripleo | 20:47 | |
| slagle | scale down is busted | 20:49 |
| slagle | https://bugs.launchpad.net/tripleo/+bug/1626736 | 20:49 |
| openstack | Launchpad bug 1626736 in tripleo " Unable to delete overcloud node" [Critical,New] | 20:49 |
| *** jeckersb_gone is now known as jeckersb | 20:51 | |
| slagle | jrist: does the UI intend to support scaling down? | 20:53 |
| slagle | b/c we don't have a workflow for that. i don't think anyone is working on it | 20:54 |
| jrist | "intend"? | 20:56 |
| jrist | I haven't seen any request for that | 20:56 |
| jrist | in general | 20:56 |
| slagle | yea, intend meaning is it a requirement | 20:57 |
| slagle | i guess not :) | 20:57 |
| jrist | is it something the CLI supports? | 20:57 |
| slagle | yes, although it's currently broken. due to moving the templates to swift | 20:58 |
| jrist | so | 20:58 |
| slagle | it looks like that part would be an easy fix though | 20:58 |
| jrist | the UI should use the same workflow as the CLI | 20:58 |
| jrist | theoretically... | 20:58 |
| slagle | well, there is no mistral workflow | 20:58 |
| rbrady | there could be | 20:58 |
| rbrady | :) | 20:58 |
| slagle | yes :), that's what i was trying to ascertain | 20:59 |
| rbrady | let's file a bug on it so it doesn't get lost | 20:59 |
| slagle | if we needed one | 20:59 |
| slagle | rbrady: i think we could just use https://bugs.launchpad.net/tripleo/+bug/1626736 | 20:59 |
| openstack | Launchpad bug 1626736 in tripleo " Unable to delete overcloud node" [Critical,Confirmed] - Assigned to Carlos Camacho (ccamacho) | 20:59 |
| slagle | rbrady: feel free to reassign it if you'd like | 20:59 |
| jrist | ah that's the fancy name for scaling down? | 20:59 |
| *** mburned is now known as mburned_out | 21:00 | |
| jrist | delete? | 21:00 |
| jrist | :) | 21:00 |
| *** dmsimard|afk is now known as dmsimard | 21:00 | |
| slagle | yea b/c that's what the cli argument is called | 21:00 |
| jrist | affects UI or effects UI | 21:00 |
| jrist | affects | 21:01 |
| *** fpan has joined #tripleo | 21:01 | |
| rbrady | slagle: I've tagged it with workflows and we'll get to it | 21:01 |
| slagle | rbrady: cool, thanks | 21:01 |
| *** rhallisey has quit IRC | 21:02 | |
| *** dsneddon has quit IRC | 21:08 | |
| *** myoung is now known as myoung|gone | 21:10 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove the get_hiera_key function https://review.openstack.org/367367 | 21:10 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names https://review.openstack.org/366529 | 21:10 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the unused service_host arg from node registration https://review.openstack.org/326036 | 21:10 |
| *** dsneddon has joined #tripleo | 21:13 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/375125 | 21:16 |
| *** jbadiapa has quit IRC | 21:32 | |
| *** rcernin has quit IRC | 21:34 | |
| *** fpan has quit IRC | 21:36 | |
| *** bkopilov has quit IRC | 21:37 | |
| *** liverpooler has quit IRC | 21:37 | |
| *** nijaba has quit IRC | 21:37 | |
| *** adarazs has quit IRC | 21:37 | |
| *** zoli_gone-proxy has quit IRC | 21:37 | |
| *** onovy has quit IRC | 21:37 | |
| *** akrzos has quit IRC | 21:37 | |
| *** athomas has quit IRC | 21:37 | |
| *** rdopiera has quit IRC | 21:37 | |
| *** dbecker has quit IRC | 21:37 | |
| *** tdasilva has quit IRC | 21:37 | |
| *** colonwq has quit IRC | 21:37 | |
| *** myoung|gone has quit IRC | 21:37 | |
| *** slagle has quit IRC | 21:37 | |
| *** greghaynes has quit IRC | 21:37 | |
| *** dobson has quit IRC | 21:37 | |
| *** shadower has quit IRC | 21:37 | |
| *** dmanchad has quit IRC | 21:37 | |
| *** bswartz has quit IRC | 21:37 | |
| *** mwhahaha has quit IRC | 21:37 | |
| *** timothyb89 has quit IRC | 21:37 | |
| *** rodrigods has quit IRC | 21:37 | |
| *** jpeeler has quit IRC | 21:37 | |
| *** bandini has quit IRC | 21:37 | |
| *** kbyrne has quit IRC | 21:37 | |
| *** hewbrocca-afk has quit IRC | 21:37 | |
| *** ansiwen has quit IRC | 21:37 | |
| *** bkero has quit IRC | 21:37 | |
| *** mandre has quit IRC | 21:37 | |
| *** lazy_prince has quit IRC | 21:37 | |
| *** toure has quit IRC | 21:37 | |
| *** markmc has quit IRC | 21:37 | |
| *** rajinir has quit IRC | 21:37 | |
| *** tzumainn has quit IRC | 21:37 | |
| *** oshvartz has quit IRC | 21:37 | |
| *** ccamacho|afk has quit IRC | 21:37 | |
| *** jroll has quit IRC | 21:37 | |
| *** eggmaster has quit IRC | 21:37 | |
| *** CaptTofu has quit IRC | 21:37 | |
| *** sirushti has quit IRC | 21:37 | |
| *** jayg|g0n3 has quit IRC | 21:37 | |
| *** HenryG has quit IRC | 21:37 | |
| *** mrunge has quit IRC | 21:37 | |
| *** andreaf has quit IRC | 21:37 | |
| *** mgagne has quit IRC | 21:37 | |
| *** athomas has joined #tripleo | 21:38 | |
| *** rdopiera has joined #tripleo | 21:38 | |
| *** dbecker has joined #tripleo | 21:38 | |
| *** tdasilva has joined #tripleo | 21:38 | |
| *** colonwq has joined #tripleo | 21:38 | |
| *** myoung|gone has joined #tripleo | 21:38 | |
| *** slagle has joined #tripleo | 21:38 | |
| *** greghaynes has joined #tripleo | 21:38 | |
| *** dobson has joined #tripleo | 21:38 | |
| *** bswartz has joined #tripleo | 21:38 | |
| *** shadower has joined #tripleo | 21:38 | |
| *** dmanchad has joined #tripleo | 21:38 | |
| *** mwhahaha has joined #tripleo | 21:38 | |
| *** timothyb89 has joined #tripleo | 21:38 | |
| *** gregwork has quit IRC | 21:40 | |
| *** rajinir has joined #tripleo | 21:40 | |
| *** tzumainn has joined #tripleo | 21:40 | |
| *** oshvartz has joined #tripleo | 21:40 | |
| *** ccamacho|afk has joined #tripleo | 21:40 | |
| *** jroll has joined #tripleo | 21:40 | |
| *** eggmaster has joined #tripleo | 21:40 | |
| *** CaptTofu has joined #tripleo | 21:40 | |
| *** sirushti has joined #tripleo | 21:40 | |
| *** jayg|g0n3 has joined #tripleo | 21:40 | |
| *** HenryG has joined #tripleo | 21:40 | |
| *** mrunge has joined #tripleo | 21:40 | |
| *** andreaf has joined #tripleo | 21:40 | |
| *** mgagne has joined #tripleo | 21:40 | |
| *** mwhahaha has quit IRC | 21:40 | |
| *** fpan has joined #tripleo | 21:41 | |
| *** bkopilov has joined #tripleo | 21:41 | |
| *** liverpooler has joined #tripleo | 21:41 | |
| *** nijaba has joined #tripleo | 21:41 | |
| *** adarazs has joined #tripleo | 21:41 | |
| *** zoli_gone-proxy has joined #tripleo | 21:41 | |
| *** cylopez has quit IRC | 21:41 | |
| *** CaptTofu has quit IRC | 21:41 | |
| *** onovy has joined #tripleo | 21:41 | |
| *** akrzos has joined #tripleo | 21:41 | |
| *** NachoDuck has quit IRC | 21:41 | |
| *** rodrigods has joined #tripleo | 21:42 | |
| *** jpeeler has joined #tripleo | 21:42 | |
| *** bandini has joined #tripleo | 21:42 | |
| *** kbyrne has joined #tripleo | 21:42 | |
| *** hewbrocca-afk has joined #tripleo | 21:42 | |
| *** bkero has joined #tripleo | 21:42 | |
| *** ansiwen has joined #tripleo | 21:42 | |
| *** mandre has joined #tripleo | 21:42 | |
| *** lazy_prince has joined #tripleo | 21:42 | |
| *** toure has joined #tripleo | 21:42 | |
| *** markmc has joined #tripleo | 21:42 | |
| EmilienM | great now irc is broken | 21:43 |
| EmilienM | really a good day today | 21:43 |
| *** rajinir has quit IRC | 21:44 | |
| *** hrybacki has quit IRC | 21:45 | |
| panda | EmilienM: wait until it starts to rain. | 21:46 |
| EmilienM | it's raining here | 21:48 |
| *** jcoufal_ has quit IRC | 21:49 | |
| *** jcoufal has joined #tripleo | 21:50 | |
| *** jcoufal has quit IRC | 21:50 | |
| panda | lol | 21:52 |
| *** ayoung has joined #tripleo | 21:56 | |
| *** gregwork has joined #tripleo | 22:02 | |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder: Shorten DHCP timeout in dhcp-all-interfaces https://review.openstack.org/375073 | 22:03 |
| *** NachoDuck has joined #tripleo | 22:03 | |
| bnemec | ^should be 9 minutes off every CI run. | 22:04 |
| bnemec | You're welcome. :-) | 22:04 |
| EmilienM | it's a lot, thanks | 22:05 |
| jrist | nice bnemec | 22:06 |
| *** rajinir has joined #tripleo | 22:10 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-docs: Update minimum memory requirements https://review.openstack.org/375136 | 22:12 |
| *** hrybacki has joined #tripleo | 22:13 | |
| panda | bnemec: I'm reading http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/network_isolation.html#create-network-environment-file as a night reading .. but still have a lot of doubts .. do you have 10 minutes tomorrow to walk me through a configuration for my test env ? | 22:18 |
| *** CaptTofu has joined #tripleo | 22:18 | |
| *** cdearborn has quit IRC | 22:19 | |
| *** jrist has quit IRC | 22:19 | |
| bnemec | panda: Yeah, I should be able to. Do you generally join the DF scrum? We could maybe stay on after that talk it over. | 22:22 |
| panda | bnemec: I always intend to join, then miss it for a reason or another. Bt scrum comes late in my day, and I wanted to set up the env by the end of the week. | 22:25 |
| bnemec | panda: Okay, we can figure something out tomorrow. We can always jump on my bluejeans call too. | 22:26 |
| *** egafford has quit IRC | 22:27 | |
| panda | bnemec: thanks a lot! ping you tomorrow then. Have a nice evening | 22:28 |
| *** panda is now known as panda|zZ | 22:28 | |
| dsneddon | bnemec, Do you know of anywhere where I could find documentation on setting per-node ExtraConfig options? I think I've seen an example where a JSON map was passed to a script, but I can't find one now. | 22:40 |
| *** mwhahaha has joined #tripleo | 22:55 | |
| *** social has joined #tripleo | 22:57 | |
| *** ayoung has quit IRC | 23:13 | |
| *** HenryG has quit IRC | 23:28 | |
| *** HenryG has joined #tripleo | 23:28 | |
| *** mburned_out is now known as mburned | 23:29 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4 https://review.openstack.org/371209 | 23:48 |
| bnemec | dsneddon: http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/node_config.html | 23:53 |
| bnemec | Or http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/extra_config.html | 23:53 |
| bnemec | dsneddon: Actually maybe what you're looking for is http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/node_specific_hieradata.html | 23:54 |
| bnemec | That one uses a JSON map. | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
