openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Implement nginx sidecar https://review.openstack.org/555386 | 00:01 |
---|---|---|
*** sawblade6 has joined #openstack-helm | 00:02 | |
*** sawblade_ has quit IRC | 00:05 | |
*** unicell has quit IRC | 00:06 | |
*** sawblade_ has joined #openstack-helm | 00:20 | |
*** sawblade6 has quit IRC | 00:22 | |
*** yamamoto has joined #openstack-helm | 00:28 | |
*** yamamoto has quit IRC | 00:34 | |
*** sawblade6 has joined #openstack-helm | 00:35 | |
*** sawblade_ has quit IRC | 00:37 | |
*** sawblade_ has joined #openstack-helm | 01:04 | |
*** sawblade6 has quit IRC | 01:06 | |
*** sawblade6 has joined #openstack-helm | 01:09 | |
*** sawblade_ has quit IRC | 01:10 | |
*** sawblade_ has joined #openstack-helm | 01:12 | |
*** sawblade6 has quit IRC | 01:13 | |
*** sawblade6 has joined #openstack-helm | 01:16 | |
*** sawblade_ has quit IRC | 01:17 | |
openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Ceph: allow un-interupted operation during partial mon outage https://review.openstack.org/558661 | 01:26 |
*** yamamoto has joined #openstack-helm | 01:30 | |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon chart https://review.openstack.org/558326 | 01:30 |
*** yamamoto has quit IRC | 01:35 | |
*** yangyapeng has joined #openstack-helm | 01:48 | |
*** yangyape_ has joined #openstack-helm | 01:50 | |
*** yangyapeng has quit IRC | 01:50 | |
*** unicell has joined #openstack-helm | 02:09 | |
*** unicell has quit IRC | 02:10 | |
*** unicell has joined #openstack-helm | 02:11 | |
*** unicell has quit IRC | 02:12 | |
openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm master: [DNM] experiment: use alternate (smaller) entrypoint container https://review.openstack.org/558670 | 02:19 |
*** felipemonteiro has joined #openstack-helm | 02:22 | |
*** yamamoto has joined #openstack-helm | 02:32 | |
*** sawblade_ has joined #openstack-helm | 02:34 | |
*** sawblade6 has quit IRC | 02:34 | |
*** yamamoto has quit IRC | 02:37 | |
*** sawblade6 has joined #openstack-helm | 02:40 | |
*** mdih has joined #openstack-helm | 02:40 | |
*** sawblade_ has quit IRC | 02:42 | |
*** yangyape_ has quit IRC | 02:50 | |
*** yangyapeng has joined #openstack-helm | 02:50 | |
*** sawblade_ has joined #openstack-helm | 02:54 | |
*** felipemonteiro has quit IRC | 02:54 | |
*** yangyapeng has quit IRC | 02:55 | |
*** sawblade6 has quit IRC | 02:56 | |
*** gagehugo has quit IRC | 02:59 | |
*** gagehugo has joined #openstack-helm | 03:03 | |
*** sawblade6 has joined #openstack-helm | 03:16 | |
*** sawblade_ has quit IRC | 03:18 | |
*** yangyapeng has joined #openstack-helm | 03:19 | |
*** sawblade_ has joined #openstack-helm | 03:20 | |
*** sawblade6 has quit IRC | 03:22 | |
*** yangyapeng has quit IRC | 03:24 | |
*** yamamoto has joined #openstack-helm | 03:33 | |
*** yangyapeng has joined #openstack-helm | 03:35 | |
*** yamamoto has quit IRC | 03:38 | |
*** yangyapeng has quit IRC | 03:40 | |
*** sawblade6 has joined #openstack-helm | 03:52 | |
*** sawblade_ has quit IRC | 03:55 | |
*** yangyapeng has joined #openstack-helm | 04:01 | |
*** yangyapeng has quit IRC | 04:06 | |
*** sawblade_ has joined #openstack-helm | 04:28 | |
*** yangyapeng has joined #openstack-helm | 04:30 | |
*** sawblade6 has quit IRC | 04:31 | |
*** sawblade6 has joined #openstack-helm | 04:33 | |
*** yamamoto has joined #openstack-helm | 04:35 | |
*** sawblade_ has quit IRC | 04:35 | |
*** yamamoto has quit IRC | 04:40 | |
*** sawblade_ has joined #openstack-helm | 04:40 | |
*** sawblade6 has quit IRC | 04:42 | |
openstackgerrit | Merged openstack/openstack-helm master: RabbitMQ: only request 256Mi for PVC by default https://review.openstack.org/558258 | 04:48 |
*** yamamoto has joined #openstack-helm | 04:51 | |
*** sawblade6 has joined #openstack-helm | 04:59 | |
*** sawblade_ has quit IRC | 05:01 | |
*** yangyapeng has quit IRC | 05:18 | |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon chart https://review.openstack.org/558326 | 05:29 |
*** sawblade_ has joined #openstack-helm | 05:42 | |
*** yangyapeng has joined #openstack-helm | 05:43 | |
*** sawblade6 has quit IRC | 05:45 | |
*** yangyapeng has quit IRC | 05:48 | |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: fixes image for ceph mon-check https://review.openstack.org/558695 | 05:50 |
*** yangyapeng has joined #openstack-helm | 06:02 | |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP - splitting ceph-client chart https://review.openstack.org/558700 | 06:06 |
*** yangyapeng has quit IRC | 06:07 | |
*** yangyapeng has joined #openstack-helm | 06:23 | |
*** sawblade6 has joined #openstack-helm | 06:24 | |
*** sawblade_ has quit IRC | 06:27 | |
*** yangyapeng has quit IRC | 06:28 | |
*** sawblade_ has joined #openstack-helm | 07:06 | |
*** sawblade6 has quit IRC | 07:09 | |
openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm master: [DNM] experiment: use alternate (smaller) entrypoint container https://review.openstack.org/558670 | 07:09 |
*** sawblade6 has joined #openstack-helm | 07:15 | |
*** sawblade_ has quit IRC | 07:18 | |
openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm master: Neutron: Update daemonset labels and mixins https://review.openstack.org/555461 | 07:32 |
*** yangyapeng has joined #openstack-helm | 07:34 | |
*** gkadam has joined #openstack-helm | 07:49 | |
*** sawblade_ has joined #openstack-helm | 07:52 | |
*** sawblade6 has quit IRC | 07:54 | |
*** sawblade6 has joined #openstack-helm | 08:04 | |
*** sawblade_ has quit IRC | 08:06 | |
*** goldenfri has quit IRC | 08:27 | |
*** korzen has quit IRC | 08:31 | |
*** yangyapeng has quit IRC | 08:50 | |
*** yangyapeng has joined #openstack-helm | 09:09 | |
*** yangyapeng has quit IRC | 09:14 | |
*** sawblade_ has joined #openstack-helm | 09:17 | |
*** sawblade6 has quit IRC | 09:20 | |
*** sawblade6 has joined #openstack-helm | 09:23 | |
*** sawblade_ has quit IRC | 09:25 | |
*** yangyapeng has joined #openstack-helm | 09:26 | |
*** yangyapeng has quit IRC | 09:31 | |
openstackgerrit | Hyunsun Moon proposed openstack/openstack-helm master: openvswitch: allow setting ovsdb connection using vsctl https://review.openstack.org/558764 | 09:31 |
openstackgerrit | Hyunsun Moon proposed openstack/openstack-helm master: openvswitch: use pidfile option https://review.openstack.org/558772 | 09:49 |
*** yangyapeng has joined #openstack-helm | 09:53 | |
*** yangyapeng has quit IRC | 09:58 | |
*** yamamoto_ has joined #openstack-helm | 10:08 | |
*** anticw_ has joined #openstack-helm | 10:11 | |
*** yamamoto has quit IRC | 10:15 | |
*** anticw has quit IRC | 10:15 | |
*** sawblade_ has joined #openstack-helm | 10:26 | |
*** sawblade6 has quit IRC | 10:28 | |
*** sawblade6 has joined #openstack-helm | 10:43 | |
*** sawblade_ has quit IRC | 10:45 | |
*** pbourke has quit IRC | 10:47 | |
*** pbourke has joined #openstack-helm | 10:47 | |
*** sawblade6 has quit IRC | 10:49 | |
*** sawblade6 has joined #openstack-helm | 10:51 | |
*** sawblade_ has joined #openstack-helm | 11:07 | |
*** sawblade6 has quit IRC | 11:09 | |
*** sawblade6 has joined #openstack-helm | 11:17 | |
*** sawblade_ has quit IRC | 11:21 | |
*** mdih has quit IRC | 11:25 | |
*** sawblade_ has joined #openstack-helm | 11:52 | |
*** sawblade6 has quit IRC | 11:54 | |
*** sawblade_ has quit IRC | 11:57 | |
*** sawblade6 has joined #openstack-helm | 11:58 | |
*** yangyapeng has joined #openstack-helm | 12:02 | |
*** sawblade_ has joined #openstack-helm | 12:02 | |
*** sawblade6 has quit IRC | 12:04 | |
*** yangyapeng has quit IRC | 12:06 | |
*** yangyapeng has joined #openstack-helm | 12:14 | |
*** sawblade6 has joined #openstack-helm | 12:17 | |
*** sawblade_ has quit IRC | 12:19 | |
*** sawblade6 has quit IRC | 12:23 | |
*** yangyapeng has quit IRC | 12:25 | |
*** sawblade6 has joined #openstack-helm | 12:26 | |
*** mdih has joined #openstack-helm | 12:28 | |
*** sawblade_ has joined #openstack-helm | 12:35 | |
*** sawblade6 has quit IRC | 12:37 | |
*** adrianreza has quit IRC | 12:37 | |
*** yangyapeng has joined #openstack-helm | 12:42 | |
*** yangyapeng has quit IRC | 12:47 | |
*** sawblade6 has joined #openstack-helm | 12:51 | |
openstackgerrit | Scott Hussey proposed openstack/openstack-helm master: [WIP] Postgres Helm Test support https://review.openstack.org/558637 | 12:52 |
*** sawblade_ has quit IRC | 12:53 | |
*** sawblade_ has joined #openstack-helm | 12:56 | |
*** sawblade6 has quit IRC | 12:57 | |
*** sawblade_ has quit IRC | 13:02 | |
openstackgerrit | Scott Hussey proposed openstack/openstack-helm master: [WIP] Postgres Helm Test support https://review.openstack.org/558637 | 13:10 |
*** sawblade6 has joined #openstack-helm | 13:10 | |
*** yangyapeng has joined #openstack-helm | 13:30 | |
*** yangyapeng has quit IRC | 13:31 | |
*** goutham has joined #openstack-helm | 13:33 | |
*** mdih has quit IRC | 13:37 | |
*** julim has quit IRC | 13:39 | |
*** sawblade_ has joined #openstack-helm | 13:40 | |
*** sawblade6 has quit IRC | 13:43 | |
*** goutham has quit IRC | 13:49 | |
openstackgerrit | Scott Hussey proposed openstack/openstack-helm master: [WIP] Postgres Helm Test support https://review.openstack.org/558637 | 14:01 |
*** sawblade6 has joined #openstack-helm | 14:05 | |
openstackgerrit | Scott Hussey proposed openstack/openstack-helm master: [WIP] Postgres Helm Test support https://review.openstack.org/558637 | 14:06 |
*** sawblade_ has quit IRC | 14:08 | |
openstackgerrit | Scott Hussey proposed openstack/openstack-helm master: [WIP] Postgres Helm Test support https://review.openstack.org/558637 | 14:13 |
*** julim has joined #openstack-helm | 14:26 | |
openstackgerrit | Scott Hussey proposed openstack/openstack-helm master: [WIP] Postgres Helm Test support https://review.openstack.org/558637 | 14:28 |
*** sawblade_ has joined #openstack-helm | 14:35 | |
*** sawblade6 has quit IRC | 14:35 | |
*** mdih has joined #openstack-helm | 14:35 | |
*** mdih has quit IRC | 14:37 | |
*** eeiden has joined #openstack-helm | 14:43 | |
openstackgerrit | Scott Hussey proposed openstack/openstack-helm master: [WIP] Postgres Helm Test support https://review.openstack.org/558637 | 14:43 |
*** mdih has joined #openstack-helm | 14:46 | |
*** felipemonteiro has joined #openstack-helm | 14:52 | |
openstackgerrit | Scott Hussey proposed openstack/openstack-helm master: Postgres Helm Test support https://review.openstack.org/558637 | 15:05 |
openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: Adding TLS support for sidecar https://review.openstack.org/558248 | 15:09 |
openstackgerrit | Sean Eagan proposed openstack/openstack-helm master: Add image build for kubernetes-entrypoint https://review.openstack.org/558622 | 15:11 |
*** sawblade6 has joined #openstack-helm | 15:17 | |
*** sawblade_ has quit IRC | 15:19 | |
*** felipemonteiro_ has joined #openstack-helm | 15:22 | |
*** MarkBaker has quit IRC | 15:23 | |
*** MarkBaker has joined #openstack-helm | 15:24 | |
*** felipemonteiro has quit IRC | 15:26 | |
*** felipemonteiro__ has joined #openstack-helm | 15:33 | |
*** felipemonteiro_ has quit IRC | 15:33 | |
*** pabelanger has joined #openstack-helm | 15:34 | |
openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Ceph: Fix config helper image and bump version https://review.openstack.org/558851 | 15:35 |
*** felipemonteiro has joined #openstack-helm | 15:36 | |
pabelanger | morning, please http://lists.openstack.org/pipermail/openstack-dev/2018-April/129060.html we've proposed the removal of fedora-26 nodes from nodepool and the helm project is still using them. This is going to result in NODE_FAILURE for the project, blocking CI. I wanted to give a heads up and hope somebody is able to land the patch linked in the ML. Thanks! | 15:36 |
*** mdih has quit IRC | 15:36 | |
openstackgerrit | Rakesh Patnaik proposed openstack/openstack-helm-infra master: Alert rules in prometheus to support nagios based monitoring via alert metric queries https://review.openstack.org/555700 | 15:39 |
*** goldenfri has joined #openstack-helm | 15:45 | |
openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Ceph: Update armada manifests to remove jobs on upgrade correctly https://review.openstack.org/558854 | 15:46 |
*** ianychoi__ is now known as ianychoi | 15:50 | |
*** MarkBaker has quit IRC | 15:53 | |
openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Ceph: Update armada manifests to remove jobs on upgrade correctly https://review.openstack.org/558854 | 16:02 |
openstackgerrit | Pete Birley proposed openstack/openstack-helm master: DNM: Armada: test update/upgrade https://review.openstack.org/558860 | 16:05 |
openstackgerrit | Scott Hussey proposed openstack/openstack-helm master: Postgres Helm Test support https://review.openstack.org/558637 | 16:09 |
openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: Gate: Add heat template to bring up a KubeADM cluster with OSH-Infra https://review.openstack.org/550684 | 16:15 |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 16:17 |
*** sawblade_ has joined #openstack-helm | 16:31 | |
openstackgerrit | Paul Belanger proposed openstack/openstack-helm master: Move openstack-helm-multinode-(centos|fedora) to experimental pipeline https://review.openstack.org/558870 | 16:32 |
*** sawblade6 has quit IRC | 16:33 | |
openstackgerrit | Paul Belanger proposed openstack/openstack-helm master: Move openstack-helm-multinode-(centos|fedora) to experimental pipeline https://review.openstack.org/558870 | 16:37 |
pabelanger | portdirect: v1k0d3n: I've created a ML post about ^ too, please take a moment to review. This will help reduce the number of nodes each helm patch uses from nodepool | 16:39 |
*** sawblade6 has joined #openstack-helm | 16:39 | |
*** sawblade_ has quit IRC | 16:41 | |
*** MarkBaker has joined #openstack-helm | 16:43 | |
v1k0d3n | pabelanger looking over now. thank you for the PS and suggestion. | 16:49 |
anticw_ | pabelanger: i wondered if there might be better logic for when to run jobs | 17:00 |
anticw_ | pabelanger: or even set a priority on them, so that non-voting 'good to know' jobs only run when there is nothing more pressing | 17:01 |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 17:01 |
pabelanger | anticw_: zuul has some ability to do conditional job runs, eg; files / irrelevant-files. But ideally, non-voting jobs should only be non-voting for a minimal about of time. Otherwise, we find they tend to bitrot. In the case here, since they are 5 nodes for centos / fedora, it does get expensive to keep them non-voting | 17:03 |
pabelanger | moving them the experimental still give users the ability to run time, and help get them back to voting | 17:04 |
anticw_ | i think that's fair, we can look at running those experimental jobs internally when/if we want | 17:04 |
pabelanger | yah, and nothing saying they can be moved back to check / gate when ready | 17:05 |
pabelanger | can't* | 17:05 |
*** felipemonteiro_ has joined #openstack-helm | 17:05 | |
v1k0d3n | lamt, portdirect and mattmceuen: you guys around? | 17:05 |
v1k0d3n | portdirect | 17:06 |
mattmceuen | Hey yep | 17:06 |
v1k0d3n | sounds all fair to me, but lamt portdirect and anticw_ have done most of the work. changes look/sound gtm pabelanger | 17:06 |
pabelanger | thanks! | 17:07 |
v1k0d3n | thanks for the help/guidance pabelanger. | 17:08 |
mattmceuen | pabelanger catching up and will get back to you, afk at the moment. Thanks for the heads up! | 17:08 |
*** felipemonteiro__ has quit IRC | 17:08 | |
clarkb | anticw_: pabelanger right I think the bigger issue isn't that you want to run them only when its useful but that they don't seem to function at all | 17:13 |
clarkb | anticw_: pabelanger you need to fix the don't seem to function at all before it makes sense to optimize beyond that | 17:13 |
*** felipemonteiro_ has quit IRC | 17:13 | |
*** felipemonteiro_ has joined #openstack-helm | 17:14 | |
clarkb | the other day portdirect offered to dig into the failures but I haven't heard back on that yet. Basically the nova-db-sync container was eating up a ton of time failing then eventually succeeding but the time it consumed was enough the job timed out by the time everything behind it started moving again | 17:14 |
clarkb | and I couldn't find logs for the failing nova-db-sync | 17:14 |
*** felipemonteiro_ has quit IRC | 17:18 | |
*** eeiden has quit IRC | 17:19 | |
*** felipemonteiro_ has joined #openstack-helm | 17:20 | |
*** MarkBaker has quit IRC | 17:25 | |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 17:28 |
*** sawblade_ has joined #openstack-helm | 17:43 | |
*** sawblade6 has quit IRC | 17:46 | |
*** sawblade6 has joined #openstack-helm | 17:56 | |
anticw_ | clarkb: nova-db-sync does a *lot* | 17:58 |
*** sawblade_ has quit IRC | 17:58 | |
anticw_ | it typically runs 2-3 tims for me here, that is it does at most 100 schema updates at a time i think then terminates | 17:58 |
clarkb | anticw_: sure but it doesn't take 16 minutes normally | 17:58 |
anticw_ | so to get all 350+ it seens it has to restart a few times | 17:58 |
*** yamamoto_ has quit IRC | 17:58 | |
anticw_ | clarkb: no, but it's a *lot* of sync IO | 17:59 |
clarkb | anticw_: ya but in the failing case the container went ot lunch for 16 minutes or so | 17:59 |
clarkb | and then failed | 17:59 |
anticw_ | we should probably just make db IO async and avoid the innodb (whatever maria calls it now) double write in the gates | 17:59 |
clarkb | which prevented a bunch of others from starting behind it | 17:59 |
clarkb | the one that succeeded took ~2 minutes | 17:59 |
*** yamamoto has joined #openstack-helm | 17:59 | |
clarkb | but by then the job had timed out the test | 17:59 |
anticw_ | if here are spinning disks and it's sync IO w/ something to take the edge off, you end up feeling the rotation/seek limitations of disks | 17:59 |
anticw_ | clarkb: diff vm on a diff node with a subtle different raid config? | 18:00 |
clarkb | anticw_: no clue, we consume what are basically public cloud resources and their specific configurations are not somethign we know a ton about. We know some clouds boot off of ceph and others use local storage but thats about it | 18:00 |
anticw_ | clarkb: later i will see if i can get mariadb in 'wreckless abandon mode' (my words) and put up a PS to see if that helps | 18:00 |
anticw_ | clarkb: some builders have *really* slow IO | 18:01 |
clarkb | anticw_: it would also be good to collect the logs from failed containers | 18:01 |
*** felipemonteiro__ has joined #openstack-helm | 18:01 | |
anticw_ | clarkb: yeah, we should def do that as well | 18:01 |
clarkb | at least I don't seem to see where they are collected if they are collected | 18:01 |
anticw_ | it should be there | 18:01 |
anticw_ | do you recall the PS# ? | 18:01 |
*** felipemonteiro_ has quit IRC | 18:02 | |
clarkb | anticw_: http://logs.openstack.org/14/558214/2/check/openstack-helm-multinode-fedora/090b77f/primary/pod-logs/openstack/nova-db-sync-d7lwm/nova-db-sync.txt was the one I was looking at | 18:02 |
clarkb | it started 18 minutes after http://logs.openstack.org/14/558214/2/check/openstack-helm-multinode-fedora/090b77f/primary/pod-logs/openstack/nova-db-sync-d7lwm/init.txt said it was done initing. But then portdirect found a different instance ran an failed /me looks for that log | 18:03 |
clarkb | http://logs.openstack.org/14/558214/2/check/openstack-helm-multinode-fedora/090b77f/primary/objects/namespaced/openstack/pods/nova-db-sync-d7lwm.txt | 18:03 |
clarkb | its that missing ~16 minutes or so that we want to debug I think | 18:03 |
openstackgerrit | Sean Eagan proposed openstack/openstack-helm master: Add image build for kubernetes-entrypoint https://review.openstack.org/558622 | 18:04 |
anticw_ | oh, that looks like the db hung | 18:06 |
*** yamamoto_ has joined #openstack-helm | 18:06 | |
clarkb | anticw_: and basically by the time it succeeded 18 minutes later the test had decided thigns were timed out and failed the job | 18:07 |
anticw_ | is it usually nova-db-sync ? | 18:09 |
*** yamamoto has quit IRC | 18:10 | |
*** felipemonteiro has quit IRC | 18:10 | |
portdirect | anticw_: clarkb sorry snowed under atm | 18:10 |
anticw_ | portdirect: it's fine, this can sit on the back-burner for a bit | 18:10 |
portdirect | but it would be great if we added a `--previous` to the log collection playbooks | 18:11 |
portdirect | which would get us the logs from pods that had failed, and then been respawened | 18:11 |
clarkb | anticw_: http://logs.openstack.org/14/558214/2/check/openstack-helm-multinode-fedora/090b77f/primary/objects/namespaced/openstack/pods/nova-db-sync-d7lwm.txt if you look there it appears to be. One was terminated for reason error after ~16 minutes | 18:11 |
portdirect | sry not to be able to help atm | 18:11 |
clarkb | anticw_: then the subsequent attempt succeeds much more quikcly | 18:11 |
srwilkers | portdirect: if nobody adds it before I get back to a keyboard, I can throw something together to get em | 18:16 |
*** spiette has quit IRC | 18:16 | |
*** spiette has joined #openstack-helm | 18:19 | |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 18:21 |
*** felipemonteiro has joined #openstack-helm | 18:38 | |
*** felipemonteiro__ has quit IRC | 18:38 | |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 18:41 |
openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm master: [DNN] MariaDB: use unsafe defaults to test performance https://review.openstack.org/558895 | 18:42 |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 18:51 |
*** gkadam has quit IRC | 19:00 | |
*** sawblade_ has joined #openstack-helm | 19:08 | |
*** sawblade6 has quit IRC | 19:10 | |
*** felipemonteiro_ has joined #openstack-helm | 19:18 | |
*** felipemonteiro has quit IRC | 19:21 | |
*** sawblade6 has joined #openstack-helm | 19:24 | |
*** pabelanger has left #openstack-helm | 19:25 | |
*** sawblade_ has quit IRC | 19:26 | |
*** sawblade_ has joined #openstack-helm | 19:27 | |
*** sawblade6 has quit IRC | 19:29 | |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 19:41 |
*** gkadam has joined #openstack-helm | 19:42 | |
*** felipemonteiro__ has joined #openstack-helm | 19:48 | |
*** felipemonteiro__ has quit IRC | 19:49 | |
*** felipemonteiro__ has joined #openstack-helm | 19:49 | |
*** sawblade6 has joined #openstack-helm | 19:51 | |
*** felipemonteiro_ has quit IRC | 19:52 | |
*** sawblade_ has quit IRC | 19:53 | |
*** felipemonteiro_ has joined #openstack-helm | 19:54 | |
*** felipemonteiro__ has quit IRC | 19:57 | |
*** sawblade6 has quit IRC | 19:59 | |
openstackgerrit | Stacey Fletcher proposed openstack/openstack-helm-addons master: Jenkins chart https://review.openstack.org/544780 | 20:03 |
*** sawblade6 has joined #openstack-helm | 20:05 | |
*** yangyapeng has joined #openstack-helm | 20:05 | |
openstackgerrit | Stacey Fletcher proposed openstack/openstack-helm-addons master: Jenkins chart https://review.openstack.org/544780 | 20:05 |
*** sawblade_ has joined #openstack-helm | 20:09 | |
*** yangyapeng has quit IRC | 20:09 | |
*** sawblade6 has quit IRC | 20:11 | |
openstackgerrit | Merged openstack/openstack-helm master: Move openstack-helm-multinode-(centos|fedora) to experimental pipeline https://review.openstack.org/558870 | 20:12 |
anticw_ | clarkb: i'm not even sure if nova-db-sync is working properly, it ends with ERROR in a way i hadn't noticed until today | 20:17 |
*** sawblade6 has joined #openstack-helm | 20:17 | |
*** sawblade_ has quit IRC | 20:19 | |
openstackgerrit | Tin Lam proposed openstack/openstack-helm master: [WIP] Adding TLS support for sidecar https://review.openstack.org/558248 | 20:26 |
clarkb | anticw_: ya that is why I was hoping we could get logs for it to see how it is failing | 20:27 |
anticw_ | clarkb: so i did a change that should make it faster ... which it maybe does, hard to say | 20:28 |
anticw_ | but in testing that i see db migration stop (prematurely) in a way i never noticed | 20:28 |
clarkb | can we get logs for these types of failures too? | 20:28 |
clarkb | that way this and other issues become debuggable | 20:28 |
anticw_ | http://logs.openstack.org/70/558870/2/check/openstack-helm-dev-deploy-nfs-ovs/f60c86e/primary/pod-logs/openstack/nova-db-sync-hs25q/nova-db-sync.txt | 20:29 |
anticw_ | see the end of that | 20:29 |
anticw_ | googling i see a couple (perhaps 3) fixes over time for that error, as best as i can tell those are in the image we are using | 20:29 |
anticw_ | i don't know if portdirect has some ideas ... could the sr-iov stuff have affected this? | 20:30 |
*** yamamoto_ has quit IRC | 20:31 | |
*** unicell has joined #openstack-helm | 20:31 | |
clarkb | anticw_: ya we noticed those errors to but they occur in the much faster and successful sync aiui | 20:33 |
clarkb | but we don't have logs to see why the earlier sync is so slow | 20:33 |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 20:34 |
anticw_ | 'ktail -t --all-namespace f > some/file.log' would potentially be useful if a bit painful to deal with | 20:35 |
clarkb | it could be that the earlier failed sync leads to these errors we do see | 20:37 |
clarkb | so getting that information will be very valuable | 20:37 |
clarkb | (also we can fix these errors we do see any it may not fix the underlying issue) | 20:37 |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 20:38 |
openstackgerrit | Randeep Jalli proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon chart https://review.openstack.org/558326 | 20:40 |
*** sawblade_ has joined #openstack-helm | 20:41 | |
*** sawblade6 has quit IRC | 20:41 | |
*** julim has quit IRC | 20:42 | |
*** sawblade6 has joined #openstack-helm | 20:49 | |
*** sawblade_ has quit IRC | 20:51 | |
openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Ceph: Fix config helper image and bump version https://review.openstack.org/558851 | 20:52 |
openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm master: [DNN] MariaDB: use unsafe defaults to test performance https://review.openstack.org/558895 | 21:26 |
*** yamamoto has joined #openstack-helm | 21:28 | |
*** yamamoto has quit IRC | 21:33 | |
openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implements ceph-client chart https://review.openstack.org/558700 | 21:34 |
*** gkadam has quit IRC | 21:36 | |
*** julim has joined #openstack-helm | 21:38 | |
*** yangyapeng has joined #openstack-helm | 21:53 | |
*** yangyapeng has quit IRC | 21:58 | |
*** sawblade_ has joined #openstack-helm | 22:03 | |
*** sawblade6 has quit IRC | 22:05 | |
*** felipemonteiro_ has quit IRC | 22:21 | |
*** sawblade6 has joined #openstack-helm | 22:30 | |
*** yamamoto has joined #openstack-helm | 22:30 | |
*** sawblade_ has quit IRC | 22:33 | |
*** yamamoto has quit IRC | 22:36 | |
*** sawblade_ has joined #openstack-helm | 22:53 | |
*** sawblade6 has quit IRC | 22:55 | |
*** felipemonteiro has joined #openstack-helm | 23:00 | |
*** sawblade6 has joined #openstack-helm | 23:07 | |
*** sawblade_ has quit IRC | 23:10 | |
*** sawblade_ has joined #openstack-helm | 23:16 | |
*** sawblade6 has quit IRC | 23:18 | |
openstackgerrit | Randeep Jalli proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon chart https://review.openstack.org/558326 | 23:21 |
*** yamamoto has joined #openstack-helm | 23:32 | |
*** sawblade6 has joined #openstack-helm | 23:37 | |
*** yamamoto has quit IRC | 23:38 | |
*** sawblade_ has quit IRC | 23:39 | |
*** sawblade6 has quit IRC | 23:49 | |
*** sawblade6 has joined #openstack-helm | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!