| *** julim_ has joined #openstack-helm | 00:53 | |
| *** julim has quit IRC | 00:56 | |
| *** iceyao has joined #openstack-helm | 01:00 | |
| *** iceyao has quit IRC | 01:07 | |
| *** iceyao has joined #openstack-helm | 01:17 | |
| *** randomha1k has joined #openstack-helm | 01:18 | |
| *** randomha1k has quit IRC | 01:22 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Ironic Chart https://review.openstack.org/490271 | 01:35 |
|---|---|---|
| *** randomha1k has joined #openstack-helm | 01:50 | |
| *** randomha1k has quit IRC | 01:55 | |
| *** randomha1k has joined #openstack-helm | 02:37 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Ironic Chart https://review.openstack.org/490271 | 02:39 |
| *** randomha1k has quit IRC | 02:42 | |
| *** gouthamr has quit IRC | 03:03 | |
| *** iceyao has quit IRC | 03:49 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Ironic Chart https://review.openstack.org/490271 | 03:57 |
| osh-chatbot3 | <portdirect> @dwannamaker you may be able to get things working with 1.5 with a few changes (like the storageclass) but I'd really recomend using 1.6.4+ if possible | 04:04 |
| osh-chatbot3 | <portdirect> we have nt tested with <1.6 for a long time | 04:04 |
| osh-chatbot3 | <portdirect> *not | 04:04 |
| *** iceyao has joined #openstack-helm | 04:14 | |
| *** iceyao has quit IRC | 04:19 | |
| *** iceyao has joined #openstack-helm | 04:32 | |
| *** iceyao has quit IRC | 04:37 | |
| *** iceyao has joined #openstack-helm | 04:48 | |
| *** randomha1k has joined #openstack-helm | 04:53 | |
| *** spsurya has joined #openstack-helm | 04:55 | |
| *** randomha1k has quit IRC | 04:58 | |
| *** dims has quit IRC | 05:52 | |
| *** dims has joined #openstack-helm | 05:54 | |
| *** larainema has quit IRC | 05:56 | |
| *** randomha1k has joined #openstack-helm | 06:54 | |
| *** randomha1k has quit IRC | 06:59 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: WIP: Neutron Linuxbridge gate test https://review.openstack.org/489594 | 07:00 |
| *** iceyao has quit IRC | 07:13 | |
| *** iceyao has joined #openstack-helm | 07:21 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: WIP: Neutron Linuxbridge gate test https://review.openstack.org/489594 | 07:44 |
| *** larainema has joined #openstack-helm | 07:54 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** openstackgerrit has joined #openstack-helm | 08:52 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: WIP: Neutron Linuxbridge gate test https://review.openstack.org/489594 | 08:52 |
| *** dims has quit IRC | 09:04 | |
| *** dims has joined #openstack-helm | 09:08 | |
| openstackgerrit | Siri Kim proposed openstack/openstack-helm master: Neutron: Enable decomposition of chart into descrete manifests. https://review.openstack.org/481225 | 09:26 |
| *** julim_ has quit IRC | 09:45 | |
| openstackgerrit | Siri Kim proposed openstack/openstack-helm master: Neutron: Enable decomposition of chart into descrete manifests. https://review.openstack.org/481225 | 10:06 |
| *** julim has joined #openstack-helm | 11:03 | |
| *** iceyao has quit IRC | 11:09 | |
| *** iceyao has joined #openstack-helm | 11:17 | |
| *** iceyao has quit IRC | 11:41 | |
| *** iceyao has joined #openstack-helm | 11:44 | |
| *** iceyao has quit IRC | 11:48 | |
| *** iceyao has joined #openstack-helm | 12:13 | |
| *** iceyao has quit IRC | 12:17 | |
| *** schwicht has joined #openstack-helm | 12:21 | |
| *** schwicht has quit IRC | 12:29 | |
| *** julim has quit IRC | 12:34 | |
| *** alraddarla has joined #openstack-helm | 12:34 | |
| *** schwicht has joined #openstack-helm | 12:34 | |
| *** iceyao has joined #openstack-helm | 12:42 | |
| *** schwicht has quit IRC | 12:53 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: WIP: Neutron Linuxbridge gate test https://review.openstack.org/489594 | 12:53 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Ironic Chart https://review.openstack.org/490271 | 13:09 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Ironic Chart https://review.openstack.org/490271 | 13:10 |
| *** schwicht has joined #openstack-helm | 13:11 | |
| *** julim_ has joined #openstack-helm | 13:13 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Ironic Chart https://review.openstack.org/490271 | 13:17 |
| *** randomha1k has joined #openstack-helm | 13:27 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: WIP: Neutron Linuxbridge gate test https://review.openstack.org/489594 | 13:42 |
| *** felipemonteiro has joined #openstack-helm | 13:48 | |
| *** felipemonteiro_ has joined #openstack-helm | 13:49 | |
| *** jaypipes has joined #openstack-helm | 13:49 | |
| *** spsurya has quit IRC | 13:52 | |
| *** felipemonteiro has quit IRC | 13:53 | |
| *** irenab has left #openstack-helm | 14:14 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: WIP: Neutron Linuxbridge gate test https://review.openstack.org/489594 | 14:15 |
| *** StaceyF has joined #openstack-helm | 14:18 | |
| *** felipemonteiro_ has quit IRC | 14:22 | |
| *** marst has joined #openstack-helm | 14:28 | |
| *** StaceyF has quit IRC | 14:42 | |
| *** felipemonteiro has joined #openstack-helm | 14:52 | |
| *** felipemonteiro_ has joined #openstack-helm | 14:53 | |
| *** felipemonteiro has quit IRC | 14:56 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: WIP: Neutron Linuxbridge gate test https://review.openstack.org/489594 | 14:58 |
| *** MarkBaker has joined #openstack-helm | 15:10 | |
| *** danpawlik has joined #openstack-helm | 15:12 | |
| *** iceyao has quit IRC | 15:19 | |
| *** alraddarla has quit IRC | 15:37 | |
| *** iceyao has joined #openstack-helm | 15:38 | |
| *** alraddarla has joined #openstack-helm | 15:38 | |
| *** alraddarla has quit IRC | 15:38 | |
| *** alraddarla has joined #openstack-helm | 15:39 | |
| *** iceyao has quit IRC | 15:42 | |
| *** randomha1k has quit IRC | 15:44 | |
| *** randomha1k has joined #openstack-helm | 15:51 | |
| *** MarkBaker has quit IRC | 16:01 | |
| openstackgerrit | Larry Rensing proposed openstack/openstack-helm-addons master: WIP - MaaS chart https://review.openstack.org/472381 | 16:38 |
| openstackgerrit | Larry Rensing proposed openstack/openstack-helm-addons master: WIP - MaaS chart https://review.openstack.org/472381 | 16:39 |
| *** spsurya has joined #openstack-helm | 16:48 | |
| *** renmak_ has joined #openstack-helm | 16:59 | |
| *** renmak__ has joined #openstack-helm | 16:59 | |
| *** iceyao has joined #openstack-helm | 17:10 | |
| *** iceyao has quit IRC | 17:14 | |
| *** aric49 has joined #openstack-helm | 17:16 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: DNM - Test mariadb resiliency https://review.openstack.org/490601 | 17:18 |
| *** randomha1k has quit IRC | 17:26 | |
| bryan_att | hi anyone - any hints as to how to debug the cause for ErrImagePull for ceph containers? I'm seeing these for some reason on ceph-rgw-keyring-generator, ceph-storage-keys-generator containers. Up that point all other containers are pulled fine so I assume it's not a networking issue. | 17:27 |
| *** randomha1k has joined #openstack-helm | 17:29 | |
| anticw | bryan_att: docker logs on the node in question | 17:30 |
| anticw | also try `docker pull` on the given nodes and watch what it does | 17:30 |
| anticw | i'm assuming it's safe to run that concurrently (probably a bad assumption) | 17:31 |
| bryan_att | do I get those via kubectl logs or something else direct on the container? And how to find which container? (the error message in the output does not clarify that) ? | 17:31 |
| v1k0d3n | Others have been complaining about ceph as well lately bryan_att | 17:32 |
| bryan_att | there was one ceph container in "docker ps -a" but it had exited when the gate script exited. | 17:33 |
| bryan_att | here are the messages that the gate script outputs when it quits https://www.irccloud.com/pastebin/9aTcWGf2/ | 17:34 |
| bryan_att | anticw: here's an example of the docker pull command (assuming this is the correct container image name) https://www.irccloud.com/pastebin/VBvAsACH/ | 17:37 |
| bryan_att | v1k0d3n: where have they been complaining (here?) so I can check the threads | 17:39 |
| v1k0d3n | people not on this channel, or blocked by proxy bryan_att | 17:41 |
| bryan_att | v1k0d3n: well now *that's* fairly cryptic... btw I am not behind a proxy if that's what you meant, so I shouldn't have trouble getting connected to them (unless they are, and then it'[s doesn't matter much I guess). next step suggestions? | 17:43 |
| bryan_att | this project is like an event horizon... the closer I seem to get, the slower I progress | 17:45 |
| anticw | bryan_att: from your frame of reference you won't experience that | 17:51 |
| anticw | bryan_att: so, the image specified isn't there ... which chart is this? | 17:52 |
| bryan_att | anticw: ceph afaict | 17:53 |
| srwilkers | thats the ceph keyring job and the client key job. image is docker.io/port/ceph-config-helper:v1.6.7 | 17:53 |
| srwilkers | not docker.io/library | 17:53 |
| *** gouthamr has joined #openstack-helm | 17:53 | |
| osh-chatbot3 | <v1k0d3n> Why not the att-comdev repo, not that it matters I guess? | 17:54 |
| bryan_att | srwilkers: how do i ref that particular repo ? (I'll check the docker docs also) | 17:54 |
| anticw | bryan_att: osh ceph from master? | 17:56 |
| bryan_att | anticw: yes, master | 17:56 |
| bryan_att | meaning osh master (the gate job) | 17:57 |
| anticw | i just updated to that, i see: | 17:58 |
| anticw | 32 ceph_config_helper: docker.io/port/ceph-config-helper:v1.6.7 | 17:58 |
| anticw | and | 17:59 |
| anticw | - name: ceph-storage-keys-generator | 17:59 |
| anticw | image: {{ .Values.images.ceph_config_helper }} | 17:59 |
| *** lrensing has joined #openstack-helm | 17:59 | |
| anticw | bryan_att: want to check you don't have local deviations from master? | 18:00 |
| bryan_att | anticw: no deviations, I am doing this at least once an hour pulling a fresh clone each time | 18:01 |
| bryan_att | anticw: with the minor exception of relaxing some timeouts in the gate script (I find that things don't go as fast as expected often) | 18:01 |
| anticw | is gate using and override? (well, i guess it must be ...) | 18:02 |
| anticw | so in my repo here i don't see anything in tools/ that would override the image | 18:02 |
| lrensing | bryan_att: can you do a docker pull docker.io/port/ceph-config-helper:v1.6.7 ? | 18:02 |
| anticw | i just tested that here and it works | 18:03 |
| anticw | bryan_att: does it work for you? | 18:03 |
| lrensing | also try a `kubectl describe` on the pods that are experiencing the pull error - sometimes those can give a bit of insight | 18:04 |
| v1k0d3n | bryan_att: you're on bare metal, right? | 18:04 |
| bryan_att | v1k0d3n: yes | 18:04 |
| bryan_att | trying the other hints too - thanks | 18:04 |
| bryan_att | i have to redeploy first to make sure I have a clean env - do this every time as I haven't seen any clean scripts yet in osh | 18:07 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: DNM - Test mariadb resiliency https://review.openstack.org/490601 | 18:10 |
| bryan_att | v1k0d3n: to be exact, currently testing with 3 Dell R720's 256G ram 2.3G disk machines | 18:10 |
| bryan_att | (2.3T !) | 18:11 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Armada OpenStack deployment yaml https://review.openstack.org/481234 | 18:23 |
| *** kaspars has joined #openstack-helm | 18:23 | |
| *** kaspars has quit IRC | 18:24 | |
| *** renmak_ has quit IRC | 18:25 | |
| *** renmak__ has quit IRC | 18:25 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: DNM - Test mariadb resiliency https://review.openstack.org/490601 | 18:42 |
| bryan_att | lrensing: result of the command (I had to wait for the new deploy to fail) https://www.irccloud.com/pastebin/98ySXLIr/ | 19:01 |
| *** schwicht has quit IRC | 19:01 | |
| lrensing | bryan_att: can you run that command on every node you're deploying ceph to? | 19:01 |
| bryan_att | lrensing: ok | 19:01 |
| bryan_att | lrensing: looks like there is a DNS resolver issue somehow different from the primary and secondary nodes. something (?) set the dns server to 10.96.0.10 and that address is pingable from the primary, but not the secondary nodes. | 19:08 |
| bryan_att | lrensing: looks like that dns nameserver is set in various places in the gate script and yaml files etc. any idea why requests to it would not be routable? | 19:14 |
| osh-chatbot3 | <raymaika> @bryan_att did you check if it resolves names from other nodes? 10.96.0.10 doesn't necessarily respond to pings on all nodes in the cluster (depending on the CNI settings), but it should always be able to resolve names | 19:17 |
| bryan_att | lrensing: this is from a secondary node https://www.irccloud.com/pastebin/dzgrZJNk/ | 19:19 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: DNM/WIP -- Add proxysql to mariadb https://review.openstack.org/487588 | 19:22 |
| bryan_att | lrensing: also BTW there is no route that would send DNS requests to the master node afaict https://www.irccloud.com/pastebin/AuFk3Aig/ | 19:25 |
| *** felipemonteiro_ has quit IRC | 19:27 | |
| *** felipemonteiro_ has joined #openstack-helm | 19:27 | |
| *** schwicht has joined #openstack-helm | 19:29 | |
| *** felipemonteiro_ has quit IRC | 19:30 | |
| *** felipemonteiro_ has joined #openstack-helm | 19:30 | |
| *** julim_ has quit IRC | 19:31 | |
| *** julim has joined #openstack-helm | 19:34 | |
| *** julim has quit IRC | 19:34 | |
| *** julim has joined #openstack-helm | 19:34 | |
| *** alraddarla has quit IRC | 19:36 | |
| *** felipemonteiro_ has quit IRC | 19:47 | |
| *** randomha1k has quit IRC | 19:58 | |
| bryan_att | lrensing: thinking the dns server is not the issue here - see this log from the same process on another set of servers. same dns setup, but in this case the gate job gets farther (to the mariadb issue which is my current hard blicking issue) https://www.irccloud.com/pastebin/WM6Av2a5/ | 20:01 |
| lrensing | reading scrollback bryan_att | 20:02 |
| bryan_att | I guess the container pull action is being initiated from within one of the other containers and not the baremetal host? but I would still like to know why the resolv.conf is being updated by the script, when the setting has no useful purpose on the host. | 20:02 |
| lrensing | are any of these nodes behind a proxy? | 20:03 |
| bryan_att | no | 20:06 |
| bryan_att | all I can figure is that there are some env assumptions baked into the gate jobs as they run under OS CI. And I am running into these undocumented assumptions. If that is the case and I figure this out I will submit issues for that. I haven't heard anything yet on my earlier request for input from people doing multi-node deploys ... what is the env, what exactly is the process (if not the same as in the gate | 20:06 |
| bryan_att | readme...) | 20:06 |
| portdirect | bryan_att: I am using the gate env on ovh, and my own hardware, without issue - and have done done so on aws (but that requires more setup on the networking side to allow calico to work) | 20:08 |
| bryan_att | portdirect: can you pm me your complete process? I will do the same. | 20:10 |
| portdirect | I just do exactly as described in the readme (though to make sure theres no egg on my face I'll just double check) | 20:11 |
| anticw | portdirect: what's required on AWS to make calico work? i didn't try it but it's seems if all VMs are in the same VPC it should more-or-less JustWork™ | 20:26 |
| anticw | portdirect: same VPC & subnet i meant | 20:35 |
| osh-chatbot3 | <portdirect> @anticw just the stuff from in here: http://docs.projectcalico.org/master/reference/public-cloud/aws | 20:36 |
| osh-chatbot3 | <portdirect> though it was ages ago - I honest cannot rememebr the precise foo i needed to do - as since then its just worked :slightly_smiling_face: | 20:36 |
| anticw | portdirect: thanks, that makes sense | 20:38 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Armada OpenStack deployment yaml https://review.openstack.org/481234 | 20:44 |
| *** spsurya has quit IRC | 20:44 | |
| *** julim has quit IRC | 20:44 | |
| *** julim has joined #openstack-helm | 20:45 | |
| *** felipemonteiro has joined #openstack-helm | 20:47 | |
| *** julim has quit IRC | 20:50 | |
| *** felipemonteiro_ has joined #openstack-helm | 20:52 | |
| *** felipemonteiro has quit IRC | 20:56 | |
| *** iceyao has joined #openstack-helm | 21:13 | |
| *** iceyao has quit IRC | 21:18 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Armada OpenStack deployment yaml https://review.openstack.org/481234 | 21:33 |
| *** aric49 has quit IRC | 21:40 | |
| openstackgerrit | Larry Rensing proposed openstack/openstack-helm-addons master: WIP - MaaS chart https://review.openstack.org/472381 | 21:56 |
| *** schwicht has quit IRC | 22:01 | |
| *** renmak__ has joined #openstack-helm | 22:04 | |
| *** renmak_ has joined #openstack-helm | 22:04 | |
| *** lrensing has quit IRC | 22:04 | |
| *** renmak__ has quit IRC | 22:22 | |
| *** renmak_ has quit IRC | 22:22 | |
| *** renmak_ has joined #openstack-helm | 22:22 | |
| *** renmak__ has joined #openstack-helm | 22:22 | |
| *** renmak_ has quit IRC | 22:27 | |
| *** renmak_ has joined #openstack-helm | 22:27 | |
| *** renmak_ has quit IRC | 22:30 | |
| *** renmak__ has quit IRC | 22:30 | |
| *** renmak_ has joined #openstack-helm | 22:31 | |
| *** renmak__ has joined #openstack-helm | 22:31 | |
| *** schwicht has joined #openstack-helm | 22:32 | |
| *** felipemonteiro_ has quit IRC | 22:41 | |
| *** schwicht_at_work has joined #openstack-helm | 22:53 | |
| *** schwicht has quit IRC | 22:56 | |
| *** schwicht_at_work has quit IRC | 22:58 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Armada OpenStack deployment yaml https://review.openstack.org/481234 | 23:45 |
| *** marst has quit IRC | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!