Sunday, 2019-12-01

*** logan- has quit IRC06:29
*** logan- has joined #openstack-containers06:32
*** sapd1 has joined #openstack-containers08:39
*** sapd1 has quit IRC08:57
*** ramishra has joined #openstack-containers09:04
*** ramishra has quit IRC09:04
*** ramishra has joined #openstack-containers09:04
*** sapd1 has joined #openstack-containers09:09
*** mgoddard has quit IRC10:13
*** tbarron has joined #openstack-containers10:34
*** ramishra has quit IRC11:19
*** sapd1_x has joined #openstack-containers12:12
*** sapd1 has quit IRC12:15
*** sapd1_x has quit IRC14:28
*** mgoddard has joined #openstack-containers14:50
*** ricolin has joined #openstack-containers15:05
*** ricolin has quit IRC15:40
*** mgoddard has quit IRC15:50
*** ricolin has joined #openstack-containers16:04
*** elenalindq has joined #openstack-containers17:01
*** vesper11 has quit IRC17:11
*** kgz has quit IRC17:11
*** mnasiadka has quit IRC17:11
*** flwang has quit IRC17:11
*** vesper11 has joined #openstack-containers17:12
*** kgz has joined #openstack-containers17:12
*** mnasiadka has joined #openstack-containers17:12
*** flwang has joined #openstack-containers17:12
*** kgz has quit IRC17:12
*** ricolin has quit IRC17:47
*** dave-mccowan has joined #openstack-containers18:11
*** kgz has joined #openstack-containers18:18
*** dave-mccowan has quit IRC19:56
*** dave-mccowan has joined #openstack-containers20:30
*** dave-mccowan has quit IRC20:53
cosmicsoundHi, does someone happend to know what could cause such errors on cloud-init21:01
cosmicsound[  316.423247] cloud-init[1186]: 2019-12-01 20:47:56,988 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3.5/site-packages/cloudinit/config/cc_scripts_user.py'>) failed21:01
cosmicsoundutil.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-013 [7]21:01
cosmicsoundcc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)21:01
flwangjust manually run the script part-013 and see what's the root cause21:07
flwangseems you're using rocky?21:07
flwangcosmicsound: ^21:07
cosmicsoundStein21:11
cosmicsoundflwang using Stein and fedora 29 latest21:11
cosmicsoundWith Magnum latest21:11
cosmicsoundI also got a weird timeout on port 800421:11
cosmicsoundThat would be HEAT21:11
cosmicsoundIs there a way to see some logs in real time of the cluster deployment in Magnum via openstack cli?21:12
flwangyou can't see logs from openstack cli21:12
cosmicsoundrigh21:12
flwangyou have to ssh into the master node to understand why21:12
flwangas for your above issue, as i said,  login and re-run the script manually, then you will be able to see the issue21:13
cosmicsoundhow do i rerun manually those script parts21:13
cosmicsoundfrom the master locally21:13
cosmicsoundwill try this see what it makes21:14
flwang/var/lib/cloud/instance/scripts/part-01321:15
cosmicsoundNow I see this21:16
cosmicsoundReturnTuple(return_code=1, stdout=b'', stderr=b'time="2019-12-01T20:45:34Z" level=warning msg="\'--tls-verify\' is deprecated, please set this on the specific subcommand" \nDEPRECATED: skopeo layers is deprecated in favor of skopeo copy\ntime="2019-12-01T20:45:39Z" level=fatal msg="write layers-302929704/dir-put-blob047846042: no space left on21:16
cosmicsounddevice" \n')21:16
cosmicsound40GB Space21:18
cosmicsoundMakes no sense21:18
cosmicsoundDocker Image is 10 GB21:18
flwang:)21:20
flwangcosmicsound: are you trying to build a magnum env on prod or just testing?21:20
cosmicsoundThis is testing or pre prod testing21:23
cosmicsoundwe want to go with magnum21:23
cosmicsoundand we test it ofcourse21:23
flwangcool, may I know which company are you working for?21:28
cosmicsoundUHL-Services21:30
cosmicsoundWere new in the game21:30
cosmicsound:D21:30
cosmicsoundAnd started to like it more and more21:31
cosmicsoundKeeps me awake21:31
flwangif you're building your magnum env, then it's better to use latest version, say stable/train here21:49
cosmicsoundWe had no chance to use that one21:49
flwangwhich is the latest stable branch21:50
flwangwhy?21:50
cosmicsoundIt failed in our deployment21:50
cosmicsoundWith kolla-ansible21:50
cosmicsoundSeveral times21:50
flwangok21:50
cosmicsoundSo we stayed on Stein21:50
flwangi believe it's a kolla-ansible issue21:50
cosmicsoundOnce we stabilite it here well try upgrade21:50
cosmicsoundsurely can be21:50
flwangok, fair enough21:50
cosmicsoundbtw thanks for the info21:50
cosmicsoundI managed to login to the master21:50
cosmicsoundyet since im using a ssh key now i need to dig the default passwords :D21:51
flwangthere is no default passwords :)21:51
flwangat least afaik21:52
cosmicsoundIt asks me21:52
cosmicsoundWhen I do on this fedora 29 atomic21:52
cosmicsoundpasswd asks me old pass21:52
cosmicsoundbit weird ofc21:52
cosmicsoundyet...21:52
cosmicsoundWe had a working deployment, just for some unsaved settings now all our cluster fail21:55
cosmicsoundfor various reasons and last one maybe will need to check more if is not a bug or our mistake21:55
cosmicsoundCREATE_FAILED  CREATE aborted (Task create from ResourceGroup "kube_minions" Stack "kubernetes-4hp2ilbqtpil" [ebbef67d-f246-4479-af86-3fd401a98eb8] Timed out)21:55
cosmicsound2019-12-01 21:40:52Z [kubernetes-4hp2ilbqtpil.kube_cluster_deploy]: CREATE_FAILED  CREATE aborted (Task create from SoftwareDeployment "kube_cluster_deploy" Stack "kubernetes-4hp2ilbqtpil" [ebbef67d-f246-4479-af86-3fd401a98eb8] Timed out)21:55
cosmicsoundthese 2 time out on and on21:55
*** elenalindq has quit IRC22:02
flwangok, for time out issue22:05
flwangit's similar debug process22:06
flwangfirstly, check if the node has been created22:06
flwangthen login the node to understand which script failed22:06
flwangre run the script and check the root cause22:06
*** jakeyip has quit IRC22:12
*** dave-mccowan has joined #openstack-containers22:32
cosmicsoundWell it is weird22:35
cosmicsoundBecause both master and nodes are created22:35
cosmicsoundJust they somehow do not communicate back the message that they are done created22:36
cosmicsoundI managed to ssh both machines perfectly22:36
cosmicsoundfound now a new model and following it, seems a bit newer documentation22:36
flwangjust check the log of cloud-init and heat-container-agent22:42
flwangjournalctl -u heat-container-agent22:42
flwangvi /var/log/cloud-init-output.log22:42
cosmicsoundwhat would one choose for openstack23:03
cosmicsoundubuntu or centos23:03
cosmicsoundto be honest i like the way centos moves23:03
*** dave-mccowan has quit IRC23:32

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!