Friday, 2019-06-21

*** chem has quit IRC00:00
*** chem has joined #oooq00:22
*** apetrich has quit IRC01:57
*** Goneri has quit IRC02:09
*** skramaja has joined #oooq02:37
*** ykarel has joined #oooq03:20
*** udesale has joined #oooq04:05
*** ykarel has quit IRC04:34
*** ykarel has joined #oooq04:49
*** ratailor has joined #oooq05:02
*** kopecmartin|off is now known as kopecmartin05:12
*** jbadiapa has quit IRC05:13
*** ykarel_ has joined #oooq05:32
*** ykarel has quit IRC05:34
*** marios has joined #oooq06:00
*** ykarel_ is now known as ykarel|meeting06:01
*** jfrancoa has joined #oooq06:08
*** jtomasek has joined #oooq06:16
*** jtomasek has quit IRC06:17
*** jtomasek has joined #oooq06:17
*** udesale has quit IRC06:36
*** udesale has joined #oooq06:37
*** saneax has joined #oooq06:46
*** ykarel_ has joined #oooq06:59
*** jbadiapa has joined #oooq07:00
*** ykarel|meeting has quit IRC07:01
*** saneax has quit IRC07:08
*** amoralej|off is now known as amoralej07:13
*** ykarel_ is now known as ykarel07:13
*** tesseract has joined #oooq07:19
*** tosky has joined #oooq07:21
*** ccamacho has joined #oooq07:36
*** jpena|off is now known as jpena07:37
*** saneax has joined #oooq07:51
*** ykarel has quit IRC07:53
*** ykarel has joined #oooq07:55
*** ykarel is now known as ykarel|lunch07:56
*** rascasoft has joined #oooq08:05
*** apetrich has joined #oooq08:07
*** bogdando has joined #oooq08:10
*** udesale has quit IRC08:11
*** udesale has joined #oooq08:12
*** rascasoft has quit IRC08:15
*** rascasoft has joined #oooq08:15
*** gkadam has joined #oooq08:19
*** derekh has joined #oooq08:38
*** ykarel|lunch is now known as ykarel08:56
*** rascasoft has quit IRC08:56
*** jfrancoa has quit IRC08:57
*** tosky__ has joined #oooq09:06
*** ykarel has quit IRC09:07
*** tosky has quit IRC09:08
*** gkadam has quit IRC09:11
*** holser_ has joined #oooq09:28
*** amoralej is now known as amoralej|brb09:30
zbr|rucki am proposing a new behavior change on ansible, https://github.com/ansible/proposals/issues/167 -- if you like it please vote on it.09:35
*** tosky__ is now known as tosky09:43
*** ykarel_ has joined #oooq09:57
*** ykarel_ is now known as ykarel09:58
*** amoralej|brb is now known as amoralej10:07
*** holser_ has quit IRC10:15
*** udesale has quit IRC10:31
pandazbr|ruck: where are you hitting this ?10:55
zbr|ruckpanda: when trying to run some playbooks myself. it does not affect CI usage where we control the CWD, is purely a developer experience.11:03
*** jbadiapa has quit IRC11:09
pandaDUX11:12
* marios shoots11:13
mariospanda: duck al'orange?11:14
pandamarios: hold the rifles. DUX is for developer experience. Or maybe in latin, it stands for "leader"11:15
mariosbut they said there would be duck!11:18
pandamarios: it's the gooses spreading lies again.11:19
pandaaaaaaaa11:20
pandageese11:20
mariosgeeses?11:21
pandano, geese11:21
mariospanda: yeah i was joking !11:21
marios:D11:21
pandao jeez, if you prefer11:21
*** Goneri has joined #oooq11:21
mariospanda: anyway enough with the distraction where's me peking duck?11:22
mariossudo make me peking duck11:22
pandamake: *** No rule to make target 'a pecking duck'.  Stop.11:23
marios:/ no duck for me11:24
*** amoralej is now known as amoralej|lunch11:25
*** jpena is now known as jpena|lunch11:26
*** holser_ has joined #oooq11:47
*** holser_ has quit IRC11:48
*** holser_ has joined #oooq11:48
*** ykarel is now known as ykarel|away11:51
*** holser__ has joined #oooq11:53
*** ykarel_ has joined #oooq11:54
*** holser_ has quit IRC11:55
*** ykarel|away has quit IRC11:56
*** udesale has joined #oooq12:10
*** ykarel_ has quit IRC12:11
*** arxcruz changes topic to "Docs: https://docs.openstack.org/tripleo-quickstart/latest/"12:19
*** ratailor has quit IRC12:25
*** rlandy has joined #oooq12:36
*** vkmc has quit IRC12:40
*** jbadiapa has joined #oooq12:48
pandamarios: alias ducker=docker. Happy ducking.12:50
pandaalias gooseman=podman12:51
pandaaslo a good idea12:51
*** jpena|lunch is now known as jpena12:59
*** rfolco has joined #oooq13:02
arxcruzweshay kopecmartin chandankumar no meeting today?13:02
kopecmartinarxcruz, probably no, last time when chandankumar was not here we didn't have the meeting too13:03
weshayI'm joining13:03
weshaylet's spend 5 min13:03
*** ykarel_ has joined #oooq13:05
*** rascasoft has joined #oooq13:08
*** ykarel_ is now known as ykarel13:09
*** amoralej|lunch is now known as amoralej13:09
weshaymarios have a min to discuss image building13:41
mariosweshay: o/ here or rhos-delivery (reading now)13:42
*** skramaja has quit IRC13:45
derekhpanda: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario012-standalone13:58
*** Vorrtex has joined #oooq14:03
pandaderekh: \o/14:05
pandaderekh: just retrying that solution you mentioned ?14:06
derekhpanda: no, I think I was doing something wrong that day, there was a few other separate problems, anyways I'll go talk to jlibosva so see if he knows if anything else should be changed and clean up the patches14:20
*** holser__ has quit IRC14:21
*** holser_ has joined #oooq14:21
rfolcomarios, quick question14:23
rfolcozuul.d/nodesets.yaml:# TODO(marios): label: upstream-rhel-8 when https://review.openstack.org/64373114:23
rfolcotrying to fix this.. https://sf.hosted.upshift.rdu2.redhat.com/logs/39/174039/5/check/tripleo-ci-rhel-8-rhui-test/e40495f/job-output.txt.gz#_2019-06-21_14_01_24_41642514:24
rfolcoperhaps you know how14:25
*** apetrich has quit IRC14:26
mariosrfolco: o/14:26
*** holser_ has quit IRC14:26
*** ykarel has quit IRC14:26
mariosrfolco: we should be good to go there14:26
mariosrfolco: let me find you som context (it is in sprint 12 i think now)14:26
*** ykarel has joined #oooq14:26
mariosrfolco: for the python bindings are you using the right zuul user and also set py3 for the job vars14:27
mariosrfolco: like https://code.engineering.redhat.com/gerrit/#/c/173350/8/zuul.d/tripleo-build-containers.yaml14:27
mariosrfolco: try ansible_python_interpreter14:27
mariosrfolco: ignore the zuul user bit (that was also giving the same message but i don't think it applies in this case)14:27
mariosrfolco: re the 17:24 < rfolco> zuul.d/nodesets.yaml:# TODO(marios): label: upstream-rhel-8 when https://review.openstack.org/64373114:28
mariosrfolco: see here https://tree.taiga.io/project/tripleo-ci-board/task/1097?kanban-status=1447275  https://code.engineering.redhat.com/gerrit/#/c/172066/14:28
rfolcomarios, that error happens in base pre14:28
rfolcothe dib image should have libselinux-python... isn't it ?14:30
mariosrfolco: so we aren't using the dib yet. but the fake one we are using does have it14:30
mariosrfolco: thats the 'switch' bit 17:29 < marios> rfolco: see here https://tree.taiga.io/project/tripleo-ci-board/task/1097?kanban-status=1447275  https://code.engineering.redhat.com/gerrit/#/c/172066/14:30
rfolcooh dib!!!14:31
mariosrfolco: try the ansible-python-interpreter thing though?14:31
rfolcobasically your are using dib-rhel-8 not cloud-rhel-814:31
rfolcothats it14:32
rfolcomarios, thanks, sorry I missed that swicth14:32
mariosrfolco: other way around. all our rhel8 jobs are using cloud-rhel8 not dib-rhel-814:32
mariosrfolco: until we switch14:32
rfolcooops14:32
mariosrfolco: we should be ready to (we were blocked, info on the task) but the sf guys said check it now so we need to do that14:32
mariosrfolco: but its in sprint 1214:32
rfolcomarios, show me a log and patch of a rhel8 job please14:33
rfolcowhy your base pre does not fail that way I want to see that14:33
mariosrfolco: https://code.engineering.redhat.com/gerrit/#/c/172857/14:33
rfolcook looking14:33
rfolcothx14:33
mariosrfolco: https://code.engineering.redhat.com/gerrit/#/c/173350/14:34
mariosrfolco: TRY THE ANSIBLE INTERPRETER14:34
mariosrfolco: i had the same error14:34
rfolcomarios, ok the caps helps us to take serious a particular comment14:35
rfolcothank you :)14:35
mariosrfolco: trying to find evidence for that :D14:35
mariosrfolco: v1 https://code.engineering.redhat.com/gerrit/#/c/173350/1/zuul.d/tripleo-build-containers.yaml  https://sf.hosted.upshift.rdu2.redhat.com/logs/50/173350/1/check/periodic-tripleo-build-containers-rhel-8-train/6b34ee9/job-output.txt.gz#_2019-06-13_10_57_40_06391614:37
mariosrfolco: v2 added the ansible interpreter https://code.engineering.redhat.com/gerrit/#/c/173350/2/zuul.d/tripleo-build-containers.yaml https://sf.hosted.upshift.rdu2.redhat.com/logs/50/173350/2/check/periodic-tripleo-build-containers-rhel-8-train/c1e2327/job-output.txt.gz14:37
* marios collapse14:37
*** holser_ has joined #oooq14:42
*** kopecmartin is now known as kopecmartin|off14:47
*** holser_ has quit IRC15:04
*** ykarel is now known as ykarel|away15:05
rlandypanda: have a minute to answer a few promoter questions?15:10
*** gkadam has joined #oooq15:15
*** tosky has quit IRC15:18
rfolcomarios, it works, thanks again15:20
mariosrfolco: cool15:20
rfolcomarios, not the vars, the host-vars15:21
rfolco:)15:21
pandarlandy: sure15:24
mariosrfolco: yeah15:25
rlandypanda: https://bluejeans.com/u/rlandy - I'll be brief15:25
mariosrfolco: sorry if i told you vars before. its as host-vars on my reviews too15:25
mariosrfolco: well i have both actuallly just noticed15:25
mariosrfolco: on the containers one i mean i should remove the extra one15:26
rfolcomarios, right, I tested both to confirm :)15:26
mariosrfolco: but we came across this for the osp15 rhel8 job last sprint15:26
rfolcowithout host-vars does not work15:26
mariosrfolco: which is why i knew about it15:26
mariosrfolco: hah we have it both ways there too https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-jobs.git;a=blob;f=zuul.d/standalone-jobs.yaml#l2615:28
*** saneax has quit IRC15:33
*** bogdando has quit IRC15:33
*** saneax has joined #oooq15:33
mariosbrb15:39
*** marios has quit IRC15:39
*** marios has joined #oooq15:40
*** saneax has quit IRC15:45
*** vinaykns has joined #oooq15:46
* marios almost out15:50
mariosrfolco: thanks for checking about that host_vars i need to change it too in my reviews and also in the merged on15:50
zbr|ruckweshay: i see this quite often http://logs.openstack.org/77/662577/5/check/tripleo-ci-centos-7-containers-multinode/311fcad/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-06-21_01_48_3715:51
weshayzbr|ruck can you create a logstash query15:52
weshayseems like a very generic error, will be hard to create one15:53
*** holser_ has joined #oooq15:53
*** holser_ has quit IRC15:53
weshayzbr|ruck maybe just "SSH Error: data could not be sent to remote host "localhost". Make sure this host can be reached over ssh"15:53
weshayscope it to tripleo jobs15:53
weshayand see how often it fails15:53
*** holser_ has joined #oooq15:53
weshaymarios not sure what was fixed, or where it was fixed..  I guess dlrn-deps15:54
weshayhttps://trello.com/c/27kSfSs4/1014-cixlp1833209tripleociproa-rdotrunkrhel8-dependency-errors-for-overcloud-image-build15:54
mariosweshay: nice ok will continue monday. both that and rhui as discussed15:55
weshay+115:55
marioshave a good w/e folks bai15:58
*** marios has quit IRC15:58
*** jpena is now known as jpena|off16:01
*** panda is now known as panda-pto16:08
rlandypanda-pto++ thanks16:08
panda-pto:) happy to help.16:08
*** amoralej is now known as amoralej|off16:32
*** udesale has quit IRC16:39
*** udesale has joined #oooq16:39
weshayzbr|ruck may have an issue promoting f2816:44
weshaychanged: [localhost] => (item=ovn-nb-db-server)16:44
weshayfailed: [localhost] (item=ovn-northd) => {"changed": false, "item": "ovn-northd", "msg": "Error pulling trunk.registry.rdoproject.org/tripleomaster/fedora-binary-ovn-northd - code: None message: Get https://trunk.registry.rdoproject.org/v2/tripleomaster/fedora-binary-ovn-northd/manifests/d805e6b1285bf6e7d99163cd90fe3d04751eebb9_ba4d6a3d: Get https:16:44
weshay//trunk.registry.rdoproject.org/openshift/token?account=unused&scope=repository%3Atripleomaster%2Ffedora-binary-ovn-northd%3Apull: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"}16:44
weshayfailed: [localhost] (item=ovn-sb-db-server) => {"changed": false, "item": "ovn-sb-db-server", "msg": "Error pulling trunk.registry.rdoproject.org/tripleomaster/fedora-binary-ovn-sb-db-server - code: None message: Get https://trunk.registry.rdoproject.org/v2/tripleomaster/fedora-binary-ovn-sb-db-server/manifests/d805e6b1285bf6e7d99163cd90fe3d04751ee16:44
weshaybb9_ba4d6a3d: Get https://trunk.registry.rdoproject.org/openshift/token?account=unused&scope=repository%3Atripleomaster%2Ffedora-binary-ovn-sb-db-server%3Apull: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"}16:44
weshaychanged: [localhost] => (item=panko-api)16:44
weshayoh wait.. it's there16:45
weshayhttps://console.registry.rdoproject.org/registry#/images/tripleomaster/fedora-binary-ovn-northd:d805e6b1285bf6e7d99163cd90fe3d04751eebb9_ba4d6a3d16:45
weshayanyone have a skopeo example w/ rdo-registry?16:46
zbr|rucknot sure I can query the registry with the browser, 504 Gateway Time-out -- i guess we may be facing some scalability issues.16:47
*** udesale has quit IRC16:50
*** derekh has quit IRC16:51
ykarel|awayweshay, u looking for: skopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-cinder-api:a682e2f8c05bfd62b3784ca641951dde3c2c11ea_a7fba5f316:54
ykarel|away504 gateway timeout with skopeo as well16:55
weshayok.. you are getting the same.. you are saying?16:56
ykarel|awayyes seems there is issue, few hour back there was ooom16:56
weshaythat just worked for me16:56
ykarel|awaymay be it hit again16:56
weshayykarel|away fun..16:56
weshayykarel|away ok.. thanks for the update.. that's why we retry and retry on the promotion server :)16:57
weshayat first you don't succeed16:57
ykarel|awayyes it worked for me as well with retry16:57
ykarel|awayskopeo inspect i mean16:57
weshayaye.. thanks for looking into it16:57
ykarel|awayi requested to upgrade flavor for rdo registry16:58
ykarel|awayas it's consuming full ram16:58
ykarel|awayand it's not behaving good from last 2 days,16:59
*** gkadam has quit IRC17:00
rlandyzbr|ruck: same happens when I try access the registry17:05
rlandybeen happening for some time17:05
zbr|ruckrlandy: i remember reading something on #rdo about an upcoming migration17:06
zbr|ruckthe idea is that w/o a reliable container registry we are pretty much f***17:06
* ykarel|away out17:24
*** chem has quit IRC17:24
*** ykarel|away has quit IRC17:28
weshaybbiab17:44
weshayhrm.. not yet17:46
*** tesseract has quit IRC17:50
*** holser_ has quit IRC18:10
*** holser__ has joined #oooq18:10
*** tosky has joined #oooq18:19
rlandyweshay: hi - when you get back  - can we chat for a few about rrcockpit?18:25
*** vinaykns has quit IRC18:28
*** holser__ has quit IRC18:36
weshayrlandy ya.. on a call will ping when I'm done19:24
rlandyweshay: sure - no rush19:25
*** Vorrtex has quit IRC19:27
*** vinaykns has joined #oooq19:37
*** tosky__ has joined #oooq20:03
*** tosky has quit IRC20:03
*** holser_ has joined #oooq20:15
*** holser_ has quit IRC20:19
*** rfolco has quit IRC20:25
weshayrlandy too late?20:32
weshaylong call20:32
rlandyweshay: np - one minute20:32
rlandyweshay: k - joining bj - if you can chat now20:35
weshayrlandy yup20:35
*** Goneri has quit IRC20:45
weshayrlandy https://rootnroll.com/d/telegraf/20:48
*** jtomasek has quit IRC21:38
*** tosky__ is now known as tosky22:14
rlandyweshay: flight booked - thanks for the info22:51
weshayrlandy :)22:52
weshaythanks22:52
rlandyweshay: it was $300 cheaper than estimate - so you can add that to your flight22:52
weshayheh.. thanks22:52
*** rlandy has quit IRC23:00
*** tosky has quit IRC23:22

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!