Wednesday, 2019-10-02

« Tuesday, 2019-10-01 Index Thursday, 2019-10-03 »
openstackgerritMerged openstack/ansible-role-tripleo-modify-image master: Fix exists check  https://review.opendev.org/68413400:02
openstackgerritMerged openstack/tripleo-common master: Always close src/dst sessions and raise to retry  https://review.opendev.org/68478600:02
*** zaneb has quit IRC00:12
*** zaneb has joined #tripleo00:12
openstackgerritMerged openstack/puppet-tripleo master: Use correct paths to configure ovn dbs certs  https://review.opendev.org/68391600:20
*** toure is now known as toure|gone00:20
*** zaneb has quit IRC00:34
*** zaneb has joined #tripleo00:35
*** zaneb has quit IRC00:36
*** zaneb has joined #tripleo00:36
*** gbarros has quit IRC00:45
*** ricolin has joined #tripleo00:59
*** mschuppert has quit IRC01:06
openstackgerritToure Dunnon proposed openstack/tripleo-ansible master: [WIP] Add service management support.  https://review.opendev.org/68101701:15
openstackgerritToure Dunnon proposed openstack/tripleo-ansible master: [WIP] Add service management support.  https://review.opendev.org/68101701:19
*** ricolin has quit IRC01:34
*** ricolin has joined #tripleo01:35
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Add podman_container_facts  https://review.opendev.org/68603501:38
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: podman_image_facts: fix listing all images  https://review.opendev.org/68608001:40
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Add podman_container_facts  https://review.opendev.org/68603501:45
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP: POC of podman_container  https://review.opendev.org/68245801:46
*** spsurya has joined #tripleo01:47
*** cdearborn has quit IRC01:57
*** ayoung has quit IRC02:08
*** irclogbot_2 has quit IRC02:09
*** irclogbot_1 has joined #tripleo02:13
*** saneax has joined #tripleo03:18
openstackgerritMerged openstack/tripleo-validations master: Constrain molecule  https://review.opendev.org/68571503:30
*** saneax has quit IRC03:37
openstackgerritMerged openstack/tripleo-quickstart master: fix tq,cico setup  https://review.opendev.org/68509003:41
openstackgerritMerged openstack/tripleo-common master: Warn when looking up grandparent resource name  https://review.opendev.org/68269404:08
*** mmethot_ has joined #tripleo04:36
*** ricolin_ has joined #tripleo04:36
*** elfGamal has quit IRC04:37
*** cgoncalves has quit IRC04:37
*** numans has quit IRC04:37
*** elGamal has joined #tripleo04:38
*** d0ugal has quit IRC04:38
*** ricolin has quit IRC04:40
*** mmethot has quit IRC04:40
*** dasp has quit IRC04:40
*** jamesdenton has quit IRC04:41
*** jamesdenton has joined #tripleo04:44
*** cgoncalves has joined #tripleo04:46
*** aakarsh has quit IRC04:48
*** pcaruana has joined #tripleo04:52
*** d0ugal has joined #tripleo04:52
*** ykarel|away has joined #tripleo05:09
*** marios has joined #tripleo05:18
*** lmiccini has joined #tripleo05:27
*** yprokule has joined #tripleo05:39
*** hjensas has quit IRC05:44
*** kopecmartin|off is now known as kopecmartin05:49
*** jfrancoa has joined #tripleo05:54
*** rcernin_ has joined #tripleo05:59
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add registry authentication parameters to pull ceph related containers  https://review.opendev.org/67769906:00
*** rcernin has quit IRC06:01
*** mschuppert has joined #tripleo06:29
*** dasp has joined #tripleo06:40
*** rcernin_ has quit IRC06:45
*** rcernin has joined #tripleo06:45
*** xek has joined #tripleo06:55
*** tesseract has joined #tripleo06:57
*** jbadiapa has joined #tripleo07:00
*** iurygregory has joined #tripleo07:00
*** dciabrin has joined #tripleo07:01
*** jaosorior has joined #tripleo07:03
*** rcernin has quit IRC07:04
*** slaweq has joined #tripleo07:10
*** hjensas has joined #tripleo07:10
*** jpich has joined #tripleo07:11
*** jbadiapa has quit IRC07:13
*** tosky has joined #tripleo07:17
*** jbadiapa has joined #tripleo07:19
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] Add NetworkManager and dhcp-client for CentOS 8  https://review.opendev.org/68609007:20
*** pkopec has joined #tripleo07:23
*** pierreprinetti has joined #tripleo07:28
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade master: Introduce molecule tests in the tripleo-upgrade role  https://review.opendev.org/68504107:28
openstackgerritMerged openstack/tripleo-heat-templates stable/rocky: Add parameters for dateext in logrotate  https://review.opendev.org/68493807:29
*** amoralej|off is now known as amoralej07:30
*** bogdando has joined #tripleo07:33
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: Add a configurable delay to Nova Evacuate calls  https://review.opendev.org/68478507:33
*** holser has joined #tripleo07:34
*** jbadiapa has quit IRC07:34
*** slaweq has quit IRC07:38
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations stable/stein: New validation: detect failed containers  https://review.opendev.org/68609307:43
*** jpena|off is now known as jpena07:48
openstackgerritKeigo Noha proposed openstack/tripleo-heat-templates stable/queens: Add parameters for dateext in logrotate  https://review.opendev.org/68609807:52
*** pkopec has quit IRC07:56
*** florianf has joined #tripleo07:56
*** rpittau|afk is now known as rpittau07:57
*** slaweq has joined #tripleo07:57
*** suuuper has joined #tripleo08:03
openstackgerritJavier Peña proposed openstack/paunch master: Properly mock cpu_affinity on all unit tests  https://review.opendev.org/68610008:04
*** tkajinam has quit IRC08:04
*** jpena is now known as jpena|brb08:10
openstackgerritNatal Ngétal proposed openstack/python-tripleoclient master: Fix message of unused_params and invalid_role_specific_params.  https://review.opendev.org/67457008:11
*** jpich has quit IRC08:13
*** paramite has joined #tripleo08:14
*** weshay has quit IRC08:15
*** weshay has joined #tripleo08:16
Tengubogdando: all the patches merged! I'm pretty sure CI will be better from now on :)08:21
*** pkopec has joined #tripleo08:25
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Sync up upgrades and updates workarounds mechanism.  https://review.opendev.org/68441908:26
*** ykarel|away has quit IRC08:28
*** ykarel has joined #tripleo08:30
*** ykarel is now known as ykarel|away08:30
*** gfidente has joined #tripleo08:32
*** jpena|brb is now known as jpena08:32
*** alexmcleod has joined #tripleo08:33
*** jaosorior has quit IRC08:37
*** derekh has joined #tripleo08:38
*** ricolin_ is now known as ricolin08:41
*** panda has quit IRC08:48
*** panda has joined #tripleo08:49
*** holser has quit IRC09:02
*** holser has joined #tripleo09:02
*** holser has quit IRC09:03
*** holser has joined #tripleo09:03
*** jfrancoa has quit IRC09:03
*** florianf has quit IRC09:11
*** jpich has joined #tripleo09:17
*** jfrancoa has joined #tripleo09:18
*** ab-a has quit IRC09:30
*** jbadiapa has joined #tripleo09:30
*** ab-a has joined #tripleo09:31
*** holser has quit IRC09:34
*** holser has joined #tripleo09:36
*** florianf has joined #tripleo09:39
openstackgerritMerged openstack/tripleo-validations stable/stein: Add playbook creation for the automation  https://review.opendev.org/68593809:44
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations stable/stein: Constrain molecule  https://review.opendev.org/68611909:52
*** jaosorior has joined #tripleo09:52
*** jpich has quit IRC09:55
*** jpich has joined #tripleo09:55
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade master: Introduce molecule tests in the tripleo-upgrade role  https://review.opendev.org/68504109:56
*** iurygregory has quit IRC09:57
*** hjensas has quit IRC10:06
openstackgerritCarlos Goncalves proposed openstack/tripleo-heat-templates master: Fix placement of Octavia services in roles  https://review.opendev.org/68612510:08
*** gfidente has quit IRC10:09
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade master: Introduce molecule tests in the tripleo-upgrade role  https://review.opendev.org/68504110:12
*** iurygregory has joined #tripleo10:17
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade master: Introduce molecule tests in the tripleo-upgrade role  https://review.opendev.org/68504110:20
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates stable/queens: [Queens] Run cellv2 host discovery via workflow or deploy_steps_tasks  https://review.opendev.org/68306910:22
*** rcernin has joined #tripleo10:25
openstackgerritMerged openstack/python-tripleoclient stable/stein: Close some file descriptors that were left open  https://review.opendev.org/68601510:28
*** ykarel|away has quit IRC10:31
*** slaweq has quit IRC10:38
*** zbr|ruck is now known as zbr|lunch10:39
*** slaweq has joined #tripleo10:42
*** dsneddon has quit IRC10:44
*** tosky_ has joined #tripleo10:49
*** slaweq has quit IRC10:51
*** tosky has quit IRC10:52
*** slaweq has joined #tripleo10:57
*** jbadiapa has quit IRC10:57
*** tosky_ is now known as tosky10:58
*** tosky_ has joined #tripleo11:02
*** tosky is now known as Guest1863911:02
*** tosky_ is now known as tosky11:02
*** jaosorior has quit IRC11:02
*** rh-jelabarre has quit IRC11:04
*** Guest18639 has quit IRC11:04
*** rh-jelabarre has joined #tripleo11:05
*** csmart has quit IRC11:07
*** dsneddon has joined #tripleo11:09
*** openstackstatus has quit IRC11:09
*** rnoriega- has quit IRC11:09
*** rnoriega_ has joined #tripleo11:13
*** csmart has joined #tripleo11:18
*** gfidente has joined #tripleo11:22
*** jpena is now known as jpena|lunch11:24
*** jaosorior has joined #tripleo11:28
*** ansmith has quit IRC11:35
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Add NetworkManager and dhcp-client for CentOS 8  https://review.opendev.org/68609011:36
openstackgerritIan Wienand proposed openstack/diskimage-builder master: CentOS 8 minimal testing and support  https://review.opendev.org/68486011:36
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] use mirrors for functional test builds  https://review.opendev.org/68588511:36
*** dpeacock has quit IRC11:37
*** raildo has joined #tripleo11:38
*** rpittau is now known as rpittau|afk11:38
*** dpeacock has joined #tripleo11:40
*** artom has joined #tripleo11:47
*** raildo has quit IRC11:52
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade master: Introduce molecule tests in the tripleo-upgrade role  https://review.opendev.org/68504111:52
*** raildo has joined #tripleo11:54
*** dsneddon has quit IRC11:55
*** ekultails has joined #tripleo11:58
*** suuuper has quit IRC11:59
*** dsneddon has joined #tripleo12:03
*** jbadiapa has joined #tripleo12:06
*** amoralej is now known as amoralej|lunch12:07
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates stable/queens: [Queens] Run cellv2 host discovery via workflow or deploy_steps_tasks  https://review.opendev.org/68306912:07
*** jpich has quit IRC12:12
*** jpich has joined #tripleo12:13
*** dsneddon has quit IRC12:13
openstackgerritTom Barron proposed openstack/tripleo-heat-templates master: Fix Manila when run without pacemaker  https://review.opendev.org/68285312:15
*** gbarros has joined #tripleo12:16
*** dsneddon has joined #tripleo12:19
*** dsneddon has quit IRC12:25
*** derekh has quit IRC12:27
*** ansmith has joined #tripleo12:27
*** jpena|lunch is now known as jpena12:29
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade master: Introduce molecule tests in the tripleo-upgrade role  https://review.opendev.org/68504112:30
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Add podman_container_facts  https://review.opendev.org/68603512:32
*** gbarros has quit IRC12:32
*** jbadiapa has quit IRC12:35
slaweqchandankumar: hi12:37
slaweqchandankumar: I have a question according Your last comment in https://review.opendev.org/#/c/655423/12:39
slaweqchandankumar: how I can check deployment type and choose tobiko_cloud_name value based on that?12:39
slaweqchandankumar: is there any variable which says that deployment is standalone or multinode?12:40
*** rlandy has joined #tripleo12:40
openstackgerritMerged openstack/tripleo-validations master: Update documentation URL link for CPU & RAM Undercloud recommendations  https://review.opendev.org/68597112:43
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations stable/stein: Update documentation URL link for CPU & RAM Undercloud recommendations  https://review.opendev.org/68614512:44
*** hjensas has joined #tripleo12:48
*** amoralej|lunch is now known as amoralej12:51
*** aakarsh has joined #tripleo12:52
*** Goneri has joined #tripleo12:53
*** mcornea has joined #tripleo12:53
chandankumarslaweq: Hello12:56
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade master: Introduce molecule tests in the tripleo-upgrade role  https://review.opendev.org/68504112:57
openstackgerritMartin Mágr proposed openstack/puppet-tripleo master: Configuration changes to support Qdr-mesh topology.  https://review.opendev.org/64790812:58
*** dsneddon has joined #tripleo12:58
slaweqchandankumar: I think I found some way to get it12:58
chandankumarslaweq: cool12:58
chandankumarslaweq: you can add a value in fs override and reuse that12:59
slaweqchandankumar: I'm checking if standalone.sh script exists, if not, I assume cloud name will be "overcloud"12:59
*** derekh has joined #tripleo12:59
chandankumarslaweq: for standalone it is 'standalone' for overcloud it is overcloud and for undercloud it is undercloud12:59
slaweqchandankumar: what value I can add in fs override exactly?12:59
*** gbarros has joined #tripleo13:00
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Use tempest_cloud_name in featureset_override  https://review.opendev.org/68614913:03
chandankumarslaweq: ^6 use this patch as depends-on and just below featureset_override add tempest_cloud_name: 'standalone' and reuse the var in tobiko playbook13:04
*** jaosorior has quit IRC13:08
*** jpich has quit IRC13:14
slaweqchandankumar: thx, I will do in that way13:14
*** jpich has joined #tripleo13:15
slaweqchandankumar: one more thing, can You take a look at https://review.opendev.org/#/c/685784/ - it seems that in last patch I somehow missed to change this one name also :/13:15
slaweqand because of that multinode job with fs 016 is failing still for me :/13:16
chandankumarslaweq: checking13:16
slaweqchandankumar: thx13:16
weshayfultonj, and tom :) I need a little time to checkout the rhel containers13:19
weshay2019-10-02 08:36:41 | tripleo_common.image.exception.ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/rhel-binary-cron:fd46c80632bbd7abc8411277bfc4819f6b95a485_bf6d0acd13:19
weshayhttp://logs.rdoproject.org/53/682853/10/openstack-check/tripleo-ci-rhel-8-scenario004-standalone-rdo/d9a9074/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz13:19
weshayfultonj, to test w/ rhel 8.. I *think* putting your patch as a depends-on to: https://review.opendev.org/#/c/682853/ would be the right thing?13:20
openstackgerritSlawek Kaplonski proposed openstack/tripleo-quickstart-extras master: Add playbook and role to run tobiko  https://review.opendev.org/65542313:20
fultonjweshay: or i could make a tht patch which depends on it13:22
weshayaye13:22
fultonjone that has no content if we want to test them separately13:22
fultonjweshay: i'll make a tht review like that13:26
fultonjif tbarron wants to add a depends on too, then we'd consider that an extra test13:26
*** toure|gone is now known as toure13:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-common master: Implement Ansible fact cache for Mistral executor  https://review.opendev.org/68285513:27
*** mcornea has quit IRC13:27
*** hjensas has quit IRC13:36
*** mcornea has joined #tripleo13:42
*** ykarel has joined #tripleo13:45
*** zbr|lunch is now known as zbr|ruck13:48
openstackgerritMartin Mágr proposed openstack/puppet-tripleo master: Configuration changes to support Qdr-mesh topology.  https://review.opendev.org/64790813:49
*** bdodd has joined #tripleo13:54
*** jaosorior has joined #tripleo13:54
*** jhakimra has joined #tripleo13:59
bogdandoowalsh, Tengu: hi, some testing shows the lazy activation cannot be disabled just like that https://pastebin.com/tV87RfRC (tested for py27). Therefore no memory leaks was discovered as well. I wonder how to disable it properly and then repeat testing but for py3 perhaps?..14:06
bogdandowrt https://review.opendev.org/#/c/68569814:06
fultonjweshay: looks like https://review.opendev.org/#/c/682853 already depends on https://review.opendev.org/#/c/68544914:06
*** panda has quit IRC14:06
Tengubogdando: well, apparently that option will be removed. That's why I dropped the change.14:07
bogdandowould be nice to know more about other alternatives14:07
Tengubogdando: owalsh pointed one, should be good.14:08
*** panda has joined #tripleo14:08
bogdandoas now we have a testing snippet to easily evaluate it against possible mem leaks, owalsh ^^14:08
bogdandoWSGIImportScript ?14:08
owalshbogdando: yea, that is the only documented solution14:08
owalshI don't see how disabling lazy loading solves the problem, wsgi app stil isn't imported till the first requests IIUC14:10
bogdandowill add that to my testing then14:10
bogdandoowalsh: that's what I was observing right14:11
bogdando:)14:11
openstackgerritMerged openstack/puppet-tripleo master: Add a configurable delay to Nova Evacuate calls  https://review.opendev.org/68478514:11
owalshalso the mod_wsgi release notes say to never use it14:11
*** gkadam has joined #tripleo14:12
*** gkadam has quit IRC14:12
tbarronweshay: I dunno why we are getting that image not found in the rhel8 job:  2019-10-02 08:36:41 | tripleo_common.image.exception.ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/rhel-binary-cron:fd46c80632bbd7abc8411277bfc4819f6b95a485_bf6d0acd14:12
weshaytbarron, aye.. that's on me and arxcruz|ruck right now14:12
tbarronweshay: I think other rhel8 scenario jobs are getting it too14:13
weshayone fire at a time.. we'll get through this14:13
tbarronweshay: k, just checking :)14:13
weshaytbarron, aye.. it's not job specific14:13
weshayit's the busy season :)14:13
tbarron:)14:13
*** jpich has quit IRC14:18
*** jaosorior has quit IRC14:18
*** jpich has joined #tripleo14:18
bogdandoowalsh, Tengu: so I've tried WSGIImportScript and it changed nothing14:21
bogdandonothing loaded14:21
bogdandothe only option left is redesign health checks to be active and send a wake-up requests14:21
bogdandoto the API port14:22
owalshbogdando: really? What did you set WSGIImportScript to?14:22
bogdandoWSGIImportScript /var/www/cgi-bin/heat/heat_api14:22
bogdandowhere I put my mem-alloc tester14:23
owalshbogdando: needs more than that14:23
openstackgerritSorin Sbarnea proposed openstack/ansible-role-container-registry master: Disable docker iptables support  https://review.opendev.org/68617314:23
bogdandomhm14:23
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates master: Add Octavia driver agent service  https://review.opendev.org/65811814:23
* bogdando back to rtfms14:23
*** openstackstatus has joined #tripleo14:29
*** ChanServ sets mode: +v openstackstatus14:29
bogdandoowalsh: tried https://pastebin.com/PcLx516S but nothing changes14:30
*** rcernin has quit IRC14:30
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-common master: Implement Ansible fact cache for Mistral executor  https://review.opendev.org/68285514:35
openstackgerritSorin Sbarnea proposed openstack/ansible-role-container-registry master: Disable docker iptables support  https://review.opendev.org/68617314:35
owalshbogdando: could be that heat wsgi module isn't doing the right thing?14:35
bogdandoowalsh: it allocates memory by a request, so my substituted code works14:36
bogdandoit also returns the expected output (the name of process group)14:36
owalshbogdando: not sure what you mean, what substituted code?14:37
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-common master: Implement Ansible fact cache for Mistral executor  https://review.opendev.org/68285514:37
bogdandodid you look through the testing results?14:37
bogdandoowalsh: I'm using a mocked WSGI app14:37
*** yprokule has quit IRC14:37
owalshbogdando: what testing results?14:38
bogdandoowalsh: https://gist.githubusercontent.com/bogdando/43a6c5938c964fa4ee94f714e0638ee5/raw/6065e9ce1b146ea9cf3bf7de064f5217efaa3a10/gistfile1.txt and results are here:14:38
bogdando16:06:19 - bogdando: owalsh, Tengu: hi, some testing shows the lazy activation cannot be disabled just like that https://pastebin.com/tV87RfRC (tested for py27).  […]14:38
*** jpich has quit IRC14:38
owalshbogdando: you're wsgi app does nothing on import14:39
owalshs/you're/your14:39
*** jpich has joined #tripleo14:39
bogdandoowalsh: dunno, it works14:39
bogdandocould you look through results first :) ?14:39
owalshbogdando: yea, but when WSGIImportScript imports it it needs to do the init14:39
bogdandoit keeps memory allocated14:39
bogdandough14:40
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart master: Use custom ceph-ansible until c8 storage sig ready  https://review.opendev.org/68544914:40
bogdandothat's getting complicated to keep testing it right...14:40
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Add podman_container_info  https://review.opendev.org/68603514:40
bogdandoI give up :)14:40
zbr|ruckmwhahaha: if fixed iptrables order, but i have a question: do we really need to do this for all installs? i personally don't really like that customization.14:41
mwhahahazbr|ruck: no we don't. this is happening because you're doing this outside of tripleo. this is a tripleo ci specific thing14:41
mwhahahazbr|ruck: this should be handled elsewhere or configurable in the install14:42
mwhahahaWe actually already do this in that role14:42
mwhahahahttps://opendev.org/openstack/ansible-role-container-registry/src/branch/master/tasks/docker.yml#L22-L2814:43
* mwhahaha doesn't exactly have the correct context for this entire issue14:43
fultonjtbarron: now that the depends-on of your patch has been udpated, i think you need to rebase  https://review.opendev.org/#/c/682853/14:44
mwhahahai'd just move it and make it an option. eg. disable_iptables_management14:44
mwhahahaor something14:44
openstackgerritTom Barron proposed openstack/tripleo-heat-templates master: Fix Manila when run without pacemaker  https://review.opendev.org/68285314:44
tbarronfultonj: done14:45
owalshbogdando: if you look at the regular /var/www/cgi-bin/heat/heat_api it does it14:46
owalshf application is None:14:46
owalsh            application = init_application()14:46
openstackgerritDaniel King proposed openstack/python-tripleoclient master: Removed references to bm-deploy images  https://review.opendev.org/68606414:46
bogdandoowalsh: I'd appreciate if you could fix it for me :)14:46
*** Vorrtex has joined #tripleo14:47
bogdandoowalsh: I'll try WSGIScriptAlias14:48
bogdandojust reading https://stackoverflow.com/questions/20790077/django-startup-code-with-wsgi-lazy-loading14:48
ade_leemwhahaha, hey - when trying to debug by running container-puppet.py, its failing with message "Missing CNI default network"14:48
ade_leemwhahaha, any idea how to get past that?14:48
mwhahaha--net=host14:48
zbr|ruckmwhahaha: wow, i discovered that we already had this in container_registry_docker_options in default, which should have worked anyway.14:49
zbr|ruckweshay: ^ mainly we are talking about a change that is already in the role. where do we call the role from?14:50
mwhahahaso docker.yaml is for the registry setup14:50
mwhahahawhich should be before the docker install14:50
mwhahahaat least it is in tripleo deploys14:51
ade_leemwhahaha, so according to doc, we are running "SHOW_DIFF=True PROCESS_COUNT=1 CONFIG=glance_api.json ./container-puppet.py"  , we need to add --net=host ?14:52
mwhahahano the container needs a --net=host when it's launched14:52
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart master: Use custom ceph-ansible until c8 storage sig ready  https://review.opendev.org/68544914:52
mwhahahain THT that's usually net: host in the definition14:52
zbr|ruckif i discover that this was caused by use of tasks_from... i am going to the first pub and not return till tomorrow.14:52
mwhahahazbr|ruck: enjoy your beer then14:52
zbr|rucknot sure what is your oppinion re use of tasks_from with roles because I see it like using "jmp" in C code....14:53
zbr|ruckno way to control the entry points14:53
owalshbogdando: see the last comment here https://stackoverflow.com/questions/20958553/wsgi-startup-why-two-identical-processes14:53
openstackgerritTom Barron proposed openstack/tripleo-heat-templates master: Fix Manila when run without pacemaker  https://review.opendev.org/68285314:55
hrybackiweshay: do you happen to recall where in oooq/oooq-e the undercloud gets enrolled to the IPA server? I'm grooming through the repos but I can't find an `ipa-client-install` anywhere14:56
bogdandoowalsh: thanks, interesting14:59
owalshbogdando: there are some caveats when setting application_group though14:59
bogdandobut it seems it needs to be non-empty15:00
owalshbogdando: really, it's set to %{GLOBAL} on my deployment15:01
owalshwhich means ''15:01
bogdandoright15:01
bogdandonow I'm editing it directly in /etc/httpd/conf.d/10-heat_api_wsgi.conf15:01
bogdandobut it has yet worked for me15:01
openstackgerritTom Barron proposed openstack/tripleo-heat-templates master: Fix Manila when run without pacemaker  https://review.opendev.org/68285315:02
mwhahahaweshay: i figured out how to speed up container processing even more btw.15:03
weshaymwhahaha++ how's that?15:03
mwhahahamagic15:03
mwhahaharm -rf /* and leaving15:03
weshaymwhahaha, and cloudnull I think we need to figure out how to make the case w/ infra about the proxies15:03
weshaymwhahaha, :)15:03
mwhahahathe proxies are likely fine15:04
weshayhrm.. why the huge diff per provider then?15:04
weshaylocation?15:04
owalshbogdando: what does worked mean?15:04
mwhahahafew different reasons. let me post a patch and we can talk15:04
bogdandoit it does not start15:04
bogdandoalthough I have no ideas for the proper initialization in the wsgi app , what you've mentioned earlier15:05
owalshbogdando: don't see how you can tell if that wsgi app was imported at process start or now. If you add a call to application() at the end maybe then you'll see the memory being allocated on import15:05
bogdandojust dumb print works for some folks15:05
weshayzbr|ruck, how can I help?15:07
zbr|ruckhttps://bluejeans.com/265541792815:07
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Switch to ProcessPoolExecutor  https://review.opendev.org/68618115:08
mwhahahaweshay, EmilienM, bogdando: -^15:08
mwhahahacut my processing down to 10 mins15:08
mwhahahawe started with 32 in my env15:08
mwhahaha22 after randomizing15:08
mwhahaha10 after ProcessPoolExecutor15:08
EmilienMnicely done15:08
mwhahahanext we can tweak the workers for that and we might get more15:08
Tenguwow.15:10
*** Vorrtex has quit IRC15:10
*** jpich has quit IRC15:11
*** jpich has joined #tripleo15:12
ykarelmwhahaha, re. ^^ is the issue only with py2? or both py2/py315:14
ykareli noticed in rhel8 container prep was faster15:14
EmilienMi think it's the same for both py2/py315:15
EmilienM(on multiprocess vs thread)15:15
ykarelin rhel8 i saw container prep was taking around 5 minutes15:15
EmilienMin which infra?15:15
ykarelso i thought the performance was related to py2 only15:15
ykarelrdo15:15
ykareli also thought it can be related to subprocess perf on py2, but can be other reasons as well15:17
mwhahahamight be mostly py215:18
mwhahahai haven't been testing under py315:18
mwhahahai think we'll see improvements in both15:18
openstackgerritMerged openstack/tripleo-validations stable/stein: New validation: detect failed containers  https://review.opendev.org/68609315:18
mwhahahasince CI is currently py2, it's the primary target :D15:19
mwhahahapy3 likely has better stream processing15:19
*** gfidente has quit IRC15:20
*** mcornea has quit IRC15:21
*** dtrainor has quit IRC15:21
ykarelmwhahaha, okk15:22
ykarelmwhahaha, EmilienM just for reference:- https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario002-standalone-master/164c330/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-10-01_19_29_1315:22
ykarelhttps://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario003-standalone-master/a289723/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-10-01_19_28_2315:23
mwhahahastandalone tends to be faster than undercloud i think15:24
mwhahahai think it's less containers15:24
mwhahahabut yea that's definately faster15:24
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-quickstart master: Fix secondary (overcloud) subnode indexes  https://review.opendev.org/68234615:25
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-ci master: Fallback to public IPv4 when there is no priv. IP  https://review.opendev.org/68209215:25
mwhahahabuildah is probably also better on rhel8 because of the correct kernel15:25
mwhahahaso in general, it's going to have better perf for a lot of things15:25
bogdandoowalsh: I've tried wsgi for a few hours only and already hate it by all my soul :D15:26
owalshowalsh: lets go back to cgi scripts :-)15:26
ykarelmwhahaha, same data for undercloud/overcloud as well:- https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/a000b39/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-09-15_09_58_3115:26
ykarelaround 5 minutes15:26
bogdandoyea15:26
mwhahahaykarel: yea buildah tasks 10-15 seconds instead for like 30-60 on centos. though i'm not seeing any actual yum updates occuring15:26
mwhahahawho knows :D15:27
bogdandomwhahaha, ykarel: weshay said periodic jobs never update it15:27
ykarelmwhahaha, yes yum update not happening as dnf strict fails if repo doesn't exist15:27
bogdandoso there is always an empty run15:27
mwhahahayea so it's apples to ornages on that then15:27
ykarelmwhahaha, to rule out yum_update, i tried disabling yum_update, but no improvement with centos15:28
mwhahahaall my testing is without yum update15:28
mwhahahaso just the base fetching15:28
bogdandoowalsh: my best state of testing it now auto starts but fails with bad python code it seems I have no idea how to debug15:28
mwhahahayum update adds a bunch of time. in my testing it was adding back the 10 mins i gained from shuffling the list15:28
bogdando500 Internal Server Error :/15:28
openstackgerritCarlos Goncalves proposed openstack/tripleo-common stable/stein: Fixup octavia-api healthcheck  https://review.opendev.org/68618615:29
bogdandomwhahaha: my profiling shows it's always like 11-23 min for 20-50 packages updated per container15:29
*** aakarsh has quit IRC15:29
ykarelbogdando, mwhahaha so to rule out yum_update.sh i tried with https://review.rdoproject.org/r/#/c/22366/15:30
bogdandobut the most weird was there is always a delay of a 45m -1h before yum updating the last two containers o.O15:30
ykarelbut not much improvement with centos15:30
*** gbarros has quit IRC15:30
ykareland rhel8 was always faster w or without it15:30
* mwhahaha waits for centos8 to get on with it15:31
Tenguan official image for centos-8 is in the QA pipe afaik.15:31
openstackgerritCarlos Goncalves proposed openstack/tripleo-common stable/rocky: Fixup octavia-api healthcheck  https://review.opendev.org/68618915:32
openstackgerritCarlos Goncalves proposed openstack/tripleo-common stable/queens: Fixup octavia-api healthcheck  https://review.opendev.org/68619015:32
*** marios has quit IRC15:35
*** mcornea has joined #tripleo15:36
*** gbarros has joined #tripleo15:36
*** iurygregory has quit IRC15:37
openstackgerritMerged openstack/tripleo-quickstart master: Switching default centos url for image promotion  https://review.opendev.org/68570915:38
openstackgerritMerged openstack/paunch master: Properly mock cpu_affinity on all unit tests  https://review.opendev.org/68610015:38
*** tosky_ has joined #tripleo15:41
*** tosky is now known as Guest8305815:41
*** tosky_ is now known as tosky15:41
*** Guest83058 has quit IRC15:43
openstackgerritMerged openstack/ansible-role-tripleo-modify-image master: Enable saving of rpms  https://review.opendev.org/68580615:47
openstackgerritSorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Disable docker iptables  https://review.opendev.org/68619215:48
*** mdbooth has quit IRC15:50
*** aakarsh has joined #tripleo15:50
*** dtrainor has joined #tripleo15:51
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Use tenancity to backoff when layer is locked  https://review.opendev.org/68619315:51
*** mdbooth has joined #tripleo15:51
*** ekultails has quit IRC15:54
*** ekultails has joined #tripleo15:58
*** ykarel is now known as ykarel|away16:01
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP - Introduce tripleo-container-manage role  https://review.opendev.org/68619616:02
*** florianf has quit IRC16:03
*** jfrancoa has quit IRC16:04
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade master: Introduce molecule tests in the tripleo-upgrade role  https://review.opendev.org/68504116:07
openstackgerritJeremy Stanley proposed openstack/diskimage-builder master: Add security mirror override for debian-minimal  https://review.opendev.org/68619916:09
*** gfidente has joined #tripleo16:09
openstackgerritLuke Short proposed openstack/python-tripleoclient master: [WIP] Handle a deployment that was manually cancelled.  https://review.opendev.org/68620216:10
*** pcaruana has quit IRC16:10
*** pcaruana has joined #tripleo16:11
bogdandoowalsh: I ended up just patching the original heat wsgi app16:12
bogdandowith that allocator16:12
bogdandoso now it works as usual but just allocates 1G per worker16:13
bogdandoand whatever options I played, it never auto loads it after container restart ...16:13
bogdandoowalsh: here is the end configs state http://paste.openstack.org/show/kycalFEyFNAsgVyFJjLN/16:14
Tengubogdando: "For some cases, such as when WSGIScriptAlias is being used, it is actually possible to preload the application code when the processes first starts, rather than when the first request arrives. To preload an application see the WSGIImportScript directive."16:15
Tengubogdando: the "SGIScriptAlias" is a typo right?16:16
openstackgerritMerged openstack/tripleo-validations stable/stein: Constrain molecule  https://review.opendev.org/68611916:16
bogdandoyeah, fixed that16:17
bogdandowow, it worked if moved to the main 10-* script16:17
bogdandonever works if placed in the modules config16:17
Tenguorder does matter16:17
*** aakarsh has quit IRC16:17
bogdandoTengu, owalsh: so I re-tested it with autoload, no memory leaks16:18
bogdandoI think we can use that approach then16:19
bogdandoTengu: just amend 10-heat_api_wsgi.conf instead16:19
Tengubogdando: do you have a way to inject that autoload in t-h-t or puppet? not 100% sure hiera is the place to be16:19
Tenguah? wait.16:19
bogdandoyea16:19
bogdandojust that file16:19
bogdandogtg see you tomorrow16:20
Tenguwhere/how is it generated? that's the question. in puppet, yes. but does it take things out of t-h-t?16:20
bogdandothat's prolly puppet-foo for each16:20
Tenguhmmmmm16:20
*** jfrancoa has joined #tripleo16:20
TenguMAYBE heat::wsgi::apache_api:: can be the entry within heat.16:21
openstackgerritJeremy Stanley proposed openstack/diskimage-builder master: Add security mirror override for debian-minimal  https://review.opendev.org/68619916:21
Tenguneed to dig a bit in the puppet foo.16:21
mwhahahait comes from puppet-apache16:21
mwhahahapuppet-heat just wraps the apache stuff16:21
*** rlandy is now known as rlandy|brb16:22
Tengumwhahaha: meaning we need to edit every single puppet module then ? no way to inject that from the mighty puppet-tripleo?16:22
mwhahahalikely16:22
Tenguhere we're talking about heat, but nova and friends will (probably) need that change as well.16:22
mwhahahahttps://github.com/openstack/puppet-heat/blob/master/manifests/wsgi/apache.pp#L14016:23
mwhahahamight be able to patch openstacklib16:23
Tengupfrrttt16:23
Tenguah, would be best.16:23
mwhahahahttps://github.com/openstack/puppet-openstacklib/blob/master/manifests/wsgi/apache.pp#L28816:24
Tenguyep.16:24
TenguI'm in the file.16:24
mwhahahabasically just an apache::vhost, so if that exposes what you are looking for then you can put it in there16:24
mwhahahaor do something else16:24
Tenguyeah. we might want to make that optional though.16:24
bogdandoit should be alrady duable via custom_wsgi_process_options16:25
Tenguso it might be good to add a new param. Need to talk with owalsh when he's connected (not sure about his TZ).16:25
bogdandodoable*16:25
Tengubogdando: lemme check.16:25
bogdandoor rather vhost_custom_fragment16:25
Tenguapache::vhost::WSGIImportScript16:26
bogdandonice16:26
Tenguwe might get a shortcut in the module.16:26
bogdandohanding that over to you folks :)16:26
bogdandocu16:26
Tengujust need to ensure it's present in our version.16:26
Tenguyeah, won't do that today ;).16:26
Tengustill here due to a late mtg, but that's it :D16:26
*** tesseract has quit IRC16:29
*** bogdando has quit IRC16:30
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: nova-libvirt: set 'cpuset_cpus' to 'all'  https://review.opendev.org/68602516:37
*** rlandy|brb is now known as rlandy16:37
*** jpich has quit IRC16:37
*** tosky_ has joined #tripleo16:40
*** aakarsh has joined #tripleo16:40
*** tosky has quit IRC16:43
*** derekh has quit IRC16:43
*** rfolco is now known as rfolco|dentist16:44
*** tosky_ is now known as tosky16:45
owalshTengu: I'm GMT+!16:48
owalsherr GMT+116:48
*** lmiccini has quit IRC16:58
*** dsneddon has quit IRC16:58
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates master: Support connecting OVN DB over SSL  https://review.opendev.org/68034517:01
*** holser has quit IRC17:02
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates master: Support connecting OVN DB over SSL  https://review.opendev.org/68034517:06
*** dtrainor has quit IRC17:07
Tenguowalsh: ah, +2 here (for now) :D17:11
*** dtrainor has joined #tripleo17:11
Tenguowalsh: can we talk a bit more about that wsgi thingy and how to implement it within puppet-openstacklib tomorrow?17:11
owalshTengu: sure17:11
Tenguowalsh: good! signing off for today.17:12
Tenguhave a great evening/night ;).17:12
owalshyou too17:12
*** gfidente has quit IRC17:22
*** ykarel|away has quit IRC17:23
openstackgerritMerged openstack/tripleo-validations master: Add new healthcheck-service-status validation  https://review.opendev.org/68237717:24
*** jpena is now known as jpena|off17:27
*** dsneddon has joined #tripleo17:29
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP - Introduce tripleo-container-manage role  https://review.opendev.org/68619617:31
*** jhakimra has quit IRC17:33
*** dsneddon has quit IRC17:37
*** amoralej is now known as amoralej|off17:40
*** dsneddon has joined #tripleo17:41
*** pcaruana has quit IRC17:45
openstackgerritLuke Short proposed openstack/python-tripleoclient master: Handle a deployment that was manually cancelled.  https://review.opendev.org/68620217:52
*** dsneddon has quit IRC17:53
openstackgerritMerged openstack/tripleo-heat-templates master: Remove leftovers after NeutronExternalNetworkBridge option  https://review.opendev.org/68338017:53
openstackgerritLuke Short proposed openstack/python-tripleoclient master: Handle a deployment that was manually cancelled.  https://review.opendev.org/68620217:54
*** ykarel|away has joined #tripleo17:57
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Add podman_volume_info ansible module  https://review.opendev.org/68623318:13
*** aakarsh has quit IRC18:13
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Add podman_volume_info ansible module  https://review.opendev.org/68623318:13
*** dsneddon has joined #tripleo18:15
*** ykarel|away has quit IRC18:16
openstackgerritMerged openstack/tripleo-common stable/queens: [Queens] Add workflow to do cellv2 host discovery  https://review.opendev.org/68306718:16
dking_desktopI'm attempting to deploy an overcloud. The last message, just under an hour ago was [overcloud.Controller.0.DeploymentActions]: CREATE_COMPLETE  state changed. How can I check on the status?18:18
EmilienMopenstack overcloud status18:20
*** iurygregory has joined #tripleo18:21
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP - Introduce tripleo-container-manage role  https://review.opendev.org/68619618:23
openstackgerritRaildo Mascena proposed openstack/tripleo-quickstart-extras master: Add keycloak-setup role  https://review.opendev.org/67126218:31
*** cdearborn has joined #tripleo18:33
*** alexmcleod has quit IRC18:33
dking_desktopEmilienM: Thank you. It says DEPLOYING. I supplse that means that I should just wait for it to finish, and that it's just taking an abnormally long time?18:33
weshaycloudnull, hey.. looks like tripleo-bootstrap network config does not work w/ rhel 818:35
weshayhttp://logs.rdoproject.org/93/22793/1/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/6b8af1e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz18:35
weshaynetwork.service is not a native service,18:35
ekultailsweshay: He's on a business trip. I can have a look.18:37
weshayekultails, hey cool.. which is weird.. maybe the ansible module is busted or not updated enough for rhel818:37
weshay  network.service                                                                                                                      loaded    inactive   dead            LSB: Bring up/down networking18:37
weshay  NetworkManager.service                                                                                                               loaded    active     running         Network Manager18:38
weshayhttp://logs.rdoproject.org/93/22793/1/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/6b8af1e/logs/undercloud/var/log/extra/services.txt.gz18:38
weshayekultails, we can probably just loop on network vs. networkManager18:39
weshaylike we do https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo-bootstrap/tasks/main.yml#L7418:39
ekultailsI imagine we have it hardcoded in the role to use the network service instead of NetworkManager. We can have a check for the distribution. If EL7 then use network, elseif EL8||Fedora then use NetworkManager.18:40
weshayaye18:41
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates stable/stein: Add NeutronPermittedEthertypes on OVS agent  https://review.opendev.org/68624518:42
weshaywrote it up https://bugs.launchpad.net/tripleo/+bug/184639118:42
openstackLaunchpad bug 1846391 in tripleo "tripleo-bootstrap network service fails on rhel 8" [Critical,Triaged]18:42
ekultailsOur roles have functionality to include distro specific variables, too. Looks like they finally get to be useful. :-)18:43
ekultailshttps://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo-bootstrap/tasks/main.yml#L26-3118:43
ekultailsweshay: Cool, thanks for writing that bug report up. I'll get a patch out soon.18:44
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates stable/rocky: Add NeutronPermittedEthertypes on OVS agent  https://review.opendev.org/68624618:49
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates stable/queens: Add NeutronPermittedEthertypes on OVS agent  https://review.opendev.org/68624718:52
openstackgerritLuke Short proposed openstack/tripleo-ansible master: Only use the network service for EL 7.  https://review.opendev.org/68624818:55
ekultailsweshay: ^ That should do the trick.18:56
weshayekultails, k.. testing here https://review.rdoproject.org/r/#/c/22793/18:57
*** holser has joined #tripleo19:02
EmilienMdking_desktop: very likely19:05
*** kopecmartin is now known as kopecmartin|off19:05
dking_desktopWell, now, it's almost been an hour and a half. The baremetal nodes I'm using are powered off. Is that normal during that deploy step?19:08
mwhahahadking_desktop: i seem to have lost that patch you posted the other day, can you send that to me again so i can review?19:08
*** spsurya has quit IRC19:09
dking_desktopmwhahaha: Sure: https://review.opendev.org/68606419:10
mwhahahathanks19:10
mwhahahaseems like the tests are fine19:10
dking_desktopUnfortunately, even though it passed the tox unit tests and pep8 now (I had to shorten a couple lines), it's now failing a CI.19:10
dking_desktopOh? What about tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates ?19:11
mwhahahalikely unrelated but i'll look19:11
mwhahahawe don't actually exercise this code in ci (clearly)19:11
mwhahahayea un related19:13
*** ykarel has joined #tripleo19:14
dking_desktopOkay. That makes me feel a little better. I wasn't quite sure how to do that test locally. It looks like it's an Ansible playbook run, but I hadn't been able to find the details in the logs yet.19:15
dking_desktopI'm busy trying to get this overcloud deploy to work, but I can't tell if it's stuck. There's been no output in over an hour. The status is still DEPLOYING, and I see that nova processes are getting some commands, but I don't know enough yet to troubleshoot.19:16
mwhahahawhat the last thing outputed from the deploy?19:17
mwhahahaif it's in the ansible section, you can also check the ansible log in /var/lib/mistral/overcloud/ansible.log (or something like that)19:17
dking_desktop[overcloud.Controller.0.DeploymentActions]: CREATE_COMPLETE  state changed19:18
mwhahahacheck openstack server list19:18
mwhahahaare all the servers provisioned?19:18
dking_desktophttp://paste.openstack.org/show/780766/19:20
dking_desktopI just noticed that they both have the baremetal flavor, where I would have thought they would have had the compute and control flavors.19:21
openstackgerritArx Cruz proposed openstack/tripleo-ansible master: Only use the network service for EL 7.  https://review.opendev.org/68625519:24
*** ccamacho has joined #tripleo19:24
mwhahahaweshay, arxcruz|ruck: is scenario010 broken? https://review.opendev.org/#/c/683735/19:24
weshaycontainers-multinode19:25
* weshay looks19:25
mwhahahaweshay: not featureset 1019:25
mwhahahascenario1019:25
weshayoh gotcha19:25
weshaymwhahaha, yup.. looks like to broke yesterday19:26
weshaytripleo-ci-centos-7-scenario010-standaloneopenstack/tripleo-commonmastercheck678206,767112019-10-01T20:47:5119:26
weshayhttp://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-standalone19:26
weshaydang it19:26
mwhahahawhat is 10 anyway? octavia?19:26
mwhahahaor is that 1219:26
mwhahahaTASK [octavia-controller-config : get management port net mask broke19:27
*** ccamacho has quit IRC19:28
ykarelomg that scenario010 job is voting and not in promotion criteria19:29
ykarelthings started happening just after promotion19:29
mwhahahaplz switch to nonvoting19:29
ykareland it's not even running in promotion pipeline19:29
weshay2019-10-02 07:21:22 | TASK [Configure octavia on overcloud] ******************************************19:30
weshayawesomeness19:30
* weshay gets on it19:30
ykarelack19:30
*** ykarel is now known as ykarel|away19:30
openstackgerritwes hayutin proposed openstack/tripleo-ci master: move standalone-scenario010 to non-voting  https://review.opendev.org/68625819:33
weshayhttps://review.opendev.org/#/c/652089/19:34
ykarel|awayweshay, good to change at job definition level19:35
ykarel|awayline 52119:35
ykarel|awayas i see that job is used in tripleo-ansible as well19:35
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common stable/queens: NeutronPhysnetNUMANodesMapping parameter issue  https://review.opendev.org/68625919:36
dking_desktopmwhahaha: Would you have any tips on how I can troubleshoot this deploy?19:38
mwhahahadking_desktop: check nova/ironic logs19:38
mwhahahadking_desktop: looks like it's not provisioning for whatever reason19:38
dking_desktopShould the nodes be powered on?19:39
mwhahahait should do that itself unless there are ipmi erorr or something19:39
openstackgerritJames Slagle proposed openstack/tripleo-common master: Raise exception when no IPs found  https://review.opendev.org/68626119:40
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: scenario010 nonvoting  https://review.opendev.org/68626219:40
*** holser has quit IRC19:40
openstackgerritJames Slagle proposed openstack/tripleo-validations master: Fix Python 3 incomaptibility in tripleo-ansible-inventory  https://review.opendev.org/68626319:42
openstackgerritJames Slagle proposed openstack/tripleo-validations master: Fix Python 3 incompatibility in tripleo-ansible-inventory  https://review.opendev.org/68626319:42
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Remove podman_image*  https://review.opendev.org/68626519:45
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP: POC of podman_container  https://review.opendev.org/68245819:45
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP - Introduce tripleo-container-manage role  https://review.opendev.org/68619619:45
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Add podman_container_info  https://review.opendev.org/68603519:46
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP: POC of podman_container  https://review.opendev.org/68245819:46
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP - Introduce tripleo-container-manage role  https://review.opendev.org/68619619:46
*** ykarel|away has quit IRC19:46
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: Handle empty cert related hostvar info  https://review.opendev.org/68482619:48
openstackgerritMerged openstack/tripleo-quickstart-extras master: update centos-release openstack to rocky  https://review.opendev.org/68517719:56
openstackgerritwes hayutin proposed openstack/tripleo-ci master: move standalone-scenario010 to non-voting  https://review.opendev.org/68625819:56
openstackgerritwes hayutin proposed openstack/tripleo-ci master: move standalone-scenario010 to non-voting  https://review.opendev.org/68625819:57
weshayarxcruz|ruck, https://review.opendev.org/#/c/686262/19:57
*** ansmith_ has joined #tripleo19:58
mwhahahad0ugal: for tomorrow, can you tell me why https://review.opendev.org/#/c/686181/ breaks when running under mistral? Seems like it's not compatibile with being run under the mistral engine (see failures in containers-multinode)19:58
arxcruz|ruckweshay: https://review.rdoproject.org/r/#/c/22801/19:58
*** ansmith has quit IRC20:00
*** holser has joined #tripleo20:00
weshayarxcruz|ruck, https://trunk-primary.rdoproject.org/api-centos-queens/api/civotes_detail.html?commit_hash=e2208117a31eea79c50330333501b6581f9faafe&distro_hash=15a9b671b405f92b4fc5f2f8b12ae92dfe136fd820:01
arxcruz|ruckweshay: https://trunk-primary.rdoproject.org/api-centos-stein/api/civotes_detail.html?commit_hash=a361c5edd5734320f781e10358a602f4ba57fc04&distro_hash=647b08e47a72f7533142c09074f227628f08f9fa20:03
triple-oh-noobHi. So, trying to add an additional (compute) node. When I perform introspection, I get: "Can not transition from state 'starting' on event 'start' (no defined transition)20:07
triple-oh-noobI have defined the node's "instack" information succefully.20:07
openstackgerritMerged openstack/tripleo-quickstart-extras master: ensure periodic job requirements are visible  https://review.opendev.org/68518420:09
openstackgerritMerged openstack/tripleo-quickstart master: Added release file for train  https://review.opendev.org/68393220:09
openstackgerritMerged openstack/os-refresh-config stable/stein: Update UPPER_CONSTRAINTS_FILE for stable/stein  https://review.opendev.org/65375820:09
*** aakarsh has joined #tripleo20:09
*** cylopez has quit IRC20:12
arxcruz|ruckweshay: https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-upgrade-stein20:14
*** ekultails has quit IRC20:16
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: [WIP] Enable external public endpoint for MetricsQdr  https://review.opendev.org/68628220:24
openstackgerritMartin Mágr proposed openstack/puppet-tripleo master: [WIP] Enable metrics qdr to use public VIP  https://review.opendev.org/68628720:30
dking_desktopmwhahaha: I see that the process is progress for Heat resources Controller and Compute, and nova had sent them to the scheduler. There are log entries in nova logs for "Unknown column 'resources' in 'field list'"20:32
* mwhahaha shrugs20:32
*** gbarros has quit IRC20:32
dking_desktop...and there, it just finally failed for timeout creating the ResourceGroup.20:33
mwhahahacheck the ironic logs20:34
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add prometheus port parameter  https://review.opendev.org/68546520:40
triple-oh-noobAnyone any ideas? I'm just trying to launch introspect on a new server to add. Nothing much in the ironic logs other than this.20:41
*** paramite has quit IRC20:42
weshaymwhahaha,  https://review.opendev.org/#/c/685777/ is that role run w/ a playbook w/ hosts: all20:43
weshay?20:43
mwhahahalikely20:44
mwhahahait's in the output20:44
mwhahahait's run in the container tho20:44
mwhahahaer no it's not20:45
weshayerror's on20:45
weshayremove /var/lib/containers/storage/overlay/9cdd774a37289bdfa7e58dfba15dd357cfbf17f4e630ba5e45fe34bdbbe53a2a/merged: device or resource busy20:45
weshaywarning on:20:45
mwhahahayea it doesn't force it20:45
weshayWARNING]: provided hosts list is empty, only localhost is available. Note\\nthat the implicit localhost does not match 'all20:45
mwhahahahosts: localhost20:45
weshay    "tripleo_common.image.exception.ImageUploaderException: Modifying image 192.168.24.1:8787/tripleomaster/rhel-binary-swift-proxy-server:968c9526ff2eeafaa13207a791f61173db604fff_9e598812 failed"20:45
mwhahahathat's something else?20:46
mwhahahalink to logs?20:46
dking_desktoptriple-oh-noob: I'm still new, myself, but I imagine that it would be helpful for you to provide your introspect rules, and/or maybe your instackenv.json? Maybe the specific commands you are running, as you can run introspection through several commands.20:46
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Add missing role name to Host prep steps task  https://review.opendev.org/68629320:46
weshaymwhahaha, http://logs.rdoproject.org/93/22793/3/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0ce4bae/logs/undercloud/home/zuul/install-undercloud.log.txt.gz20:47
weshaymwhahaha, rhel-820:47
weshayundercloud install20:47
mwhahahaof course, we can't have consistent interactions between versions20:47
mwhahahawe can slap an ignore errors on that tasks20:48
mwhahahawe want to try and remove the buildah container, those kinda odd that it's conflicting there20:49
triple-oh-noobdking_desktop: I have used the same format and detail types for the other servers successfully, I think there's something a bit "weird".20:49
weshayarxcruz|ruck, check the tmux20:50
weshayhad to cherrypick the change onto it..20:50
weshayarxcruz|ruck, updates are turned off20:50
weshaybut the service is running20:50
mwhahahawonder if we need to retry the container remove because of parallel processing20:51
triple-oh-noob.. or there's something over ipmi that's not working... Wondering if anyone can shed light on what the error is suggesting...20:51
weshaymwhahaha, if you want to test a change.. just drop a depends on https://review.rdoproject.org/r/#/c/22793/20:55
mwhahahado you mean update that change to dependon a patch?20:55
weshayaye20:56
weshayarxcruz|ruck, we need a bug on that probably20:56
arxcruz|ruckweshay: sorry, on which one?20:57
weshayarxcruz|ruck, https://review.opendev.org/#/c/685777/2/tasks/yum_update_buildah.yml20:57
*** paramite has joined #tripleo20:57
weshayarxcruz|ruck, http://logs.rdoproject.org/93/22793/3/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0ce4bae/logs/undercloud/home/zuul/install-undercloud.log.txt.gz20:57
arxcruz|ruckweshay: ack20:58
weshayarxcruz|ruck, cherry picked https://review.rdoproject.org/r/#/c/22803/20:59
openstackgerritAlex Schultz proposed openstack/ansible-role-tripleo-modify-image master: Add retires/ignore errors for removing buildah containers  https://review.opendev.org/68629420:59
mwhahahaweshay: -^20:59
weshayarxcruz|ruck, so we should see rhel8 master and stein centos7 promote20:59
weshaymwhahaha, cool.. thanks20:59
arxcruz|ruckweshay: https://bugs.launchpad.net/tripleo/+bug/184641321:00
openstackLaunchpad bug 1846413 in tripleo "RHEL-8 ovb job failing in tripleo-modify-image : Cleanup working container" [High,Triaged]21:00
weshaymwhahaha, throws tables21:00
weshayI kid21:00
weshaynow ur just causing timeouts ;)21:01
* weshay channels mwhahaha 21:01
mwhahahapffft rhel8 who cares about that21:01
weshay:)21:01
weshayf.. rhel8 I'm all about the stream21:01
mwhahahai'm kinda over this whole thing21:01
mwhahahai was over containers before, now i'm really over them21:01
mwhahahaworst idea ever21:01
mwhahahathis only cements it more and more21:02
*** Goneri has quit IRC21:02
weshayarxcruz|ruck, promoter Promoting the container images for dlrn hash 968c9526ff2eeafaa13207a791f61173db604fff on master to current-tripleo21:02
arxcruz|ruckweshay: i saw :)21:02
arxcruz|ruck\o/21:02
weshayarxcruz|ruck, I'll watch it from here.. thanks for the help.. tomorrow make sure stein and rhel8 master have gone through21:03
weshayarxcruz|ruck, i'll continue on queens21:03
*** aakarsh has quit IRC21:03
weshayarxcruz|ruck, oh.. just the bug.. so mwhahaha can note it in the commit message21:04
weshayarxcruz|ruck, mwhahaha get's paid by the patch / bug21:04
arxcruz|ruckweshay: https://bugs.launchpad.net/tripleo/+bug/184641321:04
openstackLaunchpad bug 1846413 in tripleo "RHEL-8 ovb job failing in tripleo-modify-image : Cleanup working container" [High,Triaged]21:04
weshayrock on king kong21:04
*** tosky_ has joined #tripleo21:05
openstackgerritAlex Schultz proposed openstack/ansible-role-tripleo-modify-image master: Add retires/ignore errors for removing buildah containers  https://review.opendev.org/68629421:05
*** tosky has quit IRC21:06
*** xek has quit IRC21:08
*** tosky_ is now known as tosky21:08
*** ansmith_ has quit IRC21:09
*** rakhmerov has quit IRC21:09
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Remove outdated FIXME's  https://review.opendev.org/68629921:20
*** aakarsh has joined #tripleo21:22
*** jfrancoa has quit IRC21:27
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Use single quotes for --start-at-task debug tasks  https://review.opendev.org/68630921:47
*** raildo has quit IRC21:47
*** ansmith_ has joined #tripleo21:48
openstackgerritMerged openstack/tripleo-ansible master: scenario010 nonvoting  https://review.opendev.org/68626221:48
*** holser has quit IRC21:56
*** slagle_ is now known as slagle21:58
*** rfolco|dentist is now known as rfolco22:17
*** pkopec has quit IRC22:17
*** paramite has quit IRC22:18
*** mcornea has quit IRC22:35
*** slaweq has quit IRC22:35
weshaymwhahaha, that resolved the issue on rhel822:47
mwhahahaK22:47
*** rcernin has joined #tripleo22:48
*** tkajinam has joined #tripleo22:54
weshayhrm.. container-prep time is way down :)) http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-6h&to=now23:07
*** tosky has quit IRC23:16
*** cdearborn has quit IRC23:23
*** panda is now known as panda|off23:39
*** gbarros has joined #tripleo23:42
*** gbarros has quit IRC23:55
« Tuesday, 2019-10-01 Index Thursday, 2019-10-03 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!