| *** mrunge has quit IRC | 00:15 | |
| *** spsurya has joined #tripleo | 00:16 | |
| *** mrunge has joined #tripleo | 00:19 | |
| *** mrunge_ has joined #tripleo | 00:24 | |
| *** mrunge has quit IRC | 00:24 | |
| *** vinaykns has quit IRC | 00:32 | |
| *** atarlov has quit IRC | 00:33 | |
| *** rh-jelabarre has joined #tripleo | 00:52 | |
| *** atarlov has joined #tripleo | 01:06 | |
| *** mrsoul__ has joined #tripleo | 01:14 | |
| *** openstack has joined #tripleo | 01:17 | |
| *** ChanServ sets mode: +o openstack | 01:17 | |
| *** mrsoul_ has quit IRC | 01:18 | |
| *** eck` is now known as eck`gone | 01:30 | |
| *** fzdarsky_ has joined #tripleo | 01:30 | |
| *** fzdarsky__ has quit IRC | 01:33 | |
| openstackgerrit | Nguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images https://review.openstack.org/542708 | 01:42 |
|---|---|---|
| *** ramishra has joined #tripleo | 01:54 | |
| *** atarlov has quit IRC | 02:04 | |
| *** bkopilov has quit IRC | 02:05 | |
| *** dxiri has joined #tripleo | 02:21 | |
| *** atarlov has joined #tripleo | 02:21 | |
| *** atarlov has quit IRC | 02:25 | |
| *** dxiri has quit IRC | 02:27 | |
| openstackgerrit | Vu Cong Tuan proposed openstack-infra/tripleo-ci master: Add py36 testenv https://review.openstack.org/577689 | 02:31 |
| *** psahoo has joined #tripleo | 02:36 | |
| *** rh-jelabarre has quit IRC | 02:37 | |
| *** psachin` has joined #tripleo | 02:43 | |
| *** jaganathan has joined #tripleo | 02:51 | |
| *** psachin` has quit IRC | 02:53 | |
| *** zshi has joined #tripleo | 02:54 | |
| *** atarlov has joined #tripleo | 02:57 | |
| *** skramaja has joined #tripleo | 02:57 | |
| *** atarlov has quit IRC | 03:01 | |
| *** slaweq has joined #tripleo | 03:11 | |
| *** vinaykns has joined #tripleo | 03:13 | |
| *** slaweq has quit IRC | 03:16 | |
| *** dxiri has joined #tripleo | 03:20 | |
| *** bkopilov has joined #tripleo | 03:23 | |
| *** dxiri has quit IRC | 03:26 | |
| *** dxiri has joined #tripleo | 03:35 | |
| *** dxiri has quit IRC | 03:41 | |
| *** atarlov has joined #tripleo | 03:41 | |
| *** dxiri has joined #tripleo | 03:41 | |
| *** udesale has joined #tripleo | 03:46 | |
| *** tcw has quit IRC | 03:50 | |
| *** pdeore has joined #tripleo | 03:52 | |
| *** vinaykns has quit IRC | 03:53 | |
| *** pdeore has quit IRC | 03:54 | |
| *** zshi has quit IRC | 03:56 | |
| *** tcw has joined #tripleo | 03:58 | |
| *** dxiri has quit IRC | 04:08 | |
| *** atarlov has quit IRC | 04:14 | |
| *** cshastri has joined #tripleo | 04:15 | |
| *** d0ugal has quit IRC | 04:15 | |
| *** d0ugal_ has joined #tripleo | 04:15 | |
| *** ykarel has joined #tripleo | 04:21 | |
| *** psachin` has joined #tripleo | 04:35 | |
| *** mhenkel_ has joined #tripleo | 04:36 | |
| *** noama has joined #tripleo | 04:41 | |
| *** mhenkel_ has quit IRC | 04:41 | |
| *** nyechiel_ has quit IRC | 04:43 | |
| Tengu | hello there :) | 04:48 |
| *** pdeore has joined #tripleo | 04:55 | |
| *** yprokule has joined #tripleo | 04:56 | |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Add networking-ansible ML2 plugin support https://review.openstack.org/577620 | 04:58 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-specs master: [WIP] Spec for improved privilege escalation in py-scripts https://review.openstack.org/572761 | 05:07 |
| *** nguyenhai has quit IRC | 05:07 | |
| *** janki has joined #tripleo | 05:15 | |
| *** nguyenhai has joined #tripleo | 05:15 | |
| *** atarlov has joined #tripleo | 05:17 | |
| *** atarlov has quit IRC | 05:21 | |
| *** waleedm has joined #tripleo | 05:26 | |
| *** aufi has joined #tripleo | 05:31 | |
| *** atarlov has joined #tripleo | 05:34 | |
| *** atarlov has quit IRC | 05:35 | |
| *** hamzy_ has quit IRC | 05:36 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-heat-templates master: Fix syntax for set_fact ansible task. https://review.openstack.org/577725 | 05:37 |
| *** hamzy has joined #tripleo | 05:38 | |
| *** atarlov has joined #tripleo | 05:40 | |
| *** hamzy has quit IRC | 05:40 | |
| *** hamzy has joined #tripleo | 05:42 | |
| *** ratailor has joined #tripleo | 05:45 | |
| *** waleedm has quit IRC | 05:46 | |
| *** waleedm has joined #tripleo | 05:47 | |
| *** atarlov has quit IRC | 05:47 | |
| *** atarlov has joined #tripleo | 05:48 | |
| quiquell|off | Good morning, now the qpid problem is in the promotions | 05:49 |
| *** quiquell|off is now known as quiquell|rover | 05:51 | |
| *** atarlov has quit IRC | 05:52 | |
| janki | marios, hey | 05:52 |
| *** yolanda has joined #tripleo | 05:53 | |
| *** moshele has joined #tripleo | 05:55 | |
| *** ksambor has joined #tripleo | 05:55 | |
| marios | o/ janki | 05:59 |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart master: Revert "Switch more promotion jobs to containerized undercloud" https://review.openstack.org/577727 | 06:00 |
| janki | marios, hey. How are you? | 06:00 |
| *** bogdando has joined #tripleo | 06:00 | |
| *** yprokule has quit IRC | 06:01 | |
| *** yprokule has joined #tripleo | 06:01 | |
| marios | no bad thanks janki and u? | 06:05 |
| *** dparkes has quit IRC | 06:07 | |
| janki | marios, good. thank you :). I was wondering where would the post_update_tasks logs would be. | 06:08 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778040 | 06:10 |
| openstack | Launchpad bug 1778040 in tripleo "Error at overcloud_prep_containers Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", " Requires: libqpid-proton.so.10()(64bit)" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| *** florianf has joined #tripleo | 06:10 | |
| *** mrsoul__ is now known as mschuppert | 06:10 | |
| *** nyechiel_ has joined #tripleo | 06:10 | |
| marios | janki: what are you looking for/are you looking at some particular job or local? so assuming pike/queens, those are executed by ansible (via tripleoclient) and there is a mistral execution involved to actually call ansible. | 06:13 |
| marios | janki: so 'the undercloud' for mistral/ansible | 06:13 |
| marios | janki: and on the particular node journal for checking some particular task for example but depends what you looking for | 06:13 |
| *** zshi has joined #tripleo | 06:14 | |
| janki | marios, I am suspect few of ODL-OVS tasks are not running. And didnot find anything in log file I saved by running openstack stack update run --nodes COntroller >> update_run.log. I also checked mistral workflow-list | grep update, got the UUID for tripleo.package_update.v1.update_nodes workflow but no corresponding folder in /var/lib/mistral | 06:15 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Allow custom host kernel params for libguestfs https://review.openstack.org/576772 | 06:15 |
| *** jcoufal has joined #tripleo | 06:16 | |
| marios | janki: did you get the playbook and check that (I mean like openstack overcloud config download) to see what the exact tasks collected are? | 06:16 |
| *** jcoufal has quit IRC | 06:16 | |
| janki | marios, no. let me do that | 06:16 |
| *** lvdombrkr has joined #tripleo | 06:17 | |
| *** jcoufal has joined #tripleo | 06:17 | |
| *** kopecmartin has joined #tripleo | 06:17 | |
| janki | marios, config download shows post_update_tasks playbook with the task included. But i believe the task is not run as the cahnges havenot been taken effect on controller nodes | 06:19 |
| marios | janki: ack well if the tasks themselves look good too, then i guess the next step would be to try and follow the playbook... so you can see in the logs of the node all those tasks being executed | 06:21 |
| marios | janki: i mean the update_tasks_playbook and post_update_tasks_playbook | 06:21 |
| marios | janki: and probably worth filing a bug depending where you get with that with all the info you have at that point | 06:22 |
| marios | so you don't have to explain every time :) | 06:22 |
| janki | marios, I am looking for this task https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/opendaylight-api.yaml#L229. This disables a flag on controller node in a file. Now when I check that file on controller node, the upgrade flag is still true which was set by "set" task in update_tasks | 06:23 |
| *** dciabrin has joined #tripleo | 06:27 | |
| *** mrunge_ is now known as mrunge | 06:27 | |
| quiquell|rover | Good morning all | 06:28 |
| marios | janki: did you check the odl_update_level | 06:28 |
| quiquell|rover | Do we know if pike really needs opstools ? | 06:28 |
| marios | janki: looks like you're setting that as a fact | 06:28 |
| janki | marios, yes. its 2 | 06:29 |
| marios | quiquell|rover: o/ hello i don't know sorry | 06:29 |
| janki | marios, and all update_tasks are executed as per odl_update_level = 2 | 06:29 |
| *** agurenko has joined #tripleo | 06:29 | |
| *** agurenko has quit IRC | 06:29 | |
| marios | janki: the /var/lib file exists and all the right permissions? | 06:29 |
| janki | marios, yes. the task is exactly the same as update_task just changing a value in post_update_tasks. | 06:30 |
| *** fzdarsky_ has quit IRC | 06:31 | |
| *** jbadiapa has joined #tripleo | 06:35 | |
| *** lvdombrkr89 has joined #tripleo | 06:37 | |
| *** lvdombrkr has quit IRC | 06:39 | |
| *** ffiore has joined #tripleo | 06:40 | |
| *** psahoo has quit IRC | 06:41 | |
| *** jtomasek has joined #tripleo | 06:43 | |
| *** dparkes has joined #tripleo | 06:46 | |
| *** quiquell|rover is now known as quique|rover|afk | 06:47 | |
| *** lifeless has quit IRC | 06:49 | |
| *** cylopez has joined #tripleo | 06:50 | |
| *** cylopez has left #tripleo | 06:51 | |
| quique|rover|afk | bogdando: Trying to reproduce https://bugs.launchpad.net/tripleo/+bug/1777939 here too | 06:52 |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 06:52 |
| quique|rover|afk | Have to leave know | 06:52 |
| bogdando | quique|rover|afk: I have en env already | 06:52 |
| quique|rover|afk | bogdando: nice, btw I am hitting different issue now... docker pull failed: Error: image tripleomaster/centos-binary-rsyslog-base:a51b4b4e84c1a38c0f6806286049c9396dc9940c_033c532c not found | 06:54 |
| quique|rover|afk | pufff :-( new week new blocker | 06:54 |
| bogdando | quique|rover|afk: :D | 06:54 |
| quique|rover|afk | Going to leave know will go back in a few | 06:54 |
| bogdando | I deployed on Friday, lucky me | 06:54 |
| quique|rover|afk | bogdando: Yep | 06:54 |
| *** psahoo has joined #tripleo | 06:55 | |
| *** pcaruana has joined #tripleo | 07:07 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778040 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 07:10 |
| openstack | Launchpad bug 1778040 in tripleo "Error at overcloud_prep_containers Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", " Requires: libqpid-proton.so.10()(64bit)" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 07:10 |
| janki | marios, which file/line in config-download folder says "execute post_updates_steps_playbook"? there is no deploy-steps.yaml (deploy-steps.j2 converted) file anywhere in the system. | 07:10 |
| *** ksambor has quit IRC | 07:12 | |
| *** zoli|gone is now known as zoli | 07:13 | |
| *** paramite has joined #tripleo | 07:13 | |
| *** ksambor has joined #tripleo | 07:14 | |
| marios | janki: that would be in the client sec | 07:15 |
| bogdando | quique|rover|afk: jfyi, used the connection tester https://blog.sleeplessbeastie.eu/2017/07/10/how-to-check-connection-to-the-rabbitmq-message-broker/ and it fails with the guest:password generated in configs and tripleo passwords file | 07:16 |
| bogdando | so seems like something is broken in DF | 07:17 |
| marios | janki: ah so maybe we aren't calling them :) where were the post_update_tasks added looks like they were never run/ or something changed in the client recently | 07:18 |
| marios | jistr|off: ^^^ fyi | 07:18 |
| marios | janki: sec, here are links (but you probably want to file a bug for it too cos it looks like it is one) | 07:18 |
| janki | marios, I didnt find a playbook/task that triggers update_tasks too | 07:19 |
| janki | marios, but those tasks are actually run. I can confirm from the logs | 07:19 |
| janki | marios, where were the post_update_tasks added looks like they were never run/ or something changed in the client recently - where should that be ideally? tripleoclient or mistralclient? | 07:20 |
| marios | janki: yeah. so here are the update tasks executed in the client https://github.com/openstack/python-tripleoclient/blob/3726c7750f650de83546dd0bfa8f1c7a38d6dc8f/tripleoclient/v1/overcloud_update.py#L170 | 07:20 |
| marios | janki: and where that playbooks is defined here https://github.com/openstack/python-tripleoclient/blob/3726c7750f650de83546dd0bfa8f1c7a38d6dc8f/tripleoclient/constants.py#L50 | 07:20 |
| *** tesseract has joined #tripleo | 07:20 | |
| marios | janki: now the deploy steps are included in the update playbook in the tht here https://github.com/openstack/tripleo-heat-templates/blob/e1a16a4903c935611cd0ee8ac36ced4a8a97296d/common/deploy-steps.j2#L589 | 07:21 |
| marios | janki: so the fix will be to add the post_update_tasks into the client constant | 07:21 |
| marios | janki: so it gets executed in https://github.com/openstack/python-tripleoclient/blob/3726c7750f650de83546dd0bfa8f1c7a38d6dc8f/tripleoclient/utils.py#L955 | 07:22 |
| marios | janki: so looks to me that the post_update_tasks were never executed | 07:22 |
| marios | janki: i mean, that never worked. or something changed recently | 07:22 |
| marios | janki: afaics only odl defines/uses them | 07:23 |
| janki | marios, or the logic were never ever added to execute them. only ODL has post_udpate_tasks | 07:23 |
| marios | janki: right we're saying the same thing. the post_update_tasks were never executed | 07:24 |
| janki | marios, the recent change that jistr|off added was to NOT run deploy_tasks as part of update as they are run after update as separate task. | 07:24 |
| janki | marios, that bug was discovered as part of ODL update testing so I know | 07:24 |
| janki | marios, I will file a bug and push a patch | 07:25 |
| *** psahoo has quit IRC | 07:26 | |
| janki | marios, how to build tripleoclient locally? I tried python setup.py build but that didnt work | 07:27 |
| bogdando | quique|rover|afk: though I'm not sure it's really the bad password, some services were able to connect according to the rabbit logs... we need someone to debug it in openstack services :/ | 07:28 |
| marios | janki: whats the problem with python setup.py install? see notes at http://tripleo.org/install/developer/upgrades/major_upgrade.html#making-changes-to-the-upgrades-workflow might help | 07:29 |
| openstackgerrit | Damian Szeluga proposed openstack/tripleo-heat-templates stable/queens: Adding HeatEngineVolumes and HeatEngineOptEnvVars support https://review.openstack.org/577737 | 07:29 |
| *** quique|rover|afk is now known as quiquell|rover | 07:30 | |
| marios | janki: landed that into the q upgrade docs but is relevant if you go trying to fix this in the client fyi | 07:30 |
| marios | janki: (dev docs) | 07:30 |
| janki | marios, I dont remember the exact error. But it failed for some dependencies. I will clone queens and try it | 07:30 |
| janki | marios, yes I figured that for dev docs | 07:30 |
| marios | janki: well you don't have to clone queens if you want to develop you can use master. that is just example for the queens upgrade dev docs | 07:31 |
| quiquell|rover | bogdando: What do you mean openstack services ? | 07:31 |
| marios | janki: depends what you're testing/fixing | 07:31 |
| marios | janki: but imo makes sense to fix this on master first and backport it | 07:31 |
| bogdando | quiquell|rover: to confirm if they had been able to connect for a while, then degraded somehow | 07:31 |
| janki | marios, I want to test the whole update process and my setup is queens so I will be cloning queens. Last time, I cloned master on queens setup and so might be the error | 07:32 |
| bogdando | cuz I can see ther is a lot of exchanges, queues et al, so it had been working using the same creds for some of the openstack services | 07:32 |
| quiquell|rover | bogdando: Maybe some services have different credentials | 07:33 |
| quiquell|rover | gate blocker ! | 07:33 |
| quiquell|rover | docker pull failed: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo: received unexpected HTTP status: 503 Service Unavailable | 07:33 |
| quiquell|rover | pufff | 07:33 |
| bogdando | quiquell|rover: no, those all have the same transport url configured | 07:33 |
| Tengu | guess it's time to fire up some kind of local registry :). | 07:35 |
| *** saneax has joined #tripleo | 07:35 | |
| quiquell|rover | here we go https://bugs.launchpad.net/tripleo/+bug/1778472 | 07:36 |
| openstack | Launchpad bug 1778472 in tripleo "docker pull failed: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo: received unexpected HTTP status: 503 Service Unavailable" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 07:36 |
| *** hjensas has quit IRC | 07:37 | |
| *** psahoo has joined #tripleo | 07:39 | |
| *** jpena|off is now known as jpena | 07:41 | |
| *** amoralej|off is now known as amoralej | 07:42 | |
| bogdando | quiquell|rover: https://bugs.launchpad.net/tripleo/+bug/1777939/comments/8 | 07:45 |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 07:45 |
| quiquell|rover | bogdando: Maybe credentials get changed later on in the process | 07:49 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Streamline variables passed in different environments https://review.openstack.org/573819 | 07:49 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/queens: Pass stack's name with variable overcloud_stack_name https://review.openstack.org/577617 | 07:49 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/queens: Load roles list from yaml instead of awk parsing https://review.openstack.org/577618 | 07:49 |
| bogdando | um, I cannot imagine how come it would change... | 07:49 |
| bogdando | quiquell|rover: that's just an undercloud install, one time, no upgrades or updates, right? | 07:50 |
| bogdando | at least according to the job logs | 07:50 |
| quiquell|rover | bogdando: Yep, | 07:50 |
| quiquell|rover | bogdando: =ERROR REPORT==== 22-Jun-2018::15:09:48 === | 07:51 |
| quiquell|rover | Error on AMQP connection <0.16466.1> (192.168.24.1:55330 -> 192.168.24.1:5672, state: starting): | 07:51 |
| quiquell|rover | AMQPLAIN login refused: user 'guest' - invalid credentials | 07:51 |
| quiquell|rover | And also | 07:52 |
| quiquell|rover | =ERROR REPORT==== 22-Jun-2018::14:32:50 === | 07:52 |
| quiquell|rover | ** Connection attempt from disallowed node 'rabbitmq-cli-58@undercloud' ** | 07:53 |
| quiquell|rover | Then I see a rabbitmq restart | 07:53 |
| quiquell|rover | bogdando: In fact the state is "starting" when guest fails | 07:53 |
| bogdando | not sure how to confirm that from logs, but it must be something to the rabbit state changes | 07:54 |
| bogdando | caused that degradation | 07:54 |
| quiquell|rover | bogdando: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/var/log/containers/rabbitmq/rabbit@undercloud.log.txt.gz | 07:54 |
| bogdando | yea, those tell me nothing but 'weird, hmm' | 07:55 |
| quiquell|rover | bogdando: Yep :-), I am starting to grasp this thing... Going to check with a passing one | 07:55 |
| *** jpich has joined #tripleo | 07:55 | |
| bogdando | did the rabbit ban the node? :D | 07:56 |
| quiquell|rover | bogdando: Feels like rabbimq is restarting with different config | 07:56 |
| bogdando | oh, good idea, there is a command to get its runtime config... | 07:56 |
| quiquell|rover | bogdando: Let me compare the two logs the the good and the bad (there is no ugly though) | 07:57 |
| quiquell|rover | bogdando: No restart in a good one https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/033c6ec/undercloud/var/log/rabbitmq/rabbit@undercloud.log.txt.gz | 07:57 |
| *** pgadiya has joined #tripleo | 07:59 | |
| ykarel | bogdando, quiquell|rover this is happening because of undercloud_check_idempotency | 07:59 |
| *** pgadiya has quit IRC | 07:59 | |
| ykarel | in fs002 | 07:59 |
| ykarel | so config | 08:00 |
| Tengu | dtantsur|afk: heya! please ping me when you're online - I'd like to get more info on your comment on https://review.openstack.org/573196 ;) | 08:00 |
| ykarel | quiquell|rover, bogdando looks like guest password in not updated in second run while config passwords are updated | 08:00 |
| *** radeks has quit IRC | 08:00 | |
| * Tengu sees the end of the mail pile | 08:01 | |
| bogdando | ykarel: oh | 08:01 |
| quiquell|rover | ykarel: Good one, have been there for long in the fs | 08:01 |
| bogdando | so DF bug indeed | 08:01 |
| bogdando | ykarel: thanks for the insight! | 08:01 |
| ykarel | bogdando, and should be happeing after switch to containerized undercloud | 08:02 |
| ykarel | bogdando, testing here:- https://review.openstack.org/#/c/577727/ | 08:02 |
| ykarel | hmm but fs002 is not running | 08:02 |
| quiquell|rover | ykarel: What's the relation with containers tehre ? | 08:02 |
| ykarel | quiquell|rover, both fs002 and fs020 are failing after this | 08:03 |
| quiquell|rover | ykarel: Do a change at TQE with a Depends-On to this, fs002 will run | 08:03 |
| bogdando | ykarel: no, the generated password matches the ones in the services configs | 08:03 |
| bogdando | and the password files have not been touched for the 2nd run | 08:04 |
| ykarel | bogdando, is the same password set in rabbitmq | 08:04 |
| bogdando | both passwords files have timestamps for the end of the 1st run | 08:04 |
| bogdando | ykarel: yes, it has the same | 08:05 |
| quiquell|rover | ykarel: Do we need to fix containerize more than deactivate it ? | 08:05 |
| bogdando | prolly somethingbad happening to the rabbit state | 08:05 |
| bogdando | please stop deactivating it | 08:05 |
| ykarel | quiquell|rover, definetely containers needs to be fixed | 08:05 |
| bogdando | it's gonna be default in rocky, we need to start facing containers world :) | 08:05 |
| ykarel | but side by side | 08:05 |
| ykarel | as we are 10days behind master | 08:06 |
| quiquell|rover | Humm Handler for GET /v1.26/containers/rabbitmq_bootstrap/json returned error: No such container: rabbitmq_bootstrap" | 08:06 |
| quiquell|rover | Then it tryies to re-recreawte the containers | 08:06 |
| quiquell|rover | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/var/log/journal.txt.gz#_Jun_22_14_33_13 | 08:06 |
| bogdando | as you can see, configs look correct for the idempotency check, there should be something else to figure out for rabbit restarted | 08:07 |
| *** suuuper has joined #tripleo | 08:07 | |
| * ykarel looks | 08:08 | |
| quiquell|rover | bogdando, ykarel: The first problem at rabbitmq is https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/var/log/journal.txt.gz#_Jun_22_14_33_13 | 08:08 |
| quiquell|rover | sorry | 08:08 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 08:10 |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778040 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778472 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| quiquell|rover | Just after this "** Connection attempt from disallowed node 'rabbitmq-cli-58@undercloud' **" | 08:10 |
| openstack | Launchpad bug 1778040 in tripleo "Error at overcloud_prep_containers Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", " Requires: libqpid-proton.so.10()(64bit)" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 08:10 |
| openstack | Launchpad bug 1778472 in tripleo "docker pull failed: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo: received unexpected HTTP status: 503 Service Unavailable" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 08:10 |
| quiquell|rover | Just after this ** Connection attempt from disallowed node 'rabbitmq-cli-58@undercloud' ** | 08:10 |
| ykarel | bogdando, re. and the password files have not been touched for the 2nd run, where are the password files | 08:10 |
| quiquell|rover | we get the restart | 08:10 |
| bogdando | quiquell|rover: those no such container foo are junk, ignore it :) | 08:11 |
| *** ccamacho has joined #tripleo | 08:11 | |
| quiquell|rover | bogdando: What do you mean ? | 08:11 |
| bogdando | ykarel: ~/tripleo-undercloud-passwords.yaml and undercloud-passwords.conf | 08:12 |
| bogdando | ykarel: created/updated at the end of installation | 08:12 |
| bogdando | but let me check, not sure really | 08:12 |
| bogdando | at the beginning of install, ykarel | 08:13 |
| *** jistr|off is now known as jistr | 08:14 | |
| bogdando | so I've mistaken, the timestamp points to the beginning of the 2nd run | 08:14 |
| bogdando | anyway, rabbit config and service config matches | 08:14 |
| *** leanderthal has joined #tripleo | 08:14 | |
| *** yprokule has quit IRC | 08:14 | |
| bogdando | quiquell|rover: I mean those mean snothing | 08:14 |
| ykarel | bogdando, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/home/jenkins/undercloud-ansible-I4eekS/Undercloud/undercloud/UndercloudDeployment.gz and https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/home/jenkins/undercloud-ansible-I4eekS/Undercloud/undercloud/ | 08:15 |
| ykarel | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/home/jenkins/undercloud-ansible-I4eekS/Undercloud/undercloud/UndercloudDeployment.gz | 08:15 |
| ykarel | damn same link | 08:15 |
| ykarel | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/home/jenkins/undercloud-ansible-Qo_GtC/Undercloud/undercloud/UndercloudDeployment.gz abd https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/home/jenkins/undercloud-ansible-I4eekS/Undercloud/undercloud/UndercloudDeployment.gz | 08:16 |
| *** athomas has joined #tripleo | 08:16 | |
| *** d0ugal_ has quit IRC | 08:16 | |
| ykarel | bogdando, so in both run different passwords are there, and somehow guest password is not changed in the second run only the service configs | 08:16 |
| *** dbecker has joined #tripleo | 08:16 | |
| Tengu | does anyone have an non-containerized undercloud at hand? If so, care to share the location of the httpboot directory used by ironic for its pxe images? | 08:17 |
| bogdando | ykarel: could you please elaborate? can't see diff passwords set for rabbit config vs services configs | 08:17 |
| *** d0ugal has joined #tripleo | 08:17 | |
| *** d0ugal has quit IRC | 08:17 | |
| *** d0ugal has joined #tripleo | 08:17 | |
| quiquell|rover | bogdando: "state: starting" when guest is failing could be related ? | 08:19 |
| *** ykarel_ has joined #tripleo | 08:19 | |
| jistr | marios, janki: hi, just to clarify the recent change -- we were running deploy tasks twice (once rolling, 2nd time non-rolling). Now we only run them once (rolling) as we should. | 08:20 |
| jistr | it didn't touch the post_update_tasks / playbook execution | 08:20 |
| jistr | we probably never had that | 08:20 |
| jistr | i see they were introduced here https://github.com/openstack/tripleo-heat-templates/commit/98faacad44e39a456d9fe1a1d21f5a65e8de4fc1#diff-fdfa9108b5c67b2c4ce1dae2a05ec0c2 | 08:21 |
| marios | jistr: ack, yeah i did think of that (when i went loking for the deploy tasks in the update steps output from tht | 08:21 |
| *** ykarel_ is now known as ykarel|lunch | 08:21 | |
| jistr | but not sure how we executed them back then | 08:21 |
| marios | jistr: i mean the removal from client | 08:21 |
| marios | jistr: but i don't think they were ever executed | 08:21 |
| marios | (sounds like) | 08:21 |
| marios | via the client i mean | 08:21 |
| marios | and only odl is using them | 08:21 |
| jistr | yea | 08:21 |
| jistr | now the question is *how* should we execute them | 08:21 |
| marios | jistr: should be easy enough addition | 08:21 |
| marios | jistr: well could add them after the update playbook in the client? | 08:22 |
| marios | jistr: so update tasks, deploy steps and then these | 08:22 |
| *** ykarel has quit IRC | 08:22 | |
| jistr | marios: right that's one option, but then we get rolling update+deploy and then non-rolling post_update | 08:22 |
| marios | jistr: hm right. maybe it isn't a problem (non rolling?0 | 08:22 |
| jistr | if we wanted them rolling, we need to pull them into update playbook | 08:22 |
| marios | jistr: i mean again this is only odl so i'd have to go looking at those tasks | 08:23 |
| marios | jistr: /win 19 | 08:23 |
| jistr | marios: yea maybe not... just a decision that we need to make | 08:23 |
| jistr | in case we want them used also for non-ODL things | 08:23 |
| jistr | what's the approach that would make most sense | 08:23 |
| *** shardy has joined #tripleo | 08:24 | |
| bogdando | ykarel|lunch: so indeed, the password for the rabbit wasn't updated | 08:24 |
| marios | jistr: well the other way then is to include them in the tht playbook | 08:24 |
| bogdando | it seems we have rabbit configuration ignoring its config file | 08:24 |
| marios | jistr: after the deploy steps | 08:24 |
| *** chem has joined #tripleo | 08:24 | |
| quiquell|rover | bogdando: Where do you see that ? in the undercloud reinstall ? | 08:24 |
| *** Haresh has joined #tripleo | 08:24 | |
| bogdando | quiquell|rover: just rerun the tester connection script using the password from the 1st install run | 08:25 |
| bogdando | and it worked | 08:25 |
| quiquell|rover | bogdando: So it has the previous password... ok | 08:25 |
| bogdando | but the rabbit config has the password updated from the 2nd run, but this seems ignored | 08:25 |
| quiquell|rover | bogdando: Maybe there are some clue here https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/home/jenkins/undercloud_reinstall.log.txt.gz | 08:25 |
| jistr | marios: yea i'm almost thinking that should be the way, as i think essentially the assumption is that minor update can be run on any node at any time, and should be finished on node A by the time you do it on node B | 08:26 |
| *** arxcruz|off is now known as arxcruz | 08:26 | |
| jistr | so if we need post_update_tasks to finish the update, we should probably pull them into the update playbook, and get rid of the post_update_playbook completely | 08:26 |
| jistr | cc janki ^ | 08:26 |
| marios | jistr: sounds good | 08:27 |
| quiquell|rover | bogdando: Maybe wecond one change the password outside container, and that's the issue | 08:27 |
| bogdando | quiquell|rover: AFAIK, rabbit takes user password from the config file. Not sure if it updates it on restart, prolly not | 08:28 |
| bogdando | so we'll have to recreate the user it seems | 08:28 |
| bogdando | if so, this issue also affects overclouds | 08:28 |
| bogdando | do we test idempotency for overclouds? | 08:28 |
| janki | jistr, marios will that ensure this sequence update_tasks -> normal deploy_tasks -> post_update_tasks? we are unsetting a flag in post_update_tasks which should happen after the new ODL container is pulled | 08:29 |
| bogdando | both oc and uc install use the same rabbit service from tht | 08:29 |
| janki | jistr, marios which happens in step 1 of deploy stesp | 08:29 |
| quiquell|rover | bogdando: I see also some config overwritting here https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5be41b4/undercloud/var/log/journal.txt.gz#_Jun_22_14_33_13 | 08:29 |
| quiquell|rover | Maybe that's going back to defaults | 08:30 |
| *** derekh has joined #tripleo | 08:30 | |
| quiquell|rover | There are like two content changed for rabbitmq.conf | 08:30 |
| jistr | janki: yea. And if we run the update on multiple nodes (A, B), we probably want: update A, deploy A, post_update A, update B, deploy B, post_update B, right? | 08:30 |
| quiquell|rover | ÂShouldn't be one ? | 08:30 |
| marios | janki: yeah the result should be the same, jistr is propose we add the post steps into the generated update playbook in tht, in the same way that the deployment steps are included today (i pointed you to that earlier) | 08:30 |
| jistr | instead of update A, deploy A, update B, deploy B, post_update A+B at the same time | 08:31 |
| jistr | i think the former mentioned approach fits better the assumptions around minor update in general (not specific to ODL) | 08:31 |
| marios | janki: i.e. the advantage of doing it this way is that we can use serial 1 on the update steps playbook, so it all gets executed on one node at a time (what jistr just said ^^^) | 08:31 |
| jistr | right. The `serial: 1` would apply to all (update, deploy, post_update) | 08:32 |
| bogdando | quiquell|rover: the config contains the updated password for the guest user. That's all OK. The issue is the quess user was not updated for the restarted rabbitmq | 08:32 |
| bogdando | it's not enough to alter the config | 08:32 |
| quiquell|rover | bogdando: Ahh ok, it's still using the old password | 08:33 |
| bogdando | and as I said, this also affects overclouds rabbits IMHO | 08:33 |
| marios | janki: (from earlier fyi here is where jistr is proposing we add the post steps, example point to the deploy steps in the update playbook) | 08:33 |
| marios | 10:21 < marios> janki: now the deploy steps are included in the update playbook in the tht here https://github.com/openstack/tripleo-heat-templates/blob/e1a16a4903c935611cd0ee8ac36ced4a8a97296d/common/deploy-steps.j2#L589 | 08:33 |
| quiquell|rover | bogdando: If we fix it, we will fix oc | 08:33 |
| bogdando | they use the same tht template | 08:33 |
| bogdando | right | 08:33 |
| quiquell|rover | bogdando: Let's find where guest user password is set for the client | 08:33 |
| *** dxiri has joined #tripleo | 08:34 | |
| bogdando | quiquell|rover: yeah, should be in t-h-t init containers prolly, docker steps, given some puppet tags | 08:34 |
| bogdando | letme check... | 08:34 |
| quiquell|rover | bogdando: Going to check to learn at least | 08:34 |
| janki | marios, jistr yes. the serial:1 thing sounds more right. I have filed a bug - https://bugs.launchpad.net/tripleo/+bug/1778471. would any of you like to take that up? | 08:34 |
| openstack | Launchpad bug 1778471 in tripleo "post_update_tasks are never executed" [Undecided,Confirmed] - Assigned to Janki Chhatbar (jankihchhatbar) | 08:34 |
| jistr | janki: thanks, yup i'll take it | 08:34 |
| quiquell|rover | bogdando: RabbitPassword | 08:35 |
| quiquell|rover | at THT | 08:35 |
| quiquell|rover | Is the param for the heat templates | 08:36 |
| bogdando | quiquell|rover: http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/rabbitmq.yaml#n202 | 08:37 |
| bogdando | this should be updating the user password | 08:37 |
| bogdando | oh, it has http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/rabbitmq.yaml#n209 | 08:37 |
| quiquell|rover | bogdando: this ? http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/actions/parameters.py#n282 | 08:37 |
| bogdando | ro, and prolly does not allow to alter mnesia things? | 08:38 |
| bogdando | let me try it locally... | 08:38 |
| *** dxiri has quit IRC | 08:38 | |
| bogdando | quiquell|rover: you can ignore the passwords magic for now, it justworks, the issue is how we apply the password for a user | 08:38 |
| bogdando | I think mnesia can be written so nothing gets updated | 08:39 |
| bogdando | cannot | 08:39 |
| quiquell|rover | So the config is right but nmesia doesn't update it ? | 08:39 |
| bogdando | that's my guess, right | 08:39 |
| *** sshnaidm|off is now known as sshnaidm | 08:39 | |
| bogdando | we need a rw mount prolly | 08:40 |
| quiquell|rover | Let's find evidence | 08:40 |
| bogdando | so gonna test it locally | 08:40 |
| quiquell|rover | bogdando: rabbitmq is not also reinstalling nmesia ? | 08:40 |
| quiquell|rover | It's the only one using erlant | 08:40 |
| quiquell|rover | erlang | 08:40 |
| *** karthiks has joined #tripleo | 08:40 | |
| janki | jistr, ack. thanks | 08:40 |
| bogdando | no, we need to keep it intact in general | 08:40 |
| bogdando | just bind mount it the way it can write changes produced from puppet configs | 08:41 |
| quiquell|rover | bogdando: Maybe we do a nmesia backup before reinstall undercloud and we are still reading from this | 08:41 |
| bogdando | quiquell|rover: - /var/lib/rabbitmq:/var/lib/rabbitmq:ro | 08:42 |
| bogdando | it just bind mounts the host path, no backups | 08:42 |
| quiquell|rover | bogdando: ack | 08:42 |
| bogdando | considered for that place* | 08:42 |
| janki | jistr, 1 more question - I see passwords hidden in all of the services. eg:https://github.com/openstack/tripleo-heat-templates/blob/e64c10b9c13188f37e6f122475fe02280eaa6686/puppet/services/neutron-api.yaml#L44 Where are the generated passwords stored? | 08:42 |
| quiquell|rover | bogdando: We have to find a password update that fails | 08:43 |
| *** jaosorior has joined #tripleo | 08:43 | |
| janki | jistr, I want to know password/username to access some ODL APIs but dont want to expose them in THT | 08:43 |
| bogdando | quiquell|rover: this seem not written with the default rabbit logging levels | 08:43 |
| jistr | janki: that would be in Mistral plan environment i think | 08:43 |
| jistr | one sec | 08:43 |
| bogdando | so we can only try a fix and confirm the guess | 08:43 |
| jaosorior | jistr: doesn't mistral also end up exposing those in t-h-t? | 08:44 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Removing options in configure-tempest https://review.openstack.org/570159 | 08:45 |
| jistr | jaosorior: in t-h-t? It does store them in plan-environment.yaml IIRC and it does pass them to Heat for deployment, but i'm not sure what you meant by "expose in t-h-t" | 08:45 |
| jaosorior | janki: what do you mean by "but don't want to expose them in THT"? | 08:46 |
| quiquell|rover | bogdando: maybe rabbitmq infinite "state: starting" is beacuse it cannot write at nmesia ? | 08:46 |
| jistr | jaosorior, janki: the `hidden` value for that parameter means that the value will not be queryable by `heat resource-show ...` commands i think | 08:47 |
| jistr | openstack object save overcloud plan-environment.yaml | 08:47 |
| jistr | cat plan-environment.yaml | 08:47 |
| jistr | janki: ^^ here they are | 08:47 |
| janki | jaosorior, right now we are doing - https://github.com/openstack/tripleo-heat-templates/blob/e64c10b9c13188f37e6f122475fe02280eaa6686/puppet/services/opendaylight-api.yaml#L15. I dont want "admin" value to be seen to the users but just the admin | 08:48 |
| jistr | janki, jaosorior: oh, we should probably get rid of that defaulted password, and add that parameter in the list of parameters that should get an autogenerated value | 08:49 |
| *** zshi has quit IRC | 08:50 | |
| *** pblaho has joined #tripleo | 08:50 | |
| janki | jistr, jaosorior those auto-generated are random string of numbers. can we have some value generated that is easily rememberable? and what if customers want to supply their own value? ODL has that provision but that custom password would be passed in a different THT stored on undercloud | 08:51 |
| sri_ | shardy: Hi a couple of questions related to networking, I've two nics. 2nic for provisioning and first 1nic for external/public/tenant etc.., in order generate templates I am using process-templates.py, process-templates uses network_data.yaml file to generate templates but all of the networks(External, Storage, etc...) need to use one subnet is this possible ? . except for provisioning nic similarly like this https://docs.openstack.org/tripleo-docs/latest/_i | 08:51 |
| sri_ | mages/TripleO_Network_Diagram_.jpg, I am not sure how to archive this, Do I need use single-nic-VLANs or multi-nic ? | 08:51 |
| jaosorior | janki: it's already possible to provision your own passwords if you use the appropriate interface and pass in the explicit name | 08:52 |
| jaosorior | janki: they are autogenerated by mistral by default, but it is overwriteable | 08:52 |
| *** psahoo has quit IRC | 08:54 | |
| shardy | janki: users can always provide their own passwords that override the generated ones | 08:54 |
| janki | jaosorior, so in case of ODL, if someone creates a new THT like parameters_default: ODLPassword: <myvalue> and pass it during deploy, it would be overwritten right | 08:54 |
| janki | shardy, ^ | 08:54 |
| shardy | janki: No | 08:54 |
| shardy | the user provided one should take precedence | 08:55 |
| shardy | the point is to avoid using something hard-coded if they decide not to pass any password | 08:55 |
| shardy | as that could lead to a vulnerability | 08:55 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Remove tempest compute-feature-enabled.attach_encrypted_volume https://review.openstack.org/570158 | 08:56 |
| janki | shardy, s/it would be overwritten right/it will overwrite the default one | 08:56 |
| jistr | yea the random ones are low priority. If users pass something in env file, the user-provided value is used instead of the random one. | 08:56 |
| shardy | sri_: so it sounds like you basically want two networks, with two nics? | 08:56 |
| sri_ | shardy: yes | 08:57 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Execute post_update_tasks in update playbook https://review.openstack.org/577754 | 08:57 |
| shardy | sri_: I'd create a network_data.yaml with the two networks then use the multi-nic example I think | 08:57 |
| jistr | janki: if you want to try with ODL, that's the post_update_tasks patch ^^ | 08:57 |
| bogdando | folks, which ansible command should I use to re-trigger Undercloud/docker_puppet_tasks.yaml ? | 08:58 |
| janki | jistr, will do try that. we will need a d/s bug for this and to be included in z stream | 08:58 |
| jistr | janki: ack i can report it | 08:58 |
| shardy | sri_: you may also need to pass a custom ServiceNetMap to map all services to the second network | 08:58 |
| bogdando | jistr: do you know perchance? | 08:58 |
| shardy | sri_: personally I'd probably use vlans on the second nic for all the default networks | 08:59 |
| bogdando | Undercloud/docker_puppet_tasks.yaml how can I retrigger it manually? | 08:59 |
| bogdando | shardy: ^^ | 08:59 |
| shardy | provided you can configure that on the switch | 08:59 |
| *** gfidente has joined #tripleo | 08:59 | |
| *** gfidente has quit IRC | 08:59 | |
| *** gfidente has joined #tripleo | 08:59 | |
| bogdando | that's not the task itself tho, but data for it... | 08:59 |
| bogdando | so where lives that task playbook... | 09:00 |
| *** Haresh has quit IRC | 09:01 | |
| *** agurenko has joined #tripleo | 09:01 | |
| *** Haresh has joined #tripleo | 09:01 | |
| *** pkovar has joined #tripleo | 09:01 | |
| sri_ | shardy: Yes i think that's easy way to it, I was wondering in single-nic-vlan setup can we use one vlan for all the services ? | 09:01 |
| shardy | bogdando: you can either run the deploy_steps_tasks playbook or run docker-puppet.py manually with the Undercloud/docker_puppet_tasks.yaml | 09:02 |
| bogdando | shardy: thanks! | 09:02 |
| shardy | sri_: with network isolation not all services use the same network, so it'd be easier if you created a vlan per network, e.g like in the examples/docs | 09:02 |
| shardy | but if you need to use a single vlan you'll need to modify the ServiceNetMap to avoid using the additional networks that we define by default | 09:03 |
| shardy | e.g storage etc | 09:03 |
| shardy | it does also depend what services you are deploying | 09:03 |
| *** dtantsur|afk is now known as dtantsur | 09:03 | |
| shardy | sri_: you could run two nics and still use vlans for the isolated networks | 09:04 |
| dtantsur | Tengu: hi | 09:04 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: QDR for metrics collection purposes https://review.openstack.org/572312 | 09:04 |
| sri_ | shardy: understood | 09:04 |
| Tengu | dtantsur: heya! | 09:04 |
| Tengu | dtantsur: just in order to understand your request : you'd rather want to avoid the "non-http-boot" example, and push the example with http-boot option in first position, and maybe a note stating "prior to Rocky, non-containerized undercloud, meaning --http-boot <older location>" ? | 09:06 |
| janki | jistr, so all the tasks within for "role" loop happens togehter on all roles, and others happen 1 node at a time? eg: {%- for role in roles %} | 09:06 |
| janki | - include: {{role.name}}/host_prep_tasks.yaml | 09:06 |
| janki | when: tripleo_role_name == '{{role.name}}' | 09:06 |
| janki | {%- endfor %} | 09:06 |
| *** yprokule has joined #tripleo | 09:06 | |
| dtantsur | Tengu: well, note is an optional read. as an exercise, expect the documentation to be fully valid for any consumers with all "note" removed. | 09:06 |
| dtantsur | meaning, you cannot put important information there, for any branch | 09:07 |
| Tengu | dtantsur: hmm ok. so no ".. note" | 09:07 |
| dtantsur | for stable branches we have special syntax ("admonition" - not sure what this word means) | 09:07 |
| dtantsur | try looking for examples in the documentation | 09:07 |
| Tengu | ah, that's the meaning of that tag. | 09:07 |
| Tengu | ok. | 09:07 |
| dtantsur | the default version should apply to master, which is Rocky | 09:07 |
| Tengu | will update the review then. | 09:07 |
| dtantsur | thanks! | 09:07 |
| Tengu | dtantsur: thank you for the review and remarks :). | 09:08 |
| dtantsur | np :) | 09:08 |
| dtantsur | Tengu: also sync with bogdando on whether the --http-boot thing is still required - I think he planned to change the default for master\ | 09:08 |
| bogdando | dtantsur: there is a trello card that explains the case, Tengu , https://trello.com/c/kotXZkSo/51-overcloud-image-upload-http-boot-option-defaults-incompatibility | 09:09 |
| openstackgerrit | Martin Mágr proposed openstack/puppet-tripleo master: Collectd QDR connection https://review.openstack.org/571152 | 09:09 |
| bogdando | all we need is to not specify that option and things keep working | 09:09 |
| bogdando | for the containerized undercloud | 09:09 |
| *** psahoo has joined #tripleo | 09:09 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 09:10 |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778040 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778472 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1778040 in tripleo "Error at overcloud_prep_containers Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", " Requires: libqpid-proton.so.10()(64bit)" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 09:10 |
| openstack | Launchpad bug 1778472 in tripleo "docker pull failed: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo: received unexpected HTTP status: 503 Service Unavailable" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 09:10 |
| *** jcoufal has quit IRC | 09:10 | |
| bogdando | and it happend the only accepted solution is stop using instack-undercloud for image preparation tasks | 09:10 |
| Tengu | bogdando: ah, will check that. | 09:10 |
| bogdando | cuz we declined fixing instack | 09:10 |
| Tengu | maybe my change is too early? | 09:10 |
| bogdando | what change? | 09:11 |
| Tengu | bogdando: in doc for intallation | 09:11 |
| bogdando | shardy: I forgot the way to pass in params for docker-puppet :] | 09:11 |
| bogdando | shardy: specifically, Undercloud/docker_puppet_tasks.yaml | 09:12 |
| shardy | https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/tips_tricks.html#debugging-docker-puppet-py | 09:12 |
| bogdando | Tengu: please give me a ref | 09:12 |
| shardy | bogdando: that may help, and I blogged about it | 09:12 |
| Tengu | bogdando: https://review.openstack.org/#/c/573196/ | 09:12 |
| shardy | https://hardysteven.blogspot.com/2018/06/tripleo-containerized-deployments.html | 09:13 |
| bogdando | shardy: hm, but that ha nothing to the yaml data config download produces?.. | 09:13 |
| bogdando | has | 09:13 |
| *** jcoufal has joined #tripleo | 09:13 | |
| *** jcoufal has quit IRC | 09:13 | |
| *** jcoufal has joined #tripleo | 09:14 | |
| shardy | Undercloud/docker_puppet_tasks.yaml is the per-role tasks | 09:16 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml#L134 | 09:16 |
| shardy | bogdando: ^^ | 09:16 |
| bogdando | uhm... what is really missing is a list of examples of ansible-playbook commands hooking in some different points in the deployment graph to do it one time | 09:17 |
| bogdando | shardy: really confusing... | 09:18 |
| bogdando | sorry :) | 09:18 |
| shardy | bogdando: well deploy-steps-tasks.yaml shows every task in order? | 09:18 |
| bogdando | say, I need only run step1, docker_puppet_tasks | 09:18 |
| quiquell|rover | bogdando: Maybe ARA give you that info ? | 09:18 |
| bogdando | yes, but now I have to fifure out those --limit things | 09:18 |
| bogdando | and --tag things | 09:18 |
| shardy | bogdando: docker-puppet.py only ever runs on the first step | 09:19 |
| shardy | no limit or tag needed | 09:19 |
| bogdando | to not redeploy the world occasionally | 09:19 |
| shardy | just run it manually | 09:19 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml#L190# | 09:19 |
| *** mhenkel_ has joined #tripleo | 09:19 | |
| * shardy shrugs | 09:19 | |
| bogdando | shardy: I did, but it seems didn't updated rabbit containers :. | 09:19 |
| bogdando | :/ | 09:19 |
| shardy | bogdando: docker-puppet.py doesn't update containers | 09:20 |
| shardy | paunch does | 09:20 |
| shardy | I tried to explain this in the blog post, I'm sorry if it was not clear enough :( | 09:20 |
| bogdando | shardy: wanted to try a local change done into http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/rabbitmq.yaml#n209 | 09:20 |
| bogdando | and touch nothing else | 09:20 |
| bogdando | still not sure how to trigger that... ok, I better off just redeploy | 09:21 |
| shardy | bogdando: you can just re-run docker-puppet.py with that applied manually, or by running config download first | 09:21 |
| shardy | or you can run deploy_steps_tasks | 09:21 |
| shardy | docker-puppet.py will regenerate the config, which you can verify by inspection | 09:21 |
| shardy | if you want to restart the service containers you'll also have to re-run paunch | 09:22 |
| *** Petersingh has joined #tripleo | 09:23 | |
| shardy | bogdando: you can also copy and modify the generated deploy_steps_playbook.yaml | 09:23 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L391 | 09:23 |
| shardy | to test the bits you want | 09:23 |
| *** Petersingh has quit IRC | 09:23 | |
| shardy | there are tags, but no way to limit to a single step atm | 09:23 |
| shardy | so you may need to delete the later steps if you just want the first one | 09:23 |
| bogdando | shardy: yeah, that makes sense. thank you for hints! | 09:24 |
| shardy | bogdando: np, hope that helps | 09:24 |
| bogdando | just thought there is a simple way though I don't know of it ;) | 09:24 |
| shardy | bogdando: yeah maybe we could make it easier by having a way to run deploy_steps_playbook for a single step | 09:25 |
| shardy | previously it was a loop, but since that loop got unrolled that's not super-easy without hacking on the yaml | 09:25 |
| *** zoli is now known as zoli|lunch | 09:28 | |
| *** mhenkel_ has quit IRC | 09:28 | |
| *** mhenkel_ has joined #tripleo | 09:29 | |
| *** ykarel|lunch is now known as ykarel | 09:29 | |
| *** psahoo has quit IRC | 09:29 | |
| bogdando | shardy: the best I could think of is "ansible-playbook -i inventory.yaml --limit Undercloud -e @Undercloud/docker_puppet_tasks.yaml -e step=1 deploy_steps_playbook.yaml" | 09:30 |
| bogdando | still includes more than I wanted, but ... ok | 09:30 |
| *** salmankhan has joined #tripleo | 09:31 | |
| *** hjensas has joined #tripleo | 09:33 | |
| *** hjensas has quit IRC | 09:33 | |
| *** hjensas has joined #tripleo | 09:33 | |
| shardy | bogdando: you could perhaps add --tags deploy_steps | 09:33 |
| bogdando | shardy: I was not sure this wouldn't cut off the docker_puppet_tasks :) | 09:34 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L508 | 09:34 |
| shardy | bogdando: that's run via the deploy_steps_tasks.yaml include | 09:34 |
| bogdando | shardy: do you know which names the containers executed for the docker_puppet_tasks are given? | 09:36 |
| bogdando | can't find anything for rabbitmq_* | 09:36 |
| bogdando | I have only rabbitmq_init_logs and rabbitmq_bootstrap and the rabbitmq itself | 09:36 |
| bogdando | where goes that init container for docker_puppet_takss? | 09:37 |
| jistr | janki: the for loop is during playbook generation time, not playbook run time. It just makes sure that all roles get their tasks included, but it doesn't really loop over roles during run time. The main loop is over individual nodes with `serial: 1` and then there are smaller loops over steps in update tasks, then steps in deploy tasks, then steps in post update tasks. | 09:37 |
| bogdando | prolly I have them removed | 09:37 |
| shardy | bogdando: the containers only run for a short time while docker-puppet.py runs | 09:37 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-puppet.py#L131 | 09:38 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-puppet.py#L296 | 09:39 |
| bogdando | shardy: right, just started forgetting that code :) | 09:39 |
| bogdando | thanks a lot | 09:39 |
| shardy | they are named docker-puppet-* | 09:39 |
| bogdando | that command still redeploys the world :) | 09:40 |
| bogdando | just wanted to limit changes to rabbit service | 09:40 |
| * bogdando sigh | 09:40 | |
| shardy | something else in the config must have changed | 09:41 |
| janki | jistr, yupe got it. serial is actually ansilbe keyword for rolling updates. so all taks without "serial" happen in parallel and ones with "serial" 1 node at a time | 09:41 |
| *** bkopilov has quit IRC | 09:41 | |
| *** psahoo has joined #tripleo | 09:41 | |
| shardy | if you really only want to touch rabbit, run docker-puppet and paunch outside ansible with the existing configs | 09:41 |
| shardy | if nothing has changed you can re-run the deploy steps and nothing at all should get restarted | 09:42 |
| *** moshele has quit IRC | 09:43 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: Enable collectd to connect to metrics QDR https://review.openstack.org/576057 | 09:46 |
| jistr | janki: yea | 09:46 |
| janki | jistr, please share the d/s BZ number for post_update_tasks once you file it. I would like to add dependecy for ODL update BZ | 09:48 |
| jistr | janki: ack i'll add you to cc there | 09:48 |
| numans | shardy, bogdando Hi, can you please add it in your review queue. Thanks | 09:49 |
| janki | jistr, thanks :) | 09:49 |
| bogdando | numans: which patch? | 09:49 |
| numans | bogdando, oops.. | 09:49 |
| numans | one sec | 09:49 |
| numans | bogdando, shardy https://review.openstack.org/#/c/571958 | 09:50 |
| *** hjensas has quit IRC | 09:51 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: WIP: use the tq ansible.cfg for the undercloud install https://review.openstack.org/575535 | 09:51 |
| shardy | numans: you may need to temporarily abandon https://review.openstack.org/#/c/577376/ if you want to merge that, or the Depends-On will block the merge until the puppet-neutron backport merges | 09:52 |
| shardy | or we can chase getting that landed | 09:53 |
| jistr | janki: filed a BZ, you're CCed https://bugzilla.redhat.com/show_bug.cgi?id=1594731 | 09:55 |
| openstack | bugzilla.redhat.com bug 1594731 in openstack-tripleo-heat-templates "Update procedure doesn't trigger post_update_tasks" [High,Assigned] - Assigned to jstransk | 09:55 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Add users container images file into tarball https://review.openstack.org/576086 | 09:56 |
| *** psahoo has quit IRC | 09:57 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient stable/pike: Add --stack to update, upgrade and ffwd-upgrade 'run' CLI. https://review.openstack.org/568273 | 09:58 |
| *** hjensas has joined #tripleo | 09:59 | |
| *** hjensas has quit IRC | 09:59 | |
| *** hjensas has joined #tripleo | 09:59 | |
| *** nyechiel_ has quit IRC | 10:05 | |
| *** nyechiel has joined #tripleo | 10:05 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778040 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778472 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 10:10 |
| openstack | Launchpad bug 1778040 in tripleo "Error at overcloud_prep_containers Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", " Requires: libqpid-proton.so.10()(64bit)" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 10:10 |
| openstack | Launchpad bug 1778472 in tripleo "docker pull failed: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo: received unexpected HTTP status: 503 Service Unavailable" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 10:10 |
| *** cshastri_ has joined #tripleo | 10:11 | |
| *** cshastri has quit IRC | 10:11 | |
| *** psahoo has joined #tripleo | 10:15 | |
| *** psahoo_ has joined #tripleo | 10:19 | |
| *** psahoo has quit IRC | 10:20 | |
| quiquell|rover | bogdando: Can we test removing "ro" ? | 10:21 |
| *** gfidente has quit IRC | 10:23 | |
| sri_ | shardy: My overcloud deployment fails this is the errors and config http://paste.openstack.org/show/724208/, I am not able to find exact error why it failed | 10:24 |
| *** milan has joined #tripleo | 10:26 | |
| shardy | sri_: try running openstack stack failures list --long overcloud | 10:27 |
| shardy | os-net-config failed, so you need the error, and/or re-run it manually | 10:27 |
| shardy | it means there's a problem with your nic configuration | 10:27 |
| sri_ | shardy: failures list --long not giving much info http://paste.openstack.org/show/724212/ | 10:32 |
| sri_ | shardy: compute and controller network config http://paste.openstack.org/show/724211/ | 10:32 |
| sri_ | shardy, according to the network_data.yaml , "The ordering of the networks below will determine the order in which NICs | 10:33 |
| sri_ | are assigned in the network/config/multiple-nics templates, beginning with | 10:33 |
| sri_ | NIC2, Control Plane is always NIC1." | 10:33 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Running containerized tempest only in containerized environment https://review.openstack.org/577780 | 10:34 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: WIP: use the tq ansible.cfg for the undercloud install https://review.openstack.org/575535 | 10:35 |
| sri_ | shardy: in my case Control Plane in nic2, is that something has to with this error ! | 10:35 |
| bogdando | quiquell|rover: trying that | 10:35 |
| quiquell|rover | bogdando: ack | 10:36 |
| bogdando | quiquell|rover: either it didn't work, or I can't test it the short way :D | 10:36 |
| bogdando | so I have to redeploy | 10:37 |
| quiquell|rover | bogdando: Create a review with it at the same time | 10:37 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add uuid for vfat https://review.openstack.org/576765 | 10:38 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates master: ovn: Add dns_servers configuration support https://review.openstack.org/571958 | 10:41 |
| quiquell|rover | bogdando: btw, do you need some help checking migrated periodic jobs to zuul v3 ? | 10:42 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Adding missing pacemaker definition for scenario000. https://review.openstack.org/577783 | 10:43 |
| numans | shardy, i missed your comment here and i updated the patch removing the Depends-On tag. | 10:43 |
| chem | marios: ^ this should fix the queen scenario000 upgrade job | 10:43 |
| *** shardy has quit IRC | 10:44 | |
| *** edmondsw has joined #tripleo | 10:45 | |
| bogdando | quiquell|rover: I think I'm as I have no idea why I'd have to do that :D | 10:45 |
| quiquell|rover | bogdando: I have the same feeling about my question :-) so we are on the same page :-P | 10:45 |
| Tengu | dtantsur: trying to figure out how to update the doc - "Rocky" isn't stable for now right? Or shall I already use the "stable" admonition class for rocky.. ? or bogdando ? (still the doc for the boot-http thingy) | 10:45 |
| dtantsur | Tengu: rocky is master, stable is <= queens | 10:46 |
| Tengu | hm. no "master" class for admonition apparently. | 10:46 |
| bogdando | Tengu: satable for now means < Rocky | 10:46 |
| bogdando | and all the text goes for master | 10:46 |
| bogdando | which is rocky | 10:47 |
| Tengu | ah, ok. | 10:47 |
| Tengu | so I push the text with the http-boot /var/lib/.... and adminition class stable for http-boot /httpboot | 10:47 |
| Tengu | *admonition | 10:47 |
| dtantsur | yep | 10:48 |
| *** edmondsw_ has joined #tripleo | 10:48 | |
| *** edmondsw has quit IRC | 10:49 | |
| quiquell|rover | marios: So pacamaker was not even installed at https://bugs.launchpad.net/tripleo/+bug/1777132 | 10:50 |
| openstack | Launchpad bug 1777132 in tripleo "queens branch tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades is broken" [High,Triaged] - Assigned to Quique Llorente (quiquell) | 10:50 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/queens: Missing puppet restart resource definition in scenario000. https://review.openstack.org/577784 | 10:50 |
| *** lvdombrkr89 has quit IRC | 10:51 | |
| *** dhill_ has quit IRC | 10:53 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-docs master: Added missing information regarding containerized undercloud and ironic https://review.openstack.org/573196 | 10:53 |
| chem | quiquell|rover: hey I've got a review on this ... did I miss yours ? | 10:53 |
| Tengu | dtantsur: bogdando -^^ | 10:53 |
| chem | quiquell|rover: https://review.openstack.org/#/c/577783/ | 10:53 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Fix rabbitmq user password updates https://review.openstack.org/577785 | 10:54 |
| bogdando | quiquell|rover: https://review.openstack.org/#/c/577785/ feel free to try it with the periodic job | 10:55 |
| bogdando | I'll keep my local testing | 10:55 |
| bogdando | my futile local testing* | 10:55 |
| bogdando | those docker puppet and paunch dances... | 10:55 |
| bogdando | giving no effect :) | 10:55 |
| Tengu | bogdando: change the music, take some other tempo :) | 10:56 |
| bogdando | indeeed | 10:56 |
| Tengu | hmm. ok. my afternoon seems to be dedicated to meetings :]. | 10:57 |
| Tengu | 4 in a row. | 10:57 |
| *** gfidente has joined #tripleo | 10:57 | |
| *** jpena is now known as jpena|lunch | 11:00 | |
| *** bkopilov has joined #tripleo | 11:01 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-quickstart-extras master: Remove ctlplane data from CI network-environment https://review.openstack.org/577786 | 11:03 |
| *** zoli|lunch is now known as zoli | 11:04 | |
| *** slaweq has joined #tripleo | 11:04 | |
| *** pchavva has joined #tripleo | 11:04 | |
| *** quiquell|rover has quit IRC | 11:06 | |
| *** jcoufal has quit IRC | 11:08 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778040 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778472 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 11:10 |
| openstack | Launchpad bug 1778040 in tripleo "Error at overcloud_prep_containers Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", " Requires: libqpid-proton.so.10()(64bit)" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 11:10 |
| openstack | Launchpad bug 1778472 in tripleo "docker pull failed: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo: received unexpected HTTP status: 503 Service Unavailable" [Critical,Fix committed] - Assigned to Quique Llorente (quiquell) | 11:10 |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates master: Delete ovs ports blocking logic https://review.openstack.org/576210 | 11:10 |
| *** pgadiya has joined #tripleo | 11:14 | |
| *** atoth has joined #tripleo | 11:15 | |
| bogdando | hm, given http://git.openstack.org/cgit/openstack/puppet-tripleo/tree/manifests/profile/base/rabbitmq.pp#n240 is for step >=2, and docker_puppet_tasks are only for the step1, does it mean we have this puppet config http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/rabbitmq.yaml#n209 never executed for non HA rabbit?.. | 11:17 |
| bogdando | HA rabbit use another way of running the rabbitmq_users hook, at another step | 11:18 |
| bogdando | dciabrin, bandini: here perchance? | 11:18 |
| bogdando | ^^ | 11:18 |
| *** cshastri_ has quit IRC | 11:18 | |
| bogdando | seems like we have the docker_puppet_steps broken for anything we rely on step > 1 in puppet ? | 11:19 |
| * bogdando confused | 11:19 | |
| *** quiquell has joined #tripleo | 11:19 | |
| *** lvdombrkr89 has joined #tripleo | 11:20 | |
| bogdando | or rather, I should just fix it for the rabbit to be executred on step_2 | 11:20 |
| quiquell | bogdando: Let's try the thing | 11:21 |
| bogdando | yupp | 11:21 |
| quiquell | bogdando: What's the Signed-off-by ? | 11:21 |
| quiquell | bogdando: Have just reproduced the thing a RDO cloud let's just test it | 11:22 |
| *** cshastri_ has joined #tripleo | 11:23 | |
| *** slaweq has quit IRC | 11:23 | |
| quiquell | bogdando: We will to backport the change for queens, don't know about pike or ocata | 11:23 |
| bogdando | quiquell: https://stackoverflow.com/questions/1962094/what-is-the-sign-off-feature-in-git-for | 11:24 |
| bogdando | so, just a fancy sign | 11:25 |
| *** cshastri_ has quit IRC | 11:25 | |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-docs master: Add BIOS settings doc https://review.openstack.org/577343 | 11:25 |
| marios | quiquell: yeah i was chatting with rlandy about it on friday. there is no pcs ther and we couldn't spot the difference from the master job (same scenario/features etc) | 11:25 |
| *** cshastri has joined #tripleo | 11:25 | |
| marios | quiquell: sorry was fighting with the reproducer script and libvirt pools missed irc (wrt https://bugs.launchpad.net/tripleo/+bug/1777132 | 11:26 |
| openstack | Launchpad bug 1777132 in tripleo "queens branch tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades is broken" [High,Triaged] - Assigned to Quique Llorente (quiquell) | 11:26 |
| *** bfournie has quit IRC | 11:26 | |
| quiquell | marios: OOOk, no problem, so may fights, didn't have time to help you :-/ | 11:26 |
| marios | quiquell: i added a note there and intend to revisit. rlandy was trying to find when the last successful was but wasn't obvious. it looks like puppet tries to start pacemaker so there is another bug there | 11:26 |
| bogdando | quiquell: pike is affected as well, overclouds non-Ha rabbits | 11:26 |
| bogdando | queens the same way | 11:26 |
| marios | quiquell: i mean, that puppet fails to start the cluster but it didn't fail the deploy | 11:26 |
| *** lvdombrkr has joined #tripleo | 11:27 | |
| *** pradk has joined #tripleo | 11:27 | |
| quiquell | marios: So we don' thave a fail fast for this ? | 11:27 |
| *** udesale has quit IRC | 11:27 | |
| quiquell | bogdando: All except Bocata, I mean Ocata | 11:27 |
| Tengu | dtantsur: http://logs.openstack.org/96/573196/2/check/build-openstack-sphinx-docs/7d84f46/html/install/basic_deployment/basic_deployment_cli.html#upload-images are you OK with this display? | 11:27 |
| marios | quiquell: right, the thing that fails is on the upgrade tasks (it is a check if the cluster is up, that fails as it isn't) which means the deployment passed fine (but if you see controller does this Jun 22 09:36:59 centos-7-ovh-bhs1-0000291507 puppet-user[8404]: Puppet::Type::Service::ProviderPacemaker: file pcs does not exist | 11:28 |
| *** tosky has joined #tripleo | 11:28 | |
| dtantsur | Tengu: yeah, looks good, modulo grammar ("stable" what? "Hence" goes to the front) | 11:28 |
| marios | quiquell: i.e. tried to but failed to start cluster but the deploy passed fine (and the check fails on the upgrade tasks later) | 11:28 |
| *** lvdombrkr89 has quit IRC | 11:28 | |
| *** moguimar has quit IRC | 11:28 | |
| Tengu | dtantsur: ah, "Stable branch" I guess then. | 11:28 |
| marios | quiquell: looks like chem has proposed a fix for it | 11:29 |
| dtantsur | Tengu: even better "Before the Rocky release" or something. remember that Rocky will also become stable one dau | 11:29 |
| dtantsur | day | 11:29 |
| Tengu | dtantsur: true. | 11:29 |
| marios | chem: nice :) indeed i was looking at queens tht | 11:29 |
| dtantsur | and that most of users are consuming releases, not branches | 11:29 |
| marios | chem: i mean for scenario000 | 11:29 |
| *** moguimar has joined #tripleo | 11:30 | |
| *** zshi has joined #tripleo | 11:30 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-docs master: Added missing information regarding containerized undercloud and ironic https://review.openstack.org/573196 | 11:30 |
| Tengu | should be OK then :). | 11:30 |
| *** hkominos has joined #tripleo | 11:31 | |
| *** amoralej is now known as amoralej|lunch | 11:32 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: WIP: use the tq ansible.cfg for the undercloud install https://review.openstack.org/575535 | 11:32 |
| *** vpickard has quit IRC | 11:34 | |
| *** cshastri has quit IRC | 11:35 | |
| *** cshastri has joined #tripleo | 11:36 | |
| *** vpickard has joined #tripleo | 11:36 | |
| *** noama has quit IRC | 11:37 | |
| quiquell | bogdando: About migration, do you know why this is happening ? https://logs.rdoproject.org/67/14467/3/check/legacy-rdoinfo-tripleo-pike-testing-centos-7-multinode-1ctlr-featureset006/0209252/job-output.txt.gz#_2018-06-25_09_45_03_379935 | 11:38 |
| quiquell | chown: cannot access '/opt/git': No such file or directory | 11:39 |
| openstackgerrit | Martin Schuppert proposed openstack/instack-undercloud stable/queens: Set undercloud nova notification_format to 'unversioned' https://review.openstack.org/577790 | 11:40 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Setup Ironic in Overcloud https://review.openstack.org/509728 | 11:40 |
| openstackgerrit | Martin Schuppert proposed openstack/instack-undercloud stable/pike: Set undercloud nova notification_format to 'unversioned' https://review.openstack.org/577792 | 11:41 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common stable/queens: Include --xattrs when creating the undercloud backup. https://review.openstack.org/577793 | 11:41 |
| *** cshastri has quit IRC | 11:41 | |
| *** slagle has joined #tripleo | 11:42 | |
| *** vpickard has quit IRC | 11:42 | |
| *** psahoo_ has quit IRC | 11:43 | |
| bogdando | quiquell: have no idea | 11:43 |
| quiquell | bogdando: Stupid me, wrong nick :-/ | 11:43 |
| bogdando | folks, we have a funny bug in stack_action detection in puppets for the undercloud | 11:43 |
| *** fzdarsky_ has joined #tripleo | 11:43 | |
| bogdando | hiera always shows its CREATE | 11:43 |
| *** vpickard has joined #tripleo | 11:43 | |
| bogdando | for consequent redeploys as well, as we always create a new ephemeral stack! | 11:44 |
| bogdando | never update | 11:44 |
| bogdando | so everything in puppet-triple relying on stack_action != CREATE is broken for undercloud ;] | 11:44 |
| bogdando | that's why I could not test the rabbitmq_user update | 11:44 |
| bogdando | now we need to think of a hack for t-h-t, an undercloud-only specific hack | 11:45 |
| bogdando | to not break logic for overcloud | 11:45 |
| bogdando | in the shared services templates | 11:45 |
| *** fzdarsky_ is now known as fzdarsky | 11:50 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/tripleo-heat-templates master: ceilometer: deprecation cleanup https://review.openstack.org/577797 | 11:53 |
| *** edmondsw_ has quit IRC | 11:53 | |
| bogdando | https://bugs.launchpad.net/tripleo/+bug/1778505 | 11:54 |
| openstack | Launchpad bug 1778505 in tripleo "Undercloud heat installer cannot rely on stack_action" [Critical,Triaged] | 11:54 |
| *** pdeore has quit IRC | 11:54 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove unnecessary code from toci_* scripts https://review.openstack.org/576834 | 11:55 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove unnecessary code from toci_* scripts https://review.openstack.org/576834 | 11:56 |
| *** psahoo_ has joined #tripleo | 11:56 | |
| *** dhill_ has joined #tripleo | 11:58 | |
| *** jpena|lunch is now known as jpena | 12:00 | |
| dciabrin | bogdando, hhmm yes i doubt the docker_puppet_task will do anything meaningfull past step1 for non-HA | 12:01 |
| dciabrin | should double check with bandini, he's still on pto today | 12:02 |
| bogdando | dciabrin: then we have another crit bug :) | 12:02 |
| bogdando | sigh | 12:02 |
| openstackgerrit | Nguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images https://review.openstack.org/542708 | 12:02 |
| bogdando | we have a plenty of it in tht for step_5 or something | 12:02 |
| bogdando | folks, who knows how to confirm that from the deploy steps in tht? | 12:02 |
| quiquell | bogdando: This block the other fix ' | 12:03 |
| quiquell | ? | 12:03 |
| *** raildo has joined #tripleo | 12:03 | |
| bogdando | I think shardy pointer out to that as well 12:19:01 PM GMT+3 - shardy: bogdando: docker-puppet.py only ever runs on the first step | 12:04 |
| bogdando | just not sure how to find that code | 12:04 |
| bogdando | quiquell: this (docker puppet.py @ step1) and also https://bugs.launchpad.net/tripleo/+bug/1778505 blocks the fix for https://bugs.launchpad.net/bugs/1777939 | 12:05 |
| openstack | Launchpad bug 1778505 in tripleo "Undercloud heat installer cannot rely on stack_action" [Critical,Triaged] | 12:05 |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 12:05 |
| bogdando | what a nice snowball of bugs we pulled out | 12:05 |
| quiquell | bogdando: Perfect sprint to be rover :-) | 12:06 |
| bogdando | :D | 12:06 |
| quiquell | bogdando: What's the docker-puppet.py ? | 12:06 |
| quiquell | what's the issue ? | 12:06 |
| bogdando | quiquell: whatever step for docker_puppet_tasks we specify in t-h-t, it only works for step1 | 12:07 |
| openstackgerrit | yolanda.robla proposed openstack/python-tripleoclient master: WIP: Allow to skip not existing images when uploading https://review.openstack.org/577799 | 12:08 |
| quiquell | bogdando: So we don't execute the step5 with the config change ? | 12:08 |
| bogdando | in puppet we have things expecting to be run for step > 1, and we call this in docker_puppet_steps | 12:08 |
| *** bfournie has joined #tripleo | 12:08 | |
| bogdando | I have to test this somehow, dunno | 12:08 |
| bogdando | and another bug, that undercloud only has stack_action=CREATE | 12:09 |
| quiquell | bogdando: Confirmed it's still failing with the change :-( | 12:09 |
| quiquell | I ahve reproduced it | 12:09 |
| bogdando | that's that about that https://bugs.launchpad.net/tripleo/+bug/1778505 is | 12:09 |
| openstack | Launchpad bug 1778505 in tripleo "Undercloud heat installer cannot rely on stack_action" [Critical,Triaged] | 12:09 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: Add scenario011 to install ironic in the overcloud https://review.openstack.org/485261 | 12:09 |
| bogdando | quiquell: how did you test? | 12:09 |
| quiquell | bogdando: quickstart-reproducer.sh at RDO cloud | 12:09 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Add featureset044 for Ironic in the Overcloud https://review.openstack.org/509829 | 12:09 |
| bogdando | ah, yes | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778040 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778472 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778505 | 12:10 |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| bogdando | nevermind | 12:10 |
| openstack | Launchpad bug 1778040 in tripleo "Error at overcloud_prep_containers Package: qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", " Requires: libqpid-proton.so.10()(64bit)" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 12:10 |
| openstack | Launchpad bug 1778472 in tripleo "docker pull failed: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo: received unexpected HTTP status: 503 Service Unavailable" [Critical,Fix committed] - Assigned to Quique Llorente (quiquell) | 12:10 |
| *** yprokule has quit IRC | 12:11 | |
| *** asbishop has joined #tripleo | 12:11 | |
| *** ratailor has quit IRC | 12:12 | |
| *** shardy has joined #tripleo | 12:12 | |
| *** pkovar has left #tripleo | 12:13 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Fix centos opstools repo deactivation https://review.openstack.org/577145 | 12:13 |
| bogdando | shardy, dciabrin: hmm, I think docker puppet tasks work for step >1 https://github.com/openstack/tripleo-heat-templates/blob/e1a16a4903c935611cd0ee8ac36ced4a8a97296d/common/deploy-steps-tasks.yaml#L184 | 12:14 |
| *** dprince has joined #tripleo | 12:14 | |
| Damjanek | Howdy. Can I get someone to do the review here: https://review.openstack.org/#/c/577737/ | 12:14 |
| Damjanek | It has already been merged to master. Now I'm hoping to get it to queens. | 12:15 |
| bogdando | shardy: what did you mean as > | 12:15 |
| bogdando | 12:19:01 PM GMT+3 - shardy: bogdando: docker-puppet.py only ever runs on the first step | 12:15 |
| *** trown|outtypewww has quit IRC | 12:15 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Move get_extra_vars_from_release to functions https://review.openstack.org/577801 | 12:15 |
| sshnaidm | marios, ^^ | 12:15 |
| *** rh-jelabarre has joined #tripleo | 12:15 | |
| bogdando | shardy: so whould this work http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/octavia-worker.yaml#n131 ? | 12:16 |
| bogdando | or only for step_1? | 12:16 |
| marios | sshnaidm: ack | 12:17 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: Enable collectd to connect to metrics QDR https://review.openstack.org/576057 | 12:17 |
| quiquell | bogdando: Where do I run puppet apply -e "notice(hiera('stack_action'))"? | 12:18 |
| quiquell | undercloud ? | 12:18 |
| *** ansmith has quit IRC | 12:18 | |
| bogdando | quiquell: yes | 12:18 |
| quiquell | bogdando: Could not find data item stack_action in any Hiera data file and no default supplied at line 1:8 on node undercloud.localdomain | 12:19 |
| bogdando | quiquell: hm, works for me :D try sudo hiera -c /etc/puppet/hiera.yaml stack_action | 12:20 |
| *** ykarel_ has joined #tripleo | 12:20 | |
| quiquell | bogdando: sudo did the trick | 12:20 |
| *** weshay_ is now known as weshay|ruck | 12:22 | |
| *** ykarel has quit IRC | 12:22 | |
| *** yprokule has joined #tripleo | 12:23 | |
| quiquell | bogdando: Why step_1 is not executed again even if stack_action is CREATE ? | 12:23 |
| quiquell | weshay|ruck: Hello there | 12:23 |
| weshay|ruck | good morning | 12:23 |
| Tengu | hello weshay|ruck | 12:23 |
| weshay|ruck | Tengu, wazzzz up | 12:24 |
| bogdando | quiquell: /var/lib/docker-puppet/docker-puppet-tasks | 12:24 |
| bogdando | bad paste | 12:24 |
| bogdando | quiquell: http://git.openstack.org/cgit/openstack/puppet-tripleo/tree/manifests/profile/base/rabbitmq.pp#n240 | 12:24 |
| *** ykarel_ is now known as ykarel | 12:24 | |
| bogdando | is only executed when the line 229 is true | 12:24 |
| shardy | bogdando: https://github.com/openstack/tripleo-heat-templates/blob/e1a16a4903c935611cd0ee8ac36ced4a8a97296d/common/deploy-steps-tasks.yaml#L190 | 12:24 |
| Tengu | weshay|ruck: unpiling emails, pushing reviews, reading stuff, what 'bout ya? | 12:24 |
| shardy | when: step == "1" | 12:24 |
| bogdando | shardy: oh, so we have a mess | 12:25 |
| shardy | bogdando: why? | 12:25 |
| bogdando | in tht, there is a lot of things like http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/octavia-worker.yaml#n131 | 12:25 |
| bogdando | or step2 for mysql | 12:25 |
| bogdando | will never work? | 12:25 |
| shardy | yes those tasks aren't run by docker-puppet.py | 12:25 |
| Tengu | weshay|ruck: hey btw, who should I ping in order to get a feedback on https://review.openstack.org/#/c/570841/ ? As it's oooq related, I'm not really sure who's in. | 12:25 |
| shardy | paunch runs them | 12:25 |
| weshay|ruck | Tengu, slaughtering chickens, reading spells, and chanting to unf--- ci :) | 12:25 |
| bogdando | ohhhh | 12:25 |
| bogdando | that's a little bit complicatd) | 12:26 |
| Tengu | weshay|ruck: :] do you want some goat as well for the sacrifice? | 12:26 |
| shardy | bogdando: well we have complexity due to the constraint to maintain backwards compatibility with puppet, sure | 12:26 |
| bogdando | shardy: thanks, FWIW I'm happy if http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/octavia-worker.yaml#n131 just works | 12:26 |
| shardy | but conceptually it's pretty simple, we generate config, then we do the deploy steps to start the containers, including any bootstrapping | 12:26 |
| bogdando | then my fix for 1777939 should work as well | 12:27 |
| shardy | bogdando: oh wait | 12:27 |
| weshay|ruck | Tengu, only if the goat has virtical iris's | 12:27 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/e1a16a4903c935611cd0ee8ac36ced4a8a97296d/common/deploy-steps-tasks.yaml#L240 | 12:27 |
| bogdando | shardy: one more thing please, could you ack/nack 1777939 | 12:27 |
| bogdando | shardy: oops, bad paste! https://bugs.launchpad.net/tripleo/+bug/1778505 | 12:27 |
| openstack | Launchpad bug 1778505 in tripleo "Undercloud heat installer cannot rely on stack_action" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 12:27 |
| shardy | bogdando: those are run via the second invocation of docker-puppet.py, my mistake | 12:27 |
| Tengu | weshay|ruck: damn. won't work then. | 12:27 |
| shardy | bogdando: there are also some tasks run via paunch | 12:27 |
| shardy | but those don't use puppet | 12:28 |
| *** ohochman has joined #tripleo | 12:28 | |
| bogdando | shardy: ack, for me it's enough to know now it just works ;) | 12:28 |
| bogdando | so I have to fix that 1778505 and then 1777939 | 12:28 |
| quiquell | bogdando: Have assing it to me, as rover, but you can take it | 12:28 |
| shardy | bogdando: can we drop some status file on the undercloud to set stack_action for updates? | 12:28 |
| bogdando | shardy: if I override stack_action UPDATE in hiera for undercloud, would that be a terrible hack? | 12:29 |
| bogdando | I do not want change stack_action logic in puppets, this highly likely would annihilate overcloud deployments and upgrades in 1000 places | 12:29 |
| shardy | bogdando: Personally I'd prefer to pass an extra -e file to the heat used, then any template conditionals will still work | 12:29 |
| *** moguimar has quit IRC | 12:29 | |
| bogdando | shardy: that's what I'm thinking of, overriding in hiera, via UndercloudExtraConfig? | 12:30 |
| shardy | bogdando: I'm thinking of a slightly different approach where we set the StackAction heat parameter | 12:30 |
| shardy | vs the hiera override | 12:30 |
| bogdando | would this merge to that we define in /etc/puppet/hieradata/undercloud_extraconfig.json: "ironic::drivers::ssh::libvirt_uri": "qemu:///session" ? | 12:30 |
| shardy | both will probably work, but my suggestion means any heat conditionals consuming StackAction will still work | 12:30 |
| *** edmondsw has joined #tripleo | 12:30 | |
| bogdando | shardy: +1 for StackAction. great idea | 12:30 |
| bogdando | (didn't know we have that StackAction) | 12:31 |
| *** amoralej|lunch is now known as amoralej | 12:31 | |
| bogdando | quiquell: that's a pure DF bug | 12:31 |
| *** moguimar has joined #tripleo | 12:31 | |
| shardy | bogdando: yeah that's what sets the stack_action hiera | 12:31 |
| bogdando | so I can take it | 12:31 |
| *** rlandy has joined #tripleo | 12:32 | |
| quiquell | bogdando: DF ? | 12:33 |
| shardy | ack thanks bogdando | 12:33 |
| bogdando | quiquell: yea, DFG deployment framework | 12:33 |
| mwhahaha | bogdando: https://review.openstack.org/#/c/576990/ | 12:34 |
| mwhahaha | bogdando: that should fix the rabbitmq issues | 12:34 |
| bogdando | mwhahaha: the issue is more subtle :) | 12:34 |
| bogdando | see https://bugs.launchpad.net/tripleo/+bug/1778505 | 12:34 |
| openstack | Launchpad bug 1778505 in tripleo "Undercloud heat installer cannot rely on stack_action" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 12:34 |
| mwhahaha | bogdando: there's a couple of problems, it's also the fact that we change the passwords int eh file | 12:35 |
| bogdando | mwhahaha: technically, we can change it | 12:35 |
| bogdando | ur a user | 12:35 |
| bogdando | or | 12:35 |
| *** trown has joined #tripleo | 12:35 | |
| bogdando | and then it won't get updated via docker_puppet_tasks | 12:35 |
| mwhahaha | bogdando: right, so that's fine but we change it incorrectly up front | 12:35 |
| mwhahaha | bogdando: so 1) we shouldn't do that and then 2) we should fix it so we can change it | 12:35 |
| bogdando | I'll take the latter | 12:35 |
| bogdando | thanks for the former fixing :) | 12:36 |
| quiquell | mwhahaha, bogdando: Testing it with the reproducer | 12:37 |
| bogdando | quiquell: that would help to hide the issue :) | 12:37 |
| quiquell | bogdando: This will keep the same password ? | 12:37 |
| bogdando | we need both fixes | 12:37 |
| quiquell | the second time ? | 12:37 |
| bogdando | yes | 12:37 |
| quiquell | bogdando: Why we need the other fix ? | 12:38 |
| bogdando | when user updates the password | 12:38 |
| bogdando | so prolly we can fix it as a separated, out of the promotion blocker | 12:38 |
| bogdando | I think, mwhahaha's patch will unblock as well | 12:38 |
| quiquell | bogdando: I am testing with mwhahaha fix, to see if fs002 passes, so we can unblock promotions | 12:39 |
| quiquell | bogdando: As you said we can fix the ActionStack in parallel | 12:39 |
| *** psahoo_ has quit IRC | 12:40 | |
| bogdando | shardy: hm, do we have role name in hiera? | 12:41 |
| bogdando | prolly it's better to checking in puppets if that's undercloud or stack == update | 12:41 |
| bogdando | I'm not sure changing http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/environments/undercloud.yaml#n40 wouldn't break undercloud in 1000 places :) | 12:42 |
| shardy | bogdando: why is that better, you'll have to modify every place consuming stack_action and if anyone ever changes the role name (e.g for standalone vs undercloud use) it won't work? | 12:42 |
| *** chem has quit IRC | 12:42 | |
| bogdando | ok, I'll try it with StackAction | 12:43 |
| shardy | bogdando: You just need to pass an extra environment on update that overrides StackAction: CREATE | 12:43 |
| shardy | so it's StackAction: UPDATE | 12:43 |
| shardy | seems pretty low risk to me? | 12:43 |
| shardy | you just need some way to determing CREATE/UPDATE in the client? | 12:43 |
| shardy | determine | 12:43 |
| bogdando | shardy: my concern for the change is mostly places like http://git.openstack.org/cgit/openstack/puppet-tripleo/tree/manifests/profile/pacemaker/cinder/backup.pp#n64 | 12:44 |
| bogdando | and its impact on undercloud puppet steps | 12:44 |
| shardy | we don't deploy cinder on the undercloud? | 12:45 |
| bogdando | bad example, take manilla | 12:45 |
| bogdando | :) | 12:45 |
| bogdando | yeah, we don't .. yet! | 12:45 |
| bogdando | there is also stack update logic for haproxy bundle | 12:45 |
| shardy | Anyway, surely you just want stack_action set to UPDATE on, uh, UPDATE? | 12:45 |
| shardy | sure, it's in many places | 12:45 |
| shardy | which is why I'm proposing a 1 line fix vs tons of risky puppet changes | 12:45 |
| bogdando | which one? | 12:46 |
| bogdando | I'm lost now | 12:46 |
| shardy | maybe there's a better way | 12:46 |
| shardy | StackAction: UPDATE | 12:46 |
| shardy | done | 12:46 |
| bogdando | ok) | 12:46 |
| *** kopecmartin has quit IRC | 12:47 | |
| bogdando | shardy: > an extra environment on update | 12:48 |
| bogdando | there is no update of stack in undercloud | 12:48 |
| bogdando | we always create a new ephemeral | 12:48 |
| shardy | I know | 12:48 |
| bogdando | so how to detect it>? | 12:48 |
| shardy | but you can detect when it's an update and pass another environment file | 12:48 |
| bogdando | hmmm | 12:48 |
| shardy | well for a start all the openstack stuff will be running | 12:48 |
| *** kopecmartin has joined #tripleo | 12:48 | |
| shardy | or we can drop an explicit state file in the undercloud post script? | 12:48 |
| bogdando | makes sense! | 12:49 |
| shardy | touch /var/lib/undercloud_deploy/something | 12:49 |
| quiquell | bogdando: Have remove me from https://bugs.launchpad.net/tripleo/+bug/1778505 | 12:49 |
| openstack | Launchpad bug 1778505 in tripleo "Undercloud heat installer cannot rely on stack_action" [Critical,Triaged] | 12:49 |
| quiquell | and also alerts ans blocker | 12:49 |
| quiquell | Let's just use the other one to fix it | 12:49 |
| *** chem has joined #tripleo | 12:49 | |
| *** liverpooler has joined #tripleo | 12:50 | |
| bogdando | PTAL folks https://review.openstack.org/#/c/576498/ | 12:50 |
| bogdando | shardy, jaosorior, mandre: ^^ :) | 12:51 |
| *** trozet has quit IRC | 12:53 | |
| shardy | bogdando: looks like it needs a rebase? | 12:54 |
| *** toure|gone is now known as toure | 12:54 | |
| bogdando | oops | 12:54 |
| *** trozet has joined #tripleo | 12:54 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: turn off undercloud idempotency check on fs002 https://review.openstack.org/577809 | 12:58 |
| weshay|ruck | quiquell, mwhahaha ^ | 12:58 |
| *** Haresh has quit IRC | 12:58 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Leverage log_file option to capture more UC logs https://review.openstack.org/576498 | 12:59 |
| quiquell | weshay|ruck: Ok | 13:00 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common master: filter available role by tags https://review.openstack.org/576814 | 13:00 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common master: Add workflow to get available plans https://review.openstack.org/577127 | 13:00 |
| weshay|ruck | bogdando, I'll turn it on in a diff job k? | 13:00 |
| bogdando | weshay|ruck: ok | 13:00 |
| bogdando | weshay|ruck: nailed it! | 13:01 |
| bogdando | :) | 13:01 |
| *** cdearborn has joined #tripleo | 13:03 | |
| bogdando | though, mwhahaha was faster https://www.amsterdamduckstore.com/wp-content/uploads/2015/07/Fireman-red-rubber-duck-Amsterdam-Duck-Store.jpg | 13:04 |
| *** ramishra has quit IRC | 13:04 | |
| *** nyechiel_ has joined #tripleo | 13:05 | |
| *** nyechiel has quit IRC | 13:05 | |
| *** lblanchard has joined #tripleo | 13:06 | |
| *** ramishra has joined #tripleo | 13:06 | |
| *** Guest71763 is now known as honza | 13:07 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Fix non-HA rabbitmq user password updates https://review.openstack.org/577785 | 13:07 |
| *** lblanchard has quit IRC | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1778472 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 13:10 |
| openstack | Launchpad bug 1778472 in tripleo "docker pull failed: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo: received unexpected HTTP status: 503 Service Unavailable" [Critical,Fix committed] - Assigned to Quique Llorente (quiquell) | 13:10 |
| *** Haresh has joined #tripleo | 13:10 | |
| *** bdodd_ has joined #tripleo | 13:14 | |
| *** agopi has joined #tripleo | 13:15 | |
| *** bdodd has quit IRC | 13:16 | |
| *** mjturek has joined #tripleo | 13:16 | |
| *** ansmith has joined #tripleo | 13:17 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: WIP, fix container variables https://review.openstack.org/576904 | 13:18 |
| *** mcornea has joined #tripleo | 13:19 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: WIP, fix container variables https://review.openstack.org/576904 | 13:19 |
| *** kopecmartin has quit IRC | 13:19 | |
| *** pdeore has joined #tripleo | 13:20 | |
| bogdando | shardy: we need some clever stack update detection for output_only | 13:20 |
| bogdando | then ansible playbook won't be run | 13:20 |
| bogdando | playbooks* | 13:20 |
| *** kopecmartin has joined #tripleo | 13:20 | |
| *** pdeore has quit IRC | 13:20 | |
| *** itlinux has quit IRC | 13:22 | |
| *** eck`gone is now known as eck` | 13:26 | |
| *** yprokule has quit IRC | 13:28 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Properly pass id attribute to EnvironmentCheckBox input https://review.openstack.org/577818 | 13:29 |
| *** agopi has quit IRC | 13:29 | |
| *** agopi has joined #tripleo | 13:30 | |
| shardy | bogdando: I wasn't thinking ansible, can't we just have the deploy CLI look for a state file or something? | 13:31 |
| *** skramaja has quit IRC | 13:32 | |
| *** lblanchard has joined #tripleo | 13:32 | |
| *** agopi_ has joined #tripleo | 13:32 | |
| *** bdodd_ has quit IRC | 13:33 | |
| *** bdodd has joined #tripleo | 13:34 | |
| *** agopi has quit IRC | 13:34 | |
| shardy | bogdando: e.g we touch a file at the end of https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/post_deploy/undercloud_post.sh | 13:35 |
| shardy | then look for it in https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L518 | 13:36 |
| *** slaweq has joined #tripleo | 13:36 | |
| shardy | if we find the file, we create an extra tripleoclient-stack-action.yaml env (or just always create it, with different value depending on whether the file is there or not) | 13:36 |
| shardy | maybe I'm missing something but that seems a fairly simple solution to me | 13:36 |
| *** slaweq has quit IRC | 13:41 | |
| *** agopi_ is now known as agopi | 13:44 | |
| *** jaganathan has quit IRC | 13:47 | |
| sri_ | dsneddon: Hi i have couple questions related to networking | 13:47 |
| sri_ | dsneddon, in all networking examples, nic1 is a default for provisioning is that right? if I wanted to use nic2 for provisioning modifying nic1 to nic2 should do that trick? | 13:47 |
| *** zoli is now known as zoli|afk | 13:47 | |
| *** vinaykns has joined #tripleo | 13:52 | |
| *** rbrady has joined #tripleo | 13:54 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Running containerized tempest only in containerized environment https://review.openstack.org/577780 | 13:58 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Generate list of commands to run playbooks https://review.openstack.org/565740 | 13:59 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: run only quickstart in the job w/o preparations https://review.openstack.org/576913 | 13:59 |
| *** jcoufal has joined #tripleo | 14:00 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: [WIP][DNM] test to gather zuul inventory informations https://review.openstack.org/576879 | 14:01 |
| *** lvdombrkr89 has joined #tripleo | 14:01 | |
| *** nyechiel has joined #tripleo | 14:01 | |
| *** nyechiel_ has quit IRC | 14:01 | |
| *** lvdombrkr has quit IRC | 14:04 | |
| *** hjensas has quit IRC | 14:08 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: Allow fast forward upgrade of custom roles https://review.openstack.org/577520 | 14:09 |
| *** zshi has quit IRC | 14:09 | |
| *** quiquell is now known as quiquell|off | 14:10 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 14:10 |
| *** dxiri has joined #tripleo | 14:11 | |
| *** zshi has joined #tripleo | 14:14 | |
| *** mhenkel_ has quit IRC | 14:14 | |
| *** mrsoul_ has joined #tripleo | 14:14 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: DNM: test patch for deps https://review.openstack.org/577830 | 14:15 |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-common master: Let TripleO generate ODLPassword https://review.openstack.org/577831 | 14:16 |
| *** mschuppert has quit IRC | 14:16 | |
| *** mrsoul_ is now known as mschuppert | 14:17 | |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates master: Delete default ODL Password https://review.openstack.org/577834 | 14:20 |
| *** janki has quit IRC | 14:22 | |
| *** rpioso|afk is now known as rpioso | 14:22 | |
| verdurin | Where do I report a bug to request an update to ipxe? My Queens deployments are failing unless I manually use a newer build. | 14:22 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: better handle existing keywords files/directories https://review.openstack.org/577633 | 14:27 |
| *** suuuper has quit IRC | 14:28 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common master: Add workflow to select sample plan https://review.openstack.org/577840 | 14:33 |
| *** ykarel has quit IRC | 14:36 | |
| *** pdeore has joined #tripleo | 14:36 | |
| *** panda|off is now known as panda | 14:36 | |
| *** ykarel has joined #tripleo | 14:38 | |
| *** itlinux has joined #tripleo | 14:39 | |
| *** tcw has quit IRC | 14:42 | |
| *** hjensas has joined #tripleo | 14:42 | |
| *** udesale has joined #tripleo | 14:45 | |
| *** mhenkel_ has joined #tripleo | 14:48 | |
| *** tcw has joined #tripleo | 14:49 | |
| *** zoli|afk is now known as zoli | 14:50 | |
| *** bogdando has quit IRC | 14:51 | |
| *** mhenkel_ has quit IRC | 14:53 | |
| Tengu | so, see you tomorrow :). | 14:53 |
| *** janki has joined #tripleo | 14:55 | |
| *** waleedm has quit IRC | 14:55 | |
| *** ykarel is now known as ykarel|away | 14:55 | |
| *** stendulker has joined #tripleo | 14:57 | |
| asbishop | mwhahaha: ci/environments/ovb-ha.yaml doesn't list any cinder services?!? deliberate or oversight? | 14:57 |
| * mwhahaha checks | 14:58 | |
| mwhahaha | asbishop: it's probably deliberate | 14:59 |
| mwhahaha | asbishop: we only test cinder in scenario001/002 | 15:00 |
| mwhahaha | asbishop: https://github.com/openstack/tripleo-heat-templates/blob/master/README.rst#service-testing-matrix | 15:00 |
| asbishop | mwhahaha: ok, at long as it's deliberate :-/ | 15:02 |
| *** hkominos has left #tripleo | 15:03 | |
| mwhahaha | asbishop: well it's deliberate in that the goal of the ovb testing is to exercise the iamge deployment and some of the ha functionality. if adding cinder doesn't break anything and there's something you want to test we can change it | 15:03 |
| mwhahaha | asbishop: unfortunately resources being scarce upstream we don't test everything | 15:03 |
| asbishop | mschuppert: people ask me because a "minimal" tripleo-quickstart succeeds, but w/ no block storage service | 15:03 |
| *** vinaykns has left #tripleo | 15:03 | |
| mwhahaha | asbishop: right so that's not something we test upstream (it should have downstream coverage) | 15:03 |
| asbishop | mwhahaha: if goal is HA then remember that cinder-volume has a pacemaker deployment variant | 15:04 |
| mwhahaha | asbishop: honestly it's a downstream test, like i said you can try and add it and if doesn't have repercussions to the deployment times, etc then we can cover it upstream | 15:05 |
| *** rh-jelabarre has quit IRC | 15:05 | |
| mwhahaha | we can't test everything | 15:05 |
| mwhahaha | we don't have the node counts to deploy the blockstorage role | 15:06 |
| mwhahaha | so that's why it's not something we'd cover upstream | 15:06 |
| asbishop | mwhahaha: np, and ack. Just wanted to close the loop so I understood situation | 15:06 |
| *** nyechiel has quit IRC | 15:08 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 15:10 |
| *** pcaruana has quit IRC | 15:13 | |
| *** ykarel|away has quit IRC | 15:14 | |
| *** rh-jelabarre has joined #tripleo | 15:18 | |
| mwhahaha | uh oh | 15:18 |
| mwhahaha | i think the new mistral hit us | 15:18 |
| d0ugal | uh oh | 15:18 |
| *** moguimar has quit IRC | 15:18 | |
| mwhahaha | er i mean ansible | 15:18 |
| d0ugal | phew | 15:19 |
| * d0ugal comes back out of hiding | 15:19 | |
| shardy | :D | 15:19 |
| mwhahaha | but it is related to mistral | 15:19 |
| mwhahaha | the packaging uses a bad home dir | 15:19 |
| * mwhahaha shakes fist at mistral | 15:19 | |
| * d0ugal shakes his fist at packagers | 15:20 | |
| *** saneax has quit IRC | 15:20 | |
| d0ugal | mwhahaha: I don't think the mistral user needs a home dir | 15:20 |
| mwhahaha | d0ugal: it does when it runs ansible | 15:21 |
| d0ugal | so I'm not sure why it would have one | 15:21 |
| d0ugal | oh | 15:21 |
| mwhahaha | shell execution passes $HOME and if the thing doesn't exist it's a problem | 15:21 |
| mwhahaha | so it should be /var/lib/mistral | 15:21 |
| mwhahaha | but anyway | 15:21 |
| d0ugal | Right | 15:21 |
| *** moguimar has joined #tripleo | 15:21 | |
| d0ugal | mwhahaha: is there a patch/bug or is somebody handling it? | 15:22 |
| mwhahaha | i'm trying to handle it from a tripleo standpoint | 15:22 |
| mwhahaha | https://review.openstack.org/#/c/577544/ | 15:22 |
| shardy | d0ugal: re https://review.openstack.org/#/c/528213/33/workbooks/swift.yaml - do we need to do the same to check if an object exists? | 15:23 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Switch ansible tmp for local connections https://review.openstack.org/577544 | 15:23 |
| *** dciabrin_ has joined #tripleo | 15:23 | |
| shardy | I was planning to just get the object and handle when it fails, as getting the whole container listing sounded more expensive | 15:23 |
| *** dciabrin has quit IRC | 15:23 | |
| shardy | d0ugal: I want to check if there is a root plan-environment.yaml and create one from plan-samples when it's missing | 15:24 |
| shardy | building on the workflow added in https://review.openstack.org/#/c/577840/ | 15:24 |
| d0ugal | shardy: Yeah, we might want something similar. I assume get_object will add tracebacks to the logs if it doesn't exist yet. | 15:24 |
| *** ksambor has quit IRC | 15:25 | |
| shardy | d0ugal: ack Ok will add something | 15:25 |
| d0ugal | mwhahaha: that fix looks good to me, I don't think there is anything we can do in Mistral to help with this | 15:25 |
| mwhahaha | d0ugal: yea it's a packaging thing, the problem is upgrades will have the old /home/mistral | 15:25 |
| mwhahaha | i'll propose a patch to the packaging to correct the home dir going forward | 15:26 |
| d0ugal | mwhahaha: I was wondering how upgrades would work (or not) | 15:26 |
| mwhahaha | they won't which is why we'll probably have to backport this as well | 15:26 |
| *** karthiks has quit IRC | 15:27 | |
| d0ugal | Right | 15:28 |
| d0ugal | Let me know if I can help at all | 15:28 |
| *** bugzy_ is now known as bugzy | 15:35 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: Allow fast forward upgrade of custom roles https://review.openstack.org/577520 | 15:37 |
| shardy | d0ugal: why is it swiftservice in your new workbook but swift everywhere else? | 15:37 |
| *** stendulker has quit IRC | 15:38 | |
| *** pdeore has quit IRC | 15:38 | |
| *** apetrich has quit IRC | 15:42 | |
| openstackgerrit | yolanda.robla proposed openstack/python-tripleoclient master: Allow to skip not existing images when uploading https://review.openstack.org/577799 | 15:42 |
| *** apetrich has joined #tripleo | 15:45 | |
| *** matbu has quit IRC | 15:45 | |
| myoung | mwhahaha, weshay|ruck, trown: could use eyes, attempting to get doc updates for tempest squad merged: https://review.openstack.org/#/c/565161 | 15:46 |
| *** matbu has joined #tripleo | 15:46 | |
| weshay|ruck | k | 15:46 |
| *** ramishra has quit IRC | 15:47 | |
| janki | mwhahaha hey | 15:47 |
| mwhahaha | janki: what's up? | 15:47 |
| janki | mwhahaha, could you please add me to the tripleo launchpad group so that I can triage the bugs that I report. | 15:48 |
| mwhahaha | janki: i think you can request access | 15:48 |
| *** dprince has quit IRC | 15:49 | |
| janki | mwhahaha, how? I saw this "If you can't update these informations, it's because you're not member of TripleO in Launchpad group. Please ping the PTL and you'll be added." in the dialogue box where filing bug today morning and just pinged you here | 15:49 |
| mwhahaha | janki: or i can add you, what's your launchpad user? | 15:49 |
| *** stendulker has joined #tripleo | 15:49 | |
| janki | mwhahaha, Launchpad Id: jankihchhatbar | 15:50 |
| mwhahaha | janki: k added | 15:50 |
| janki | mwhahaha, thanks too much :) | 15:50 |
| *** dparkes has quit IRC | 15:57 | |
| *** dparkes has joined #tripleo | 15:58 | |
| *** dtantsur is now known as dtantsur|afl | 15:59 | |
| *** dtantsur|afl is now known as dtantsur|afk | 15:59 | |
| *** hjensas is now known as hjensas|afk | 16:00 | |
| openstackgerrit | Andy Smith proposed openstack/tripleo-heat-templates master: Update scenario003 to deploy separate messaging backends https://review.openstack.org/566483 | 16:00 |
| *** rbrady is now known as rbrady-afk | 16:01 | |
| *** brault has quit IRC | 16:02 | |
| *** zoli is now known as zoli|gone | 16:03 | |
| *** zoli|gone is now known as zoli | 16:03 | |
| *** dparkes has quit IRC | 16:04 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Improve nova statedir ownership logic https://review.openstack.org/577855 | 16:06 |
| beagles | anybody know what takes care of shutting down containers on shutdown/reboot? | 16:07 |
| *** gfidente has quit IRC | 16:08 | |
| *** ykarel|away has joined #tripleo | 16:09 | |
| d0ugal | shardy: swiftservice is just another set of actions that uses https://github.com/openstack/python-swiftclient/blob/master/swiftclient/service.py | 16:09 |
| *** lvdombrkr89 has quit IRC | 16:09 | |
| *** agurenko has quit IRC | 16:09 | |
| d0ugal | shardy: there is some higher level functionality there that isn't available otherwise iirc | 16:09 |
| d0ugal | shardy: I think I actually didn't need it in that case in the end, but I thought I did - I forget now, that patch has been up for a while. | 16:10 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 16:10 |
| shardy | d0ugal: ack, I just wasn't sure if I should copy it, I'm just doing a swiftservice.get_container so AFAIK that should work fine with the normal swift client | 16:10 |
| *** leanderthal has quit IRC | 16:10 | |
| *** mjturek has quit IRC | 16:12 | |
| d0ugal | Yup, I think so | 16:13 |
| shardy | d0ugal: ooh, looks like swiftclient/service.py has some useful stuff in it e.g SwiftCopyObject, which I just reimplemented in a workflow | 16:13 |
| shardy | I'll try that and see if it can replace the subworkflow | 16:14 |
| d0ugal | shardy: yup, it is quite nice. I discovered it when I noticed dprince use it somewhere else. | 16:14 |
| d0ugal | shardy: I added these to Mistral: https://github.com/openstack/mistral/blob/master/mistral/actions/openstack/mapping.json#L960-L967 | 16:14 |
| shardy | d0ugal: ack thanks for the info, was initially a bit confusing :) | 16:14 |
| d0ugal | shardy: btw, just FYI, swiftservice is new to Mistral in Rocky. | 16:14 |
| shardy | ah that explains why I've not spotted it before | 16:15 |
| d0ugal | It has been in swiftclient for longer, 4 years it seems! | 16:15 |
| *** stendulker has quit IRC | 16:15 | |
| shardy | why have just 1 client binding when you can have two I guess ;) | 16:16 |
| *** brault has joined #tripleo | 16:17 | |
| *** tesseract has quit IRC | 16:18 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/pike: Add host prep step for ntp time sync https://review.openstack.org/577861 | 16:20 |
| *** derekh has quit IRC | 16:24 | |
| *** ykarel|away has quit IRC | 16:24 | |
| *** shardy has quit IRC | 16:25 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Rename default stack name for standalone https://review.openstack.org/577864 | 16:28 |
| *** jpich has quit IRC | 16:29 | |
| *** ksambor has joined #tripleo | 16:30 | |
| openstackgerrit | Christopher Brown proposed openstack/tripleo-heat-templates master: Fixes InstanceNameTemplate https://review.openstack.org/577869 | 16:43 |
| *** jpena is now known as jpena|off | 16:43 | |
| *** dprince has joined #tripleo | 16:45 | |
| *** quiquell|off has quit IRC | 16:47 | |
| *** dsneddon has quit IRC | 16:48 | |
| *** dprince has quit IRC | 16:53 | |
| slagle | mandre: have you seen the feedback on https://review.openstack.org/#/c/566246 ? | 16:53 |
| *** dprince has joined #tripleo | 16:54 | |
| *** amoralej is now known as amoralej|off | 16:56 | |
| *** invsblduck has joined #tripleo | 16:56 | |
| *** udesale has quit IRC | 16:56 | |
| invsblduck | any contrail users in here? wondering what the state of contrail support is like with tripleo for queens release... | 16:59 |
| *** psachin` has quit IRC | 16:59 | |
| invsblduck | (we've sucessfully used non-containerized 4.x with tripleo newton) | 17:00 |
| mwhahaha | invsblduck: probably a question for juniper. i know they were working on it so it should work. | 17:01 |
| invsblduck | mwhahaha: yea, figured. just found #opencontrail with /list, so going to ask there after i look at their github repo. :P ty! | 17:03 |
| invsblduck | mwhahaha: when rhosp 13 coming out of rc??! eventual goal is to be on that with official juniper contrail.. | 17:04 |
| mwhahaha | when it's ready? :D | 17:05 |
| invsblduck | maybe that's a slagle question.. who works at rh?! >_< | 17:05 |
| invsblduck | hehe. | 17:05 |
| *** ffiore has quit IRC | 17:06 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Add release note about Designate https://review.openstack.org/577878 | 17:09 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 17:10 |
| *** yamahata has quit IRC | 17:22 | |
| *** kopecmartin has quit IRC | 17:27 | |
| *** srini has joined #tripleo | 17:27 | |
| *** dprince has quit IRC | 17:27 | |
| slagle | invsblduck: we don't comment on any downstream release dates in #tripleo | 17:28 |
| invsblduck | slagle: smart move. ty. | 17:31 |
| *** myoung is now known as myoung|biab | 17:33 | |
| *** salmankhan has quit IRC | 17:33 | |
| *** mhenkel_ has joined #tripleo | 17:35 | |
| *** mhenkel_ has joined #tripleo | 17:36 | |
| *** rbrady-afk is now known as rbrady | 17:36 | |
| *** pblaho has quit IRC | 17:40 | |
| *** yamahata has joined #tripleo | 17:42 | |
| *** trown is now known as trown|lunch | 17:43 | |
| *** mhenkel_ has quit IRC | 17:47 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix syntax for set_fact ansible task. https://review.openstack.org/577725 | 17:55 |
| *** janki has quit IRC | 17:59 | |
| *** Haresh has quit IRC | 18:02 | |
| openstackgerrit | Paul Belanger proposed openstack/tripleo-quickstart master: WIP: Enable pipelining for ansible https://review.openstack.org/577889 | 18:03 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Filter qrouter containers from images validation script https://review.openstack.org/577459 | 18:03 |
| *** dciabrin_ has quit IRC | 18:04 | |
| *** dciabrin_ has joined #tripleo | 18:05 | |
| *** ksambor has quit IRC | 18:05 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade stable/queens: Filter qrouter containers from images validation script https://review.openstack.org/577890 | 18:05 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 18:10 |
| beagles | mandre, slagle, do you know what happens with containers when you shut down a node? Are they killed violently or shutdown? | 18:13 |
| beagles | s/shutdown/gracefully shutdown/ | 18:14 |
| slagle | beagles: no. i'd hope they were shutdown gracefully. i fully expect they aren't | 18:16 |
| beagles | slagle, ack - I was thinking that even if they magically "docker stop" and they are slow to stop, then that is going to turn into a "kill" | 18:18 |
| *** karthiks has joined #tripleo | 18:18 | |
| beagles | slagle, even so, I don't see a mechanism that shuts them all down gracefully - unless there is some docker service magic | 18:18 |
| *** mjturek has joined #tripleo | 18:18 | |
| mwhahaha | it's containers, they don't have these kinda concepts | 18:19 |
| slagle | for realz | 18:19 |
| slagle | "containers are linux" | 18:19 |
| slagle | you just shut it down | 18:19 |
| *** hjensas|afk is now known as hjensas | 18:19 | |
| beagles | heh | 18:20 |
| mwhahaha | techincally a container stop would be the graceful shutdown | 18:21 |
| mwhahaha | but it's unlikely that we catch the signal when that happens | 18:21 |
| mwhahaha | might check with kolla to see if they catch the sigterm | 18:22 |
| beagles | mwhahaha, ack - I'm wondering if we even get that on system shutdown | 18:22 |
| slagle | aren't all processes are sent SIGQUIT or something on system shutdown | 18:22 |
| mwhahaha | so yea docker should send a stop | 18:22 |
| beagles | maybe systemd comes along and sends signals | 18:22 |
| mwhahaha | on a graceful shutdown | 18:22 |
| mwhahaha | but it might not | 18:22 |
| slagle | docker won't | 18:22 |
| mwhahaha | https://www.ctl.io/developers/blog/post/gracefully-stopping-docker-containers/ | 18:22 |
| slagle | but systemd should SIGQUIT/SIGTERM every process on the system | 18:22 |
| slagle | and hopefully whatever process is in the contaier handles those signals | 18:23 |
| mwhahaha | i don't see any sigterm references in kolla | 18:25 |
| mwhahaha | so we don't appear to be doing any extra bits there | 18:25 |
| mwhahaha | so my assumption is that containers aren't nicely shutting down on system shutdown | 18:25 |
| mwhahaha | though you could probably test it by stopping docker and sigterming a container | 18:26 |
| mwhahaha | we set the services to always restart so kill doesn't necessarily work right | 18:27 |
| beagles | I was trying to spot it by watching the VMs console but it went by too fast - have to set it up to log console to a file | 18:27 |
| beagles | mwhahaha, good point | 18:27 |
| mwhahaha | it should be in the service log file tho | 18:28 |
| mwhahaha | looks like docker stop is what you'd want ot test | 18:29 |
| *** dxiri has quit IRC | 18:29 | |
| mwhahaha | looks like it'll only wait 10 seconds for it to stop before killing it | 18:29 |
| *** mjturek has quit IRC | 18:30 | |
| openstackgerrit | Toure Dunnon proposed openstack/tripleo-common master: Send a zaqar message with 'list_validations' result https://review.openstack.org/563701 | 18:30 |
| beagles | yup | 18:30 |
| *** dciabrin_ has quit IRC | 18:32 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Mixed version R/Q deploy -- don't use config download for upgrade. https://review.openstack.org/573717 | 18:37 |
| *** sshnaidm is now known as sshnaidm|afk | 18:41 | |
| *** dciabrin_ has joined #tripleo | 18:42 | |
| *** karthiks has quit IRC | 18:42 | |
| *** jcoufal has quit IRC | 18:43 | |
| *** mjturek has joined #tripleo | 18:47 | |
| *** jcoufal has joined #tripleo | 18:53 | |
| *** atoth has quit IRC | 18:58 | |
| *** pchavva has quit IRC | 19:03 | |
| *** dxiri has joined #tripleo | 19:08 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 19:10 |
| *** dpeacock has quit IRC | 19:15 | |
| *** dprince has joined #tripleo | 19:18 | |
| *** dpeacock has joined #tripleo | 19:18 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Improve nova statedir ownership logic https://review.openstack.org/577855 | 19:20 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Improve nova statedir ownership logic https://review.openstack.org/577855 | 19:21 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: WIP: prevent upgrade from baremetal to containers with NFS backend https://review.openstack.org/577907 | 19:24 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: QDR for metrics collection purposes https://review.openstack.org/572312 | 19:25 |
| *** pcaruana has joined #tripleo | 19:27 | |
| *** slaweq has joined #tripleo | 19:29 | |
| *** dprince has quit IRC | 19:35 | |
| *** aufi has quit IRC | 19:36 | |
| *** srini has quit IRC | 19:37 | |
| *** trown|lunch is now known as trown | 19:42 | |
| weshay|ruck | need reviews on https://review.openstack.org/#/c/576990/ | 19:44 |
| weshay|ruck | please | 19:44 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: QDR for metrics collection purposes https://review.openstack.org/572312 | 19:50 |
| *** pchavva has joined #tripleo | 19:55 | |
| *** dsneddon has joined #tripleo | 19:56 | |
| hjensas | slagle: around? | 19:56 |
| slagle | hjensas: hi | 19:57 |
| hjensas | slagle: just a quick question, you might know ... Is this a ok pattern? http://paste.openstack.org/show/724257/ | 19:58 |
| *** liverpooler has quit IRC | 19:58 | |
| hjensas | slagle: see how the role file is loaded in the middle of the environment files? could that mess up the environment somehow? | 19:58 |
| slagle | hjensas: i don't see how that would cause any issues | 19:58 |
| slagle | the same environment is listed twice there fwiw | 19:59 |
| hjensas | slagle: for context, the deployment tried to change the network of the internalApi ports on update. | 19:59 |
| slagle | hjensas: the full commands would have to be compared, and all templates | 20:00 |
| hjensas | slagle: ah, yes there is a duplicate. I'll mention that to them as well. It's some infrared automation ... | 20:00 |
| *** yolanda has quit IRC | 20:01 | |
| *** yolanda has joined #tripleo | 20:01 | |
| hjensas | slagle: yeah, I need to request more logs from the undercloud. But now I can atleast drop the roles file being in the midst of environments and move on. Thanks! | 20:01 |
| *** florianf has quit IRC | 20:03 | |
| *** atarlov has joined #tripleo | 20:05 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: QDR for metrics collection purposes https://review.openstack.org/572312 | 20:06 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 20:10 |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| *** myoung|biab is now known as myoung | 20:17 | |
| *** milan has quit IRC | 20:17 | |
| *** paramite has quit IRC | 20:26 | |
| *** nyechiel has joined #tripleo | 20:28 | |
| *** ansmith has quit IRC | 20:33 | |
| *** raildo has quit IRC | 20:37 | |
| *** mjturek has quit IRC | 20:38 | |
| *** dsneddon has quit IRC | 20:43 | |
| *** pcaruana has quit IRC | 20:43 | |
| *** dxiri has quit IRC | 20:50 | |
| *** dxiri has joined #tripleo | 20:52 | |
| *** slaweq has quit IRC | 20:57 | |
| *** asbishop has quit IRC | 20:59 | |
| *** lblanchard has quit IRC | 20:59 | |
| *** trown is now known as trown|outtypewww | 21:00 | |
| *** colonwq has quit IRC | 21:02 | |
| *** EmilienM has joined #tripleo | 21:03 | |
| *** EmilienM_PTO has quit IRC | 21:04 | |
| *** EmilienM has quit IRC | 21:04 | |
| *** EmilienM has joined #tripleo | 21:04 | |
| *** ChanServ sets mode: +v EmilienM | 21:04 | |
| *** vpickard is now known as vpickard_ | 21:05 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: turn off undercloud idempotency check on fs002 https://review.openstack.org/577809 | 21:06 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: turn off undercloud idempotency check on fs002 https://review.openstack.org/577809 | 21:08 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 21:10 |
| *** colonwq has joined #tripleo | 21:15 | |
| *** nyechiel has quit IRC | 21:18 | |
| *** matbu has quit IRC | 21:22 | |
| *** ansmith has joined #tripleo | 21:26 | |
| *** dsneddon has joined #tripleo | 21:27 | |
| *** agopi has quit IRC | 21:31 | |
| *** atarlov has quit IRC | 21:34 | |
| *** atarlov has joined #tripleo | 21:35 | |
| *** dsneddon has quit IRC | 21:35 | |
| *** atarlov has quit IRC | 21:39 | |
| *** rcernin has joined #tripleo | 21:45 | |
| *** itlinux has quit IRC | 21:45 | |
| *** atarlov has joined #tripleo | 21:47 | |
| *** bfournie has quit IRC | 21:48 | |
| *** atarlov has quit IRC | 21:50 | |
| *** atarlov has joined #tripleo | 21:51 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Make BIND /var dir persistent https://review.openstack.org/575823 | 21:53 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Expose dnsmasq_local_resolv option https://review.openstack.org/577936 | 21:53 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Disable recursion in Designate-managed BIND https://review.openstack.org/577937 | 21:53 |
| *** dsneddon has joined #tripleo | 21:53 | |
| *** atarlov has quit IRC | 21:55 | |
| *** edmondsw has quit IRC | 21:58 | |
| openstackgerrit | Andreas Karis proposed openstack/tripleo-heat-templates master: Lower reserved memory for nova-compute https://review.openstack.org/577938 | 22:04 |
| *** pchavva has quit IRC | 22:06 | |
| *** cdearborn has quit IRC | 22:06 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 22:10 |
| *** yolanda_ has joined #tripleo | 22:21 | |
| *** yolanda has quit IRC | 22:22 | |
| *** yolanda__ has joined #tripleo | 22:28 | |
| *** yolanda_ has quit IRC | 22:31 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: Adjust roles_data during FFU https://review.openstack.org/558081 | 22:36 |
| *** bfournie has joined #tripleo | 22:40 | |
| *** bfournie has quit IRC | 22:41 | |
| *** bfournie has joined #tripleo | 22:42 | |
| *** pchavva has joined #tripleo | 22:51 | |
| *** edmondsw has joined #tripleo | 22:57 | |
| *** agopi has joined #tripleo | 22:58 | |
| *** invsblduck has quit IRC | 23:03 | |
| openstackgerrit | Steve Baker proposed openstack/instack-undercloud master: Add the undercloud mistral user to the docker group https://review.openstack.org/566768 | 23:06 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1777939 | 23:10 |
| openstack | Launchpad bug 1777939 in tripleo "Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED " [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| *** invsblduck has joined #tripleo | 23:11 | |
| *** pmannidi has joined #tripleo | 23:14 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Create zuul-hosts file for pre.yaml network setup https://review.openstack.org/577007 | 23:25 |
| *** marrusl has quit IRC | 23:30 | |
| *** tosky has quit IRC | 23:34 | |
| *** ohochman has quit IRC | 23:48 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Use ACL instead of docker group for mistral https://review.openstack.org/577946 | 23:49 |
| *** mjturek has joined #tripleo | 23:52 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Action to perform container image prepare https://review.openstack.org/558972 | 23:52 |
| *** rh-jelabarre has quit IRC | 23:53 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: Switch to workflow driven image prepare https://review.openstack.org/573476 | 23:54 |
| *** rpioso is now known as rpioso|afk | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
