| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/pike: GATE CHECK for TripleO https://review.openstack.org/602248 | 00:00 |
|---|---|---|
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 00:00 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Tag container image prepare tasks to allow running them for updates/upgrades https://review.openstack.org/609681 | 00:01 |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,Triaged] - Assigned to Toure Dunnon (toure) | 00:10 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: WIP replace skopeo inspect with python https://review.openstack.org/609586 | 00:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Fix list concatenation of routes in bond-with-vlan https://review.openstack.org/609481 | 00:34 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Minor documentation updates https://review.openstack.org/609636 | 00:52 |
| *** huynq has joined #tripleo | 00:56 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Honor --skip-deploy-identifier in common deploy tasks https://review.openstack.org/609038 | 01:00 |
| openstackgerrit | Merged openstack/tripleo-ui master: Removed older version of python added 3.5 https://review.openstack.org/606427 | 01:00 |
| openstackgerrit | Merged openstack/tripleo-common stable/rocky: Run prepare during package_update workflow https://review.openstack.org/609718 | 01:03 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a pre-finalise.d phase https://review.openstack.org/609863 | 01:06 |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,Triaged] - Assigned to Toure Dunnon (toure) | 01:10 |
| itlinux_ | can someone point me on what this error could be.. and how to get it fixed thanks http://paste.openstack.org/show/731929/ | 01:10 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: WIP replace skopeo inspect with python https://review.openstack.org/609586 | 01:11 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: provider: Add vexxhost https://review.openstack.org/596432 | 01:12 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: NODES_FILE definition is missing https://review.openstack.org/609846 | 01:16 |
| *** rh-jelabarre has joined #tripleo | 01:29 | |
| *** mrsoul has joined #tripleo | 01:30 | |
| *** Chaserjim has joined #tripleo | 01:30 | |
| *** mschuppert has quit IRC | 01:33 | |
| *** lblanchard has joined #tripleo | 01:35 | |
| *** Chaserjim has quit IRC | 01:35 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 02:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,Triaged] - Assigned to Toure Dunnon (toure) | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| *** apetrich has quit IRC | 02:15 | |
| *** psachin has joined #tripleo | 02:54 | |
| *** rlandy|bbl is now known as rlandy | 03:00 | |
| *** rlandy has quit IRC | 03:02 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 03:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,Triaged] - Assigned to Toure Dunnon (toure) | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| *** lblanchard has quit IRC | 03:28 | |
| *** udesale has joined #tripleo | 03:54 | |
| *** mmethot_ has joined #tripleo | 04:04 | |
| *** matbu has quit IRC | 04:06 | |
| *** matbu has joined #tripleo | 04:06 | |
| *** mmethot has quit IRC | 04:06 | |
| *** gchamoul has quit IRC | 04:06 | |
| *** jaganathan has joined #tripleo | 04:08 | |
| *** gchamoul has joined #tripleo | 04:08 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 04:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,Triaged] - Assigned to Toure Dunnon (toure) | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| *** tzumainn has quit IRC | 04:13 | |
| *** EmilienM is now known as EvilienM | 04:13 | |
| *** ykarel__ has joined #tripleo | 04:22 | |
| *** gchamoul has quit IRC | 04:30 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Switch the undercloud to deploy Podman by default starting from Stein https://review.openstack.org/608452 | 04:36 |
| EvilienM | oh bot is back | 04:36 |
| *** ykarel__ is now known as ykarel | 04:38 | |
| ykarel | looks like it's podman effect :) | 04:39 |
| itlinux_ | EvilienM: hi I am working to convert a Pike BM to container I get this gnocchi issue http://paste.openstack.org/show/731935/ can you suggest something to look in order to get it working. Thanks | 04:41 |
| *** ramishra has joined #tripleo | 04:48 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Add manila and sahara tests to tempest skip list for fs020 https://review.openstack.org/609741 | 04:54 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Create venv only when virtualenv is not there https://review.openstack.org/602347 | 04:57 |
| Tengu | hello there | 04:59 |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 05:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,Triaged] - Assigned to Toure Dunnon (toure) | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| *** ssbarnea has quit IRC | 05:11 | |
| *** janki has joined #tripleo | 05:15 | |
| *** pcaruana has joined #tripleo | 05:15 | |
| Tengu | jaosorior: hello! I've just created snapshots of the "first deploy" we made yesterday so we can rollback at will. | 05:16 |
| *** rh-jelabarre has quit IRC | 05:22 | |
| *** ratailor has joined #tripleo | 05:25 | |
| *** ssbarnea has joined #tripleo | 05:30 | |
| *** dr_gogeta86 has quit IRC | 05:32 | |
| *** chkumar|off is now known as chandankumar | 05:34 | |
| jaosorior | Tengu: excellent | 05:38 |
| jaosorior | Tengu: so, the first deployment is with pacemaker, right? | 05:39 |
| Tengu | jaosorior: while reviewing the BZ I saw the custom env files where based on deprecated versions. | 05:39 |
| Tengu | jaosorior: yeah, there's the docker-ha thingy at least | 05:39 |
| Tengu | jaosorior: and I suspect the include order also affects the behavior, once more. | 05:40 |
| jaosorior | it does | 05:40 |
| jaosorior | so, what do you think the issue is then? | 05:40 |
| Tengu | jaosorior: a mix of deprecated env file + wrong include order. | 05:41 |
| Tengu | the enable-tls.yaml is deprecated as per its very header, and it should probably NOT be at the end of the inclusion because of overriding issues. | 05:41 |
| Tengu | fact is, we should get the haproxy container logs. | 05:41 |
| Tengu | will fetch the sosreport. | 05:42 |
| jaosorior | ok | 05:42 |
| jaosorior | Tengu: anyway, the next step is to try to reproduce the error (even if we use the correct files) | 05:42 |
| *** numans has joined #tripleo | 05:42 | |
| jaosorior | so, lets try that out | 05:42 |
| Tengu | jaosorior: you can connect to the tmate and try it while I'm looking for the sosreport+logs ? | 05:43 |
| jaosorior | sure | 05:44 |
| janki | Tengu, hey | 05:46 |
| jaosorior | Tengu: no route to host | 05:46 |
| jaosorior | wtf | 05:46 |
| jaosorior | Tengu: to the overcloud | 05:47 |
| *** mburned_out is now known as mburned | 05:47 | |
| Tengu | weird. the node is up... | 05:49 |
| Tengu | janki: hey :) | 05:49 |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade stable/rocky: Replace RabbitMQ with OsloMessaging(Rpc|Notify) https://review.openstack.org/609729 | 05:49 |
| janki | Tengu, should I recheck now? | 05:49 |
| Tengu | jaosorior: and vbmc/ironic seem to be happy.... no clue | 05:49 |
| Tengu | janki: errrr what change? | 05:50 |
| janki | Tengu, https://review.openstack.org/#/c/586251/ and https://review.openstack.org/#/c/604750/ | 05:50 |
| Tengu | janki: ah, yeah, guess so, gate merged stuff this night | 05:51 |
| janki | Tengu, ack. done. fingers crossed :P | 05:51 |
| Tengu | jaosorior: rebooting the lab-controller-0 - might be due to undercloud not being ready to provide dhcp... | 05:52 |
| *** skramaja has joined #tripleo | 05:53 | |
| Tengu | jaosorior: o_O firewall's blocking on the controller-0 node. like... err.. wtf ?! | 05:53 |
| Tengu | so a reboot might help a bit. | 05:53 |
| Tengu | jaosorior: so... basically... the node doesn't have its IP... | 05:56 |
| Tengu | jaosorior: ctlplane doesn't seem to be present for some reason on the controller node. yay. | 05:57 |
| jaosorior | that makes no sense | 05:58 |
| Tengu | jaosorior: that's not really convenient, I'd say. | 05:58 |
| Tengu | jaosorior: will drop the stack and redeploy it -.-. for some reason, it's not correctly set up :/. | 06:00 |
| *** ykarel has quit IRC | 06:01 | |
| *** ykarel has joined #tripleo | 06:02 | |
| *** kopecmartin|ruck has quit IRC | 06:04 | |
| *** apetrich has joined #tripleo | 06:04 | |
| *** kopecmartin has joined #tripleo | 06:05 | |
| *** Petersingh has joined #tripleo | 06:05 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 06:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,Triaged] - Assigned to Toure Dunnon (toure) | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| jaosorior | Tengu: my deployment is working | 06:12 |
| jaosorior | I'm attempting an update with TLS | 06:12 |
| Tengu | jaosorior: funky. well. anyway. | 06:12 |
| Tengu | I'm trying to find info in the logs, but apparently we don't have anything for an "haproxy" container..... | 06:12 |
| Tengu | so that won't help. | 06:12 |
| jaosorior | marios|rover: can you check this out https://review.openstack.org/#/c/609746/2 ? | 06:13 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Add specific tempest tests for puppet project https://review.openstack.org/609918 | 06:13 |
| chandankumar | Tengu: jaosorior quiquell|off ^^ | 06:13 |
| chandankumar | just wanted to check I am in the right direction or not | 06:14 |
| *** ksambor has joined #tripleo | 06:14 | |
| jaosorior | bandini: can you check this out https://review.openstack.org/#/c/609746/2 ? | 06:16 |
| *** mschuppert has joined #tripleo | 06:17 | |
| *** janki has quit IRC | 06:19 | |
| marios | jaosorior: i set +A but unset it as i saw you ran a check rdo | 06:20 |
| *** mburned is now known as mburned_out | 06:26 | |
| Tengu | chandankumar: can't say, not really an expert with the CI config :/ | 06:28 |
| Tengu | nor quickstart featureset | 06:28 |
| Tengu | jaosorior: should I abandon https://review.openstack.org/#/c/570841/ ? | 06:30 |
| jaosorior | marios: yeah, I'll wait for the ovb result before merging | 06:31 |
| *** janki has joined #tripleo | 06:34 | |
| jaosorior | marios: any idea if there's something wrong with RDO cloud? | 06:35 |
| marios | jaosorior: not aware of something/havent seen issues yet | 06:35 |
| jaosorior | marios: the patch already failed with this: 2018-10-12 06:25:10.669423 | primary | 2018-10-12 06:25:10,668 - testenv-client - ERROR - Couldn't retrieve env | 06:35 |
| marios | :/ | 06:35 |
| marios | that patch is cursed | 06:36 |
| jaosorior | although | 06:36 |
| jaosorior | there is a dependent patch | 06:36 |
| jaosorior | that did run ovb https://review.openstack.org/#/c/608589/ | 06:36 |
| jaosorior | and it passed | 06:36 |
| jaosorior | so... I'm tempted on just merging the mistral patch | 06:36 |
| marios | jaosorior: oh i thought you were referring to that /#/c/608589/ | 06:36 |
| jaosorior | marios: that one passed in ovb | 06:37 |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates stable/rocky: Add posibilities to set tunnel_csum in ovs agent https://review.openstack.org/609923 | 06:37 |
| jaosorior | it failed pike... but passed the rest | 06:37 |
| marios | jaosorior: ack (yeah looking) | 06:37 |
| marios | k and its in the gate lets see | 06:37 |
| marios | i was laughing at weshay comments :) | 06:38 |
| marios | "recheck omg" | 06:38 |
| marios | "recheck weeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" | 06:38 |
| jaosorior | lol | 06:38 |
| *** dciabrin has joined #tripleo | 06:39 | |
| jaosorior | Tengu: reproduced the issue | 06:49 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-common master: Fix nova-api healthcheck in case of nova metadata wsgi https://review.openstack.org/609927 | 06:49 |
| ykarel | marios, jaosorior i seen one unreachable issue in promotion job, so looks there is some issue in RDO cloud | 06:49 |
| ykarel | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-rocky/6e27361/job-output.txt.gz#_2018-10-12_06_32_03_116500 | 06:49 |
| Tengu | jaosorior: ah, cool. with the same order than the BZ? | 06:50 |
| jaosorior | Tengu: kinda | 06:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/609928 | 06:51 |
| jaosorior | Tengu: tmate and bluejeans? | 06:51 |
| Tengu | jaosorior: yep, gimme the links :) | 06:51 |
| *** kopecmartin is now known as kopecmartin|ruck | 06:52 | |
| Tengu | marios: yeah, also saw weshay comments :). They kind of reflect my thoughts lately ;) | 06:52 |
| jaosorior | Tengu: gonna brew some coffee | 06:52 |
| jaosorior | bandini: where do the pacemaker container resource definitions get stored in the overcloud nodes? | 06:52 |
| Tengu | jaosorior: already have my coffee :D. no idea for the container resources themselves. but in some volume ? | 06:53 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Provide network name associated with the role to ServiceNetMap https://review.openstack.org/609929 | 06:56 |
| bandini | jaosorior: in the cluster CIB normally (if I got the question right), i.e.: pcs cluster cib cib.xml | 06:59 |
| jaosorior | thanks | 07:00 |
| jaosorior | that's exactly what I needed :D | 07:00 |
| bandini | :) | 07:00 |
| *** rcernin has quit IRC | 07:03 | |
| *** aufi has joined #tripleo | 07:08 | |
| *** sam_wan has joined #tripleo | 07:08 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 07:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,Triaged] - Assigned to Toure Dunnon (toure) | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-docs master: Add commands in the doc to generate the nic configs https://review.openstack.org/596188 | 07:10 |
| chandankumar | Tengu: Let me put test patches to test it | 07:11 |
| *** ykarel_ has joined #tripleo | 07:12 | |
| *** ykarel_ has quit IRC | 07:13 | |
| *** ykarel_ has joined #tripleo | 07:14 | |
| *** ykarel has quit IRC | 07:14 | |
| *** Chaserjim has joined #tripleo | 07:15 | |
| jaosorior | bandini: do we enable pacemaker resource updates enabled by default? | 07:16 |
| bandini | jaosorior: as in 'if the pacemaker resource definition change we update the definition in the cluster cib'? yes since rocky | 07:16 |
| jaosorior | uhm... not seeing that reflected in master | 07:17 |
| *** apetrich has quit IRC | 07:17 | |
| jaosorior | let me check | 07:17 |
| *** shardy has joined #tripleo | 07:18 | |
| *** jtomasek has joined #tripleo | 07:19 | |
| *** Chaserjim has quit IRC | 07:19 | |
| bandini | ack, lemme know | 07:19 |
| bandini | mschuppert, hjensas, jaosorior: now that https://review.openstack.org/#/c/607492/ landed, can we actually remove the neutron-metadata-agent and the iptables nat rule from the undercloud? | 07:20 |
| jaosorior | bandini: I guess we can (only for master) | 07:21 |
| bandini | right only for master | 07:21 |
| mschuppert | bandini: yes should be good | 07:21 |
| *** ykarel_ is now known as ykarel | 07:22 | |
| jaosorior | bandini: the resource is not updated :( | 07:22 |
| jaosorior | in master | 07:22 |
| hjensas | jaosorior: bandini: I guess we need to wait for promitions etc ? | 07:22 |
| jaosorior | hjensas: you're right. | 07:23 |
| hjensas | jaosorior: bandini: I would like to see those changes backported as well - https://bugzilla.redhat.com/show_bug.cgi?id=1635370 | 07:23 |
| openstack | bugzilla.redhat.com bug 1635370 in rhosp-director "TLS everywhere is not compatible with routed spine/leaf" [High,On_dev] - Assigned to hjensas | 07:23 |
| jaosorior | hjensas: as long as TLS everywhere keeps working, I'm fine with backporting that too | 07:23 |
| hjensas | jaosorior: yes, the routed networks thing can work over the network. But it becomes difficult if the routed network have multiple clouds. | 07:25 |
| *** bogdando has joined #tripleo | 07:26 | |
| *** cylopez has joined #tripleo | 07:26 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates stable/queens: Add posibilities to set tunnel_csum in ovs agent https://review.openstack.org/609935 | 07:28 |
| jaosorior | bandini: got a minute? | 07:28 |
| *** apetrich has joined #tripleo | 07:30 | |
| bandini | jaosorior: sure | 07:33 |
| *** hjensas is now known as hjensas|afk | 07:34 | |
| *** sam_wan has quit IRC | 07:35 | |
| iurygregory | good morning o/ | 07:40 |
| *** sam_wan has joined #tripleo | 07:41 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Revert "Skopeo based uploader" https://review.openstack.org/609941 | 07:41 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-upgrade master: Set container_cli for undercloud https://review.openstack.org/608462 | 07:41 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs050: upgrade the undercloud to Podman containers https://review.openstack.org/608463 | 07:42 |
| *** aufi has quit IRC | 07:43 | |
| *** tosky has joined #tripleo | 07:45 | |
| jaosorior | good morning! | 07:45 |
| *** jpich has joined #tripleo | 07:46 | |
| *** phuongnh has joined #tripleo | 07:50 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/rocky: Honor --skip-deploy-identifier in common deploy tasks https://review.openstack.org/609944 | 07:51 |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates stable/pike: Add posibilities to set tunnel_csum in ovs agent https://review.openstack.org/609946 | 07:55 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: DNM: Testing neutron/OVN rootwrap containers https://review.openstack.org/609947 | 07:58 |
| huynq | jaosorior: Is the Friday community meeting of TripleO Upgrade Squad still available in https://redhat.bluejeans.com/5192173135 ? | 07:59 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 08:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 08:00 |
| chandankumar | bogdando: Hello | 08:01 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: DNM: Testing neutron/OVN rootwrap containers https://review.openstack.org/609947 | 08:01 |
| chandankumar | bogdando: https://review.openstack.org/609918 -> running specific tempest tests against puppet project | 08:01 |
| chandankumar | bogdando: https://review.openstack.org/#/q/topic:tempest_standalone+(status:open+OR+status:merged) testing patches, feedback is needed on that | 08:01 |
| bogdando | chandankumar: hi | 08:02 |
| chandankumar | once the test job finishes I will start adding failed tests as black regex then start fixing one by one | 08:02 |
| bogdando | chandankumar: nice idea with that whitelist case | 08:03 |
| *** Petersingh is now known as Petersingh|lunch | 08:04 | |
| jaosorior | huynq: great question. I'm not sure tbh | 08:05 |
| *** skramaja_ has joined #tripleo | 08:06 | |
| *** d0ugal has quit IRC | 08:07 | |
| *** skramaja has quit IRC | 08:08 | |
| *** d0ugal has joined #tripleo | 08:08 | |
| *** numans has quit IRC | 08:08 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Toure Dunnon (toure) | 08:10 |
| *** skramaja has joined #tripleo | 08:10 | |
| *** skramaja_ has quit IRC | 08:11 | |
| huynq | jaosorior: thank you! | 08:11 |
| *** rdopiera has joined #tripleo | 08:11 | |
| *** akrivoka has joined #tripleo | 08:12 | |
| *** chem has joined #tripleo | 08:13 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Rename no-tls environment https://review.openstack.org/607841 | 08:16 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Remove deprecated TLS-related environment files https://review.openstack.org/607555 | 08:16 |
| jaosorior | skramaja: thanks for the reviews | 08:16 |
| d0ugal | marios: I replied to your comment here btw: https://review.openstack.org/#/c/609746/ | 08:18 |
| huynq | matbu: Good morning! Can you tell me the status of Friday community meeting (TripleO Upgrade Squad)? | 08:19 |
| marios | thanks d0ugal makes sense/what i expected (if it is successful, it breaks the loop anyway, you don't need to explicitly tell it when only if you want something different) | 08:19 |
| marios | d0ugal: thanks for taking the time | 08:19 |
| huynq | matbu: Is the meeting available today? | 08:19 |
| d0ugal | marios: np | 08:20 |
| d0ugal | thanks for the review! | 08:20 |
| matbu | huynq: hi | 08:22 |
| matbu | huynq: well if you have stuff to bring and share with the squad, yep, just add it to the agenda | 08:22 |
| matbu | huynq: https://etherpad.openstack.org/p/tripleo-upgrade-squad-meeting | 08:23 |
| *** aufi has joined #tripleo | 08:23 | |
| matbu | huynq: but the only problem for today is that probably most of the team is not available today (travelling / day off) | 08:23 |
| matbu | huynq: so in that case it might be better to jump to our scrum meeting on monday | 08:24 |
| huynq | matbu: Im' new in TripleO and take care about Undercloud OS upgrade | 08:25 |
| *** akrivoka has quit IRC | 08:25 | |
| *** akrivoka has joined #tripleo | 08:26 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Lint all the generated playbook after standalone deployment. https://review.openstack.org/604757 | 08:26 |
| huynq | matbu: I just want to join the meeting and track the status of that | 08:27 |
| matbu | huynq: ack then probably better to join on monday scrum | 08:27 |
| huynq | matbu: ok. what time is it? | 08:28 |
| matbu | huynq: 1:30 pm UTC | 08:28 |
| huynq | matbu: thanks! | 08:29 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Revert "Switch fs027 to deploy with podman" https://review.openstack.org/609963 | 08:31 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Revert "Switch fs027 to deploy with podman" https://review.openstack.org/609963 | 08:32 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Revert "Skopeo based uploader" https://review.openstack.org/609941 | 08:36 |
| *** derekh has joined #tripleo | 08:41 | |
| *** slaweq has quit IRC | 08:46 | |
| chandankumar | bogdando: https://review.openstack.org/#/c/609933/ the job is not picked up | 08:48 |
| chandankumar | i donot know why | 08:48 |
| Tengu | chem: hey :). so I found out why I was passing "role" instead of "name" in my ansible import_role: it was due to the fact I worked on kolla-ansible first, and they already were using that import_role, and they are using "role", not name. I didn't check the doc as there were working code already. | 08:50 |
| Tengu | chem: so I think the "role" param is probably a deprecated one still active. | 08:51 |
| chandankumar | bogdando: I am thinking of not using skiprc file at all in fs052 let's skip the tests from fs itself | 08:51 |
| chem | Tengu: hey, yeah, I thought something like this was happening (copy/paste and still working parameter) | 08:52 |
| chandankumar | bogdando: for horizon it worked http://logs.openstack.org/37/609937/1/check/tripleo-puppet-ci-centos-7-standalone/cb358b2/logs/tempest.html.gz | 08:52 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-docs master: Add diagram to minor update developer docs https://review.openstack.org/609693 | 08:52 |
| chem | Tengu: it just make ansible-lint barf badly | 08:52 |
| * chandankumar wait for all the jobs to finish then I will invoke it | 08:52 | |
| Tengu | chem: yeah ;). good catch anyway. so you have my +1 on that (can't +2 ;_;) | 08:52 |
| chem | Tengu: this little ansible-lint after standalone deploy actually catch error and take no time, hopefully we can merge it soon :) | 08:53 |
| Tengu | chem: would be good indeed. clean and valid code is good. | 08:53 |
| chem | Tengu: this is the second error I catch using it in less than 1 month | 08:54 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Enable support for running refstack tests in TQE https://review.openstack.org/570719 | 08:55 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Enable support for running refstack tests in TQE https://review.openstack.org/570719 | 08:55 |
| *** psachin has quit IRC | 08:55 | |
| Tengu | chem: :) | 08:56 |
| *** salmankhan has joined #tripleo | 08:57 | |
| *** janki has quit IRC | 08:58 | |
| *** janki has joined #tripleo | 08:58 | |
| *** salmankhan1 has joined #tripleo | 09:01 | |
| *** salmankhan has quit IRC | 09:02 | |
| *** salmankhan1 is now known as salmankhan | 09:02 | |
| *** Petersingh|lunch is now known as Petersingh | 09:02 | |
| *** paramite has joined #tripleo | 09:02 | |
| *** janki has quit IRC | 09:03 | |
| *** janki has joined #tripleo | 09:04 | |
| *** janki has quit IRC | 09:05 | |
| *** janki has joined #tripleo | 09:05 | |
| openstackgerrit | Merged openstack/tripleo-common master: Retry uploading messages to Swift up to 5 times https://review.openstack.org/609746 | 09:07 |
| kopecmartin|ruck | jaosorior, Hi, are you in power of adding me to tripleo group on launchpad so that i can triage bugs/edit statuses and so on? | 09:08 |
| *** ramishra has quit IRC | 09:09 | |
| *** ramishra has joined #tripleo | 09:09 | |
| jaosorior | kopecmartin|ruck: sure, what's your launchpad id? | 09:10 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 09:10 |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 09:10 |
| kopecmartin|ruck | jaosorior, my id is "mkopec" | 09:10 |
| *** sam_wan has quit IRC | 09:11 | |
| jaosorior | bandini, Tengu: Debug: Exists: bundle haproxy-bundle exists 0 location exists 0 deep_compare: true | 09:12 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: fedora28 standalone job definition https://review.openstack.org/595374 | 09:12 |
| jaosorior | bandini, Tengu: pcmk_resource_has_changed (ng version) returned false for resource haproxy-bundle | 09:13 |
| *** yprokule has joined #tripleo | 09:14 | |
| Tengu | hmm. | 09:16 |
| Tengu | so the comparaison is faulty. | 09:17 |
| jaosorior | Tengu: seems it is | 09:17 |
| Tengu | jaosorior: care to update the bz with that? | 09:18 |
| bandini | jaosorior: throw a tmate my way and I'll try and look at it | 09:19 |
| jaosorior | bandini: same session as before | 09:19 |
| chem | sshnaidm: hey, when you have a minute could you have a look at https://review.openstack.org/#/c/605369/, it required to have standalone upgrade featureset merged. | 09:24 |
| chem | sshnaidm: it's a short one :) | 09:24 |
| skramaja | jaosorior: sure.. | 09:27 |
| chem | jistr: when you have a minute could you push that easy one https://review.openstack.org/#/c/608921/ into the merged maelstrom ? | 09:31 |
| jistr | chem: done, thanks for that fix | 09:32 |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| Tengu | bandini: we can feel your deep love for xml, really :D | 09:38 |
| bandini | ahaha ops :) | 09:39 |
| marios | jaosorior: filed a bug for that scen3 fail on https://review.openstack.org/#/c/608589/ fyi because i saw it yesterday too https://bugs.launchpad.net/tripleo/+bug/1797537 cc kopecmartin|ruck | 09:39 |
| openstack | Launchpad bug 1797537 in tripleo "[intermittent] scenario-multinode jobs failing overcloud deploy TASK "...Start containers for step 2" Error: unable to find resource 'haproxy-bundle'" [High,Triaged] - Assigned to Marios Andreou (marios-b) | 09:39 |
| marios | jaosorior: of course that patch would hit the intermittent issue we've only seen twice so far ! | 09:39 |
| *** psachin has joined #tripleo | 09:46 | |
| jaosorior | marios, kopecmartin|ruck: so... I don't have any options to add new members to launchpad, wtf | 09:47 |
| jaosorior | shardy: can you add kopecmartin|ruck to the tripleo drivers group in launchpad? seems you're the owner. Also, could I be added to the admins? | 09:47 |
| *** numans has joined #tripleo | 09:48 | |
| shardy | jaosorior: sure sec | 09:49 |
| shardy | jaosorior, kopecmartin|ruck: done | 09:52 |
| *** slaweq has joined #tripleo | 09:57 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-common master: Fix nova-api healthcheck in case of nova metadata wsgi https://review.openstack.org/609927 | 09:58 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Increase the deploy_plan timeout in tripleoclient https://review.openstack.org/609993 | 10:01 |
| d0ugal | bogdando: ^ | 10:02 |
| *** ykarel is now known as ykarel|lunch | 10:04 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Increase the deploy_plan timeout in tripleoclient https://review.openstack.org/609993 | 10:04 |
| *** leanderthal has joined #tripleo | 10:05 | |
| chem | matbu: so the jobs takes 2h28, rigth, it's tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades in that review https://review.openstack.org/#/c/607848/9, right ? | 10:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common master: Add AllNodesConfig to config-download group vars https://review.openstack.org/605046 | 10:09 |
| chem | right | 10:09 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 10:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 10:10 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert *tasks from bootstrap_nodeid to short_bootstrap_node_name https://review.openstack.org/605430 | 10:12 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Remove unused tls-cert-inject.yaml template https://review.openstack.org/605491 | 10:12 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Add SERVICE_bootstrap_node_ip values to allNodesConfig https://review.openstack.org/605492 | 10:12 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Remove common bootstrap_nodeid from deploy_steps/tripleo-packages.yaml https://review.openstack.org/605682 | 10:12 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Don't disable keepalived in nonha-arch.yaml https://review.openstack.org/607180 | 10:14 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Enable support for running refstack tests in TQE https://review.openstack.org/570719 | 10:14 |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo master: Replace bootstrap_nodeid with SERVICE_short_bootstrap_node_name https://review.openstack.org/605728 | 10:14 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Improve deep_compare code detection https://review.openstack.org/609998 | 10:19 |
| *** psachin has quit IRC | 10:33 | |
| *** udesale has quit IRC | 10:36 | |
| *** cylopez has quit IRC | 10:38 | |
| kopecmartin|ruck | jaosorior, shardy thank you | 10:39 |
| *** jpena|off has quit IRC | 10:41 | |
| *** lhinds has quit IRC | 10:41 | |
| *** rascasoft has quit IRC | 10:42 | |
| *** jpena|off has joined #tripleo | 10:42 | |
| *** lhinds has joined #tripleo | 10:43 | |
| *** agopi has quit IRC | 10:46 | |
| *** psachin has joined #tripleo | 10:50 | |
| *** lhinds has quit IRC | 10:55 | |
| *** lhinds has joined #tripleo | 10:57 | |
| *** sshnaidm is now known as sshnaidm|off | 10:57 | |
| matbu | chem: right man | 11:00 |
| kopecmartin|ruck | marios|rover, when you have a moment, can you check these two bugs, if they are not duplicate? https://bugs.launchpad.net/tripleo/+bug/1797527 https://bugs.launchpad.net/tripleo/+bug/1797526 | 11:00 |
| openstack | Launchpad bug 1797527 in tripleo "Introspection timed out for FS35" [Undecided,New] | 11:00 |
| openstack | Launchpad bug 1797526 in tripleo "Failed to get power state for node FS01/02" [Undecided,New] | 11:00 |
| chem | matbu: sending a mail about standalone and n->n upgrade rigth now on openstack-dev fyi | 11:01 |
| chem | matbu: if that's ok with you ? | 11:02 |
| *** Petersingh is now known as Petersingh|afk | 11:03 | |
| matbu | chem: yep cool | 11:03 |
| marios | kopecmartin|ruck: yah i'd say so added comment on the first one | 11:05 |
| kopecmartin|ruck | marios, ok, thanks | 11:06 |
| *** cylopez has joined #tripleo | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 11:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 11:10 |
| *** apetrich has quit IRC | 11:10 | |
| chem | matbu: crap forgot the openstack-dev tag! | 11:11 |
| matbu | :) | 11:11 |
| *** ykarel_ has joined #tripleo | 11:11 | |
| chem | matbu: ah no I didn't, I just forgot my brain | 11:14 |
| *** ykarel|lunch has quit IRC | 11:14 | |
| *** dprince has joined #tripleo | 11:18 | |
| *** psachin has quit IRC | 11:28 | |
| *** apetrich has joined #tripleo | 11:29 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart master: Add ctlplane_masquerade for fs21 failing in ovb for ntp issue https://review.openstack.org/610009 | 11:32 |
| *** Petersingh|afk is now known as Petersingh | 11:34 | |
| openstackgerrit | Christian Schwede proposed openstack/python-tripleoclient master: Use backward-compatible Swift data directory https://review.openstack.org/610010 | 11:34 |
| jaosorior | marios: how does masquerade affect there? | 11:34 |
| *** psachin has joined #tripleo | 11:34 | |
| marios | jaosorior: thought it was necessary for the compute to talk out (compute is unable to reach ntp server) | 11:35 |
| marios | jaosorior: i may be wrong? | 11:35 |
| jaosorior | I don't know, that's why I asked :D | 11:36 |
| marios | jaosorior: heh :) well its new to me too. i came across it because i compared to fs 20 which is green for ovb and it has it set | 11:36 |
| marios | jaosorior: you can dig more in the t/q/e roles/undercloud-deploy for 'masquerade' | 11:36 |
| marios | jaosorior: weshay asked me to look into fs21 | 11:37 |
| marios | https://github.com/openstack/tripleo-quickstart-extras/blob/156d14e573c60d897083904e5fefcd460ee418e7/roles/undercloud-deploy/templates/undercloud.conf.j2#L594 jaosorior | 11:37 |
| dciabrin | jaosorior, hey hi o/ does this tls error rings a bell? http://paste.openstack.org/show/731961/ | 11:38 |
| marios | jaosorior: i suspect it is because they also have (those featuresets i mean) network isolation set false, so there is only ctlplane there | 11:38 |
| *** rh-jelabarre has joined #tripleo | 11:46 | |
| jaosorior | dciabrin: it does https://review.openstack.org/#/c/608589/ | 11:48 |
| weshay | marios|rover, kopecmartin|ruck morning.. | 11:48 |
| * weshay gets coffee | 11:48 | |
| kopecmartin|ruck | weshay, morning | 11:48 |
| dciabrin | jaosorior, ah I couldn't find the launchpad, thx! | 11:48 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Disable Swift auditors/replicators on undercloud https://review.openstack.org/610012 | 11:49 |
| chandankumar | weshay: regarding standalone tempest, we are first adding tests to skip tests and going to clear one by one | 11:52 |
| *** abishop has joined #tripleo | 11:52 | |
| chandankumar | weshay: https://review.openstack.org/#/c/609918/ -> master change {donot merge}, and here is the testing patches https://review.openstack.org/#/q/topic:tempest_standalone+(status:open+OR+status:merged) | 11:52 |
| weshay | chandankumar, update taiga not me :) | 11:53 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: fedora28 standalone job definitionIncludes:- yum/dnf compatibility- py2/py3 compatibility- calling virtualenv/pip as python modulesDepends-On: https://review.openstack.org/602492Partial-Bug: #1740928Story: https://tree.taiga.io/project/tripleo-ci-board/us https://review.openstack.org/610014 | 11:53 |
| openstack | bug 1740928 in tripleo "Fedora Support for TripleO-Quickstart" [Low,In progress] https://launchpad.net/bugs/1740928 - Assigned to Sorin Sbarnea (ssbarnea) | 11:53 |
| chandankumar | weshay: I have just one issue https://review.openstack.org/609933 standalone does not got kicked i donot know why | 11:54 |
| marios|rover | weshay: o/ | 11:55 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: fedora28 standalone job definition https://review.openstack.org/595374 | 11:57 |
| *** fhubik has joined #tripleo | 11:58 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Use single replica for standalone AIO deployments https://review.openstack.org/610017 | 11:59 |
| *** fhubik has quit IRC | 12:01 | |
| *** iurygregory has quit IRC | 12:05 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Enable use of python3 and dnf https://review.openstack.org/610018 | 12:06 |
| *** huynq has quit IRC | 12:09 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 12:10 |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 12:10 |
| *** phuongnh has quit IRC | 12:10 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Add fedora28 support for quickstart https://review.openstack.org/591652 | 12:15 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: standalone support for quickstart on libvirt https://review.openstack.org/591540 | 12:17 |
| *** ratailor has quit IRC | 12:17 | |
| *** iurygregory has joined #tripleo | 12:19 | |
| *** boazel has quit IRC | 12:20 | |
| *** ansmith has joined #tripleo | 12:24 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Improve deep_compare code detection https://review.openstack.org/609998 | 12:32 |
| *** ramishra has quit IRC | 12:34 | |
| *** cylopez has quit IRC | 12:37 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Run master to master upgrade only https://review.openstack.org/607830 | 12:38 |
| *** rlandy has joined #tripleo | 12:40 | |
| *** psachin has quit IRC | 12:45 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: standalone support for quickstart on libvirt https://review.openstack.org/591540 | 12:48 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart master: Switch upgrade job to master to master https://review.openstack.org/606948 | 12:49 |
| *** jtomasek has quit IRC | 12:49 | |
| *** lblanchard has joined #tripleo | 12:51 | |
| *** boazel has joined #tripleo | 13:00 | |
| *** mmethot_ has quit IRC | 13:00 | |
| *** agopi has joined #tripleo | 13:02 | |
| *** mmethot has joined #tripleo | 13:03 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-common master: Add crane service to overcloud_containers.yaml.j2 https://review.openstack.org/610028 | 13:04 |
| *** dtantsur is now known as dtantsur|afk | 13:07 | |
| *** agopi has quit IRC | 13:07 | |
| dpeacock | Anyone got time to review and +2 https://review.openstack.org/#/c/609694/ please? | 13:08 |
| *** tzumainn has joined #tripleo | 13:10 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 13:10 |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 13:10 |
| *** vinaykns has joined #tripleo | 13:12 | |
| *** mcornea has joined #tripleo | 13:13 | |
| *** ssbarnea_ has quit IRC | 13:18 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Crane: Add docker/services/crane.yaml https://review.openstack.org/609508 | 13:29 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: DNM, https://review.openstack.org/#/c/609586/ https://review.openstack.org/610035 | 13:31 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: DNM, testing https://review.openstack.org/#/c/609586/ https://review.openstack.org/610035 | 13:32 |
| *** agopi has joined #tripleo | 13:32 | |
| *** sai_p has joined #tripleo | 13:39 | |
| *** ade_lee has joined #tripleo | 13:40 | |
| sai_p | Hi all, in tripleo, starting from which version openvswitch will be containerized? | 13:41 |
| *** dansmith is now known as SteelyDan | 13:41 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Corrected release config location for qa collectlogs https://review.openstack.org/610038 | 13:41 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Remove old config/release files https://review.openstack.org/606944 | 13:42 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Remove old config/release files https://review.openstack.org/606944 | 13:42 |
| mwhahaha | sai_p: TBD i don't think we have that yet | 13:42 |
| sai_p | ok, thank you | 13:43 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Improve getthelogs fetching periodic RDO CI jobs https://review.openstack.org/610039 | 13:43 |
| weshay | ssbarnea, so for the recreates.. don't use the 8gb infra image | 13:44 |
| ssbarnea | weshay: sure, I already switched to the rdo 0.2gb version | 13:44 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Corrected release config location for qa collectlogs https://review.openstack.org/610038 | 13:45 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Corrected release config location for collection https://review.openstack.org/610038 | 13:46 |
| weshay | ssbarnea, rock on | 13:47 |
| ssbarnea | weshay: ^^ should be reviewable as is very short and explicit. | 13:47 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Improve getthelogs fetching periodic RDO CI jobs https://review.openstack.org/610039 | 13:48 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart master: Add ctlplane_masquerade for fs21 failing in ovb for ntp issue https://review.openstack.org/610009 | 13:49 |
| weshay | toure, ping.. sorry about yesterday.. long birthday lunch.. did you find anything interesting? | 13:50 |
| toure | weshay happy belated birthday? | 13:52 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: DNM, testing https://review.openstack.org/#/c/609941/ https://review.openstack.org/610041 | 13:52 |
| toure | weshay I did get a bit further in my deployment | 13:53 |
| weshay | toure, wasn't mine :) | 13:53 |
| weshay | toure, k.. d0ugal's patch landed.. I have not seen that error again today.. but will be looking for it | 13:54 |
| * weshay will scan the gate failures | 13:54 | |
| weshay | now | 13:54 |
| toure | ack | 13:54 |
| *** mburned_out is now known as mburned | 13:54 | |
| toure | the only failure I ran into was upgrades but that maybe an issue with my box | 13:55 |
| *** Vorrtex has joined #tripleo | 13:55 | |
| d0ugal | weshay, toure: This small patch might avoid some needless timeouts if we creep over 6 mins again https://review.openstack.org/#/c/609993/ | 13:56 |
| *** lblanchard has quit IRC | 13:56 | |
| toure | s/upgrades/updates | 13:57 |
| weshay | d0ugal, thank you.. my only concern is that if we keep allowing things to run longer and longer we'll get timeouts upstream | 13:57 |
| weshay | mwhahaha, you may want to review ^ | 13:58 |
| d0ugal | weshay: Yeah, good point. | 13:58 |
| d0ugal | weshay: I'd be happy to abandon and focus on reducing the time if we hit it again | 13:58 |
| weshay | same w/ retries.. but I'll defer to jaosorior, mwhahaha, | 13:58 |
| weshay | d0ugal, /me goes through the gate failures | 13:59 |
| nhicher | weshay, toure: I've got time issue on vexxhost (with localstorage), but not sure it the same issue https://logs.rdoproject.org/66/16566/3/check/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/8f431fe/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-10-11_00_23_07 | 13:59 |
| weshay | nhicher, ya.. that is what we hit. | 13:59 |
| Hobbestigrou | hi | 13:59 |
| weshay | nhicher, d0ugal in ovb we have more time available to us | 13:59 |
| marios | jaosorior: related to that masquerade issue earlier fyi see https://bugs.launchpad.net/tripleo/+bug/1794038 (and i'm re-using an existing bug at https://bugs.launchpad.net/tripleo/+bug/1794258 for that https://review.openstack.org/#/c/610009/ from earlier | 14:00 |
| openstack | Launchpad bug 1794038 in tripleo "undercloud masquerade_networks is now silently ignored" [High,In progress] - Assigned to Harald Jensås (harald-jensas) | 14:00 |
| openstack | Launchpad bug 1794258 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master job fails overcloud deploy as nodes can't reach pool.ntp.org" [Critical,In progress] - Assigned to Marios Andreou (marios-b) | 14:00 |
| weshay | so.. it's possible to change the setting between env.. if we wanted to | 14:00 |
| Hobbestigrou | weshay, do you know list table in rest http://docutils.sourceforge.net/docs/ref/rst/directives.html#id49 ? | 14:00 |
| weshay | nhicher, did that run have d0ugal's latest retry patch? | 14:00 |
| weshay | nhicher, https://review.openstack.org/#/c/609746/ | 14:00 |
| weshay | Hobbestigrou, sorry I do not.. what is the context of the question? | 14:01 |
| Hobbestigrou | i'm going to fix this patch https://review.openstack.org/#/c/482480/ | 14:01 |
| Hobbestigrou | weshay, yes sorry | 14:01 |
| weshay | Hobbestigrou, ah.. rock on.. that would be very helpful.. thank you! | 14:02 |
| toure | d0ugal could we perform a resource test against swift | 14:02 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Disable Swift auditors/replicators on undercloud https://review.openstack.org/610012 | 14:02 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Use single replica for standalone AIO deployments https://review.openstack.org/610017 | 14:02 |
| Hobbestigrou | about this patch, i think maybe it's better to migrate on list table | 14:02 |
| nhicher | weshay: no, I will add a depends-on, thanks | 14:02 |
| Hobbestigrou | what do you think about that ? | 14:02 |
| d0ugal | toure: What would that involve? :) | 14:03 |
| Hobbestigrou | that will be more easy to maintain | 14:03 |
| toure | I was think a small create and deletion? or would that be to invasive | 14:03 |
| *** bdodd has quit IRC | 14:03 | |
| weshay | d0ugal, I think you probably fixed the issue.. all the latest gate failures are timeouts http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen | 14:03 |
| *** skramaja has quit IRC | 14:04 | |
| *** bdodd has joined #tripleo | 14:06 | |
| *** mburned is now known as mburned_out | 14:07 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 14:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 14:10 |
| d0ugal | weshay: cool, do let me know if anything new pops! | 14:10 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Use single replica for standalone AIO deployments https://review.openstack.org/610017 | 14:11 |
| weshay | kopecmartin|ruck, I'm going to try and get some ironic eyes on https://bugs.launchpad.net/tripleo/+bug/1797526 | 14:13 |
| openstack | Launchpad bug 1797526 in tripleo "Failed to get power state for node FS01/02" [Critical,Triaged] | 14:13 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Use single replica for standalone AIO deployments https://review.openstack.org/610017 | 14:13 |
| weshay | kopecmartin|ruck, I updated the bug a bit | 14:13 |
| kopecmartin|ruck | weshay, ok, great, thanks | 14:13 |
| openstackgerrit | Chuck Short proposed openstack/os-apply-config master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610048 | 14:15 |
| Hobbestigrou | arxcruz, same question, do you know list table ? | 14:15 |
| Hobbestigrou | other question, do you think it can be interesting to migrate the table on list table here https://review.openstack.org/#/c/482480/ ? | 14:16 |
| weshay | ugh.. no body from ironic is around :( | 14:16 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: DNM Test Upgrade Ci jobs https://review.openstack.org/610049 | 14:17 |
| *** yprokule has quit IRC | 14:17 | |
| weshay | derekh, oh hey.. do you have a few minutes to help us figure out if some introspection errors are caused by ironic or rdo-cloud? | 14:17 |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1797526 | 14:18 |
| openstack | Launchpad bug 1797526 in tripleo "Failed to get power state for node FS01/02" [Critical,Triaged] | 14:18 |
| derekh | weshay: looking | 14:19 |
| toure | d0ugal better yet if we issue a check for the container tripleo.swift.v1.container_exists | 14:19 |
| bogdando | thanks dpeacock :) | 14:19 |
| bogdando | I hope anyone around do use that tool :) | 14:20 |
| *** ykarel__ has joined #tripleo | 14:20 | |
| bogdando | abishop: thanks for review! replied/fixed | 14:20 |
| *** ykarel_ has quit IRC | 14:22 | |
| abishop | bogdando: ack, about to respond to your response, looking good! still looking for cschwede to comment as swift sme | 14:23 |
| weshay | derekh, added the virtbmc log to the bug | 14:23 |
| chandankumar | mwhahaha: Hello | 14:25 |
| mwhahaha | chandankumar: hi2u | 14:25 |
| chandankumar | mwhahaha: I am good, | 14:25 |
| chandankumar | mwhahaha: regarding this review https://review.openstack.org/609918 , my main intent with this review, is to run specific group of tests which are related to a particular project in order to save time | 14:26 |
| mwhahaha | chandankumar: ok can we override that via zuul configs instead? | 14:26 |
| mwhahaha | chandankumar: i don't think we should have that logic in quickstart | 14:27 |
| chandankumar | mwhahaha: currently fs can be overriden not zuul jobs i think in tripleo | 14:27 |
| chandankumar | mwhahaha: https://review.openstack.org/609666 | 14:27 |
| mwhahaha | chandankumar: but it's not an fs config, it'd be an ansible var or soemthing | 14:27 |
| chandankumar | mwhahaha: so we want to keep in the zob itself? | 14:28 |
| d0ugal | toure: Why do we need to check for the container? | 14:28 |
| d0ugal | toure: I think I am missing the context | 14:28 |
| mwhahaha | chandankumar: i would think it would make sense to add it to the job configuration in the respective puppet-* repo instead of having this logic in quickstart | 14:28 |
| chandankumar | mwhahaha: ok, that would be doable, let me see how it goes | 14:28 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Use single replica for standalone AIO deployments https://review.openstack.org/610017 | 14:29 |
| mwhahaha | chandankumar: cool thanks, i think that'll reduce the complexity and be an improvement :) | 14:29 |
| d0ugal | toure: btw https://github.com/openstack/tripleo-common/blob/master/workbooks/messaging.yaml#L105-L112 | 14:29 |
| toure | d0ugal I am looking throught the deployment workflow, I was thinking once we create teh container we let it settle and then check it with that call | 14:29 |
| toure | ah | 14:29 |
| weshay | derekh, FYI.. we're also seeing odd errors like https://logs.rdoproject.org/46/609846/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/94f8614/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz?level=ERROR | 14:30 |
| weshay | hard to tell whats going on | 14:31 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Do not wipe disks on OpenShift gluster nodes https://review.openstack.org/605127 | 14:31 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Ensure the needed openshift resources are set https://review.openstack.org/610054 | 14:31 |
| openstackgerrit | Chuck Short proposed openstack/os-collect-config master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610055 | 14:32 |
| derekh | weshay: is it possible the undercloud is running out of ram? I've seen that ipmitool command fail when things are tight | 14:32 |
| * weshay looks | 14:32 | |
| weshay | we should have dstat | 14:32 |
| toure | d0ugal this still feels like the varification is still a bit to far removed from the creatation, but I could be modeling this incorrectly in my head | 14:32 |
| derekh | weshay: its weird because in the logs it worked and then failed 4 minutes later | 14:32 |
| toure | brbr | 14:33 |
| toure | brb | 14:33 |
| weshay | derekh, https://logs.rdoproject.org/46/609846/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/94f8614/logs/undercloud/var/log/extra/dstat.html.gz | 14:33 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Ensure the needed openshift resources are set https://review.openstack.org/610054 | 14:34 |
| weshay | and queens: https://logs.rdoproject.org/89/608589/6/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/b472dad/logs/undercloud/var/log/extra/dstat.html.gz | 14:34 |
| weshay | queens looks fairly stable w/ free memory, where master is not | 14:35 |
| derekh | weshay: could be a red herring but give me a minute to see how much it needs | 14:37 |
| weshay | k | 14:37 |
| *** faceman has quit IRC | 14:37 | |
| openstackgerrit | Chuck Short proposed openstack/os-net-config master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610059 | 14:37 |
| *** lhinds has quit IRC | 14:38 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Add OS::TripleO::Services::Rhsm to OpenShift roles https://review.openstack.org/605999 | 14:38 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Use Timesync service instead of Ntp https://review.openstack.org/606000 | 14:38 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Let openshift-ansible configure the firewall https://review.openstack.org/606001 | 14:38 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Do not modify imagestreams https://review.openstack.org/609445 | 14:38 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Set openshift_docker_insecure_registries https://review.openstack.org/609603 | 14:38 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Do not wipe disks on OpenShift gluster nodes https://review.openstack.org/605127 | 14:38 |
| *** haleyb has quit IRC | 14:38 | |
| *** rnoriega has quit IRC | 14:39 | |
| *** pliu has quit IRC | 14:39 | |
| weshay | derekh, let me know if you think we should track the error in master in a different bug than queens | 14:39 |
| openstackgerrit | Chuck Short proposed openstack/os-refresh-config master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610060 | 14:39 |
| derekh | weshay: ok, did the error with ipmitool in master and queens both start at the same time? | 14:40 |
| *** ansmith has quit IRC | 14:40 | |
| weshay | derekh, 8 hours between the two runs | 14:41 |
| derekh | k | 14:41 |
| weshay | derekh, but we've been having issues in rdo-cloud, you probably know that ;) | 14:42 |
| derekh | weshay: yup, both branches having trouble connecting to ipmi at rough;y the same time do suggest a cloud problem, but will keep looking | 14:43 |
| *** bnemec is now known as beekneemech | 14:44 | |
| weshay | derekh, and that would probably be issues w/ the networking? | 14:44 |
| weshay | derekh, any idea how I may be able to capture data in the jobs to indicate that? | 14:44 |
| weshay | tcpdump? netstat | 14:44 |
| derekh | weshay: yup, have we a env thats reproducing the problem we can jump onto? | 14:45 |
| weshay | derekh, I have an env up, but introspection worked in my recreate on my tenant | 14:45 |
| derekh | weshay: tcpdump for impi should give some clues (on both the undercloud and bmc) | 14:45 |
| weshay | I can keep trying | 14:45 |
| weshay | k.. will start looking at that.. if you have a sample command I can borrow that would be helpful | 14:46 |
| weshay | kopecmartin|ruck, marios|rover ^ | 14:46 |
| weshay | oh look at that https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-images/templates/overcloud-prep-images.sh.j2#L221 | 14:47 |
| weshay | ha, I had no idea that was there | 14:47 |
| *** lblanchard has joined #tripleo | 14:47 | |
| derekh | weshay: I'll try and recreate one also, it mightn't be ready before I finished but if it reproduces i'll poke at it before monday | 14:47 |
| weshay | sudo tcpdump -i any port 67 or port 68 or port 69 -w {{ step_introspect_debug_tcpdump_log }} & | 14:47 |
| derekh | weshay: need to add ipmi to that, /me goes to find the filter | 14:48 |
| marios|rover | weshay: ack | 14:49 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Adopt use of ansible_pkg_mgr instead of just yum https://review.openstack.org/610067 | 14:49 |
| derekh | tcpdump -i any port 67 or port 68 or port 69 or host <bmcip> ? | 14:50 |
| bogdando | derekh, weshay: > is it possible the undercloud is running out of ram? I've seen that ipmitool command fail when things are tight | 14:50 |
| bogdando | we should containerize VBMC and set cgroups limits! containers solve everything | 14:50 |
| weshay | bogdando, k.. Tengu was also looking into this.. in ovb land we are running the vbmc on it's own node | 14:51 |
| bogdando | jokes off, it really makes sens consider limiting everything in cgroups for CI at least... | 14:51 |
| weshay | derekh, bogdando would a bigger flavor help? | 14:51 |
| derekh | weshay: it would at least confirm or put an end to the theory | 14:52 |
| weshay | rlandy, fyi ^ | 14:52 |
| * bogdando reminded of https://bravenewgeek.com/take-it-to-the-limit-considerations-for-building-reliable-systems/ | 14:52 | |
| weshay | ya.. we don't run dstat on the bmc either | 14:52 |
| derekh | weshay: the memory limit I was refering to was the undercloud | 14:53 |
| weshay | bogdando, building reliable systems when they are flying in the air at 30k feet and going 500mph is hard :) | 14:53 |
| weshay | derekh, ah k | 14:53 |
| derekh | weshay: ipmu port 623 | 14:53 |
| weshay | k.. /me puts up some patches.. sec | 14:54 |
| *** ansmith has joined #tripleo | 14:57 | |
| bogdando | weshay: wrt dstat logs, I though vbmc is another VM, not undercloud? | 14:57 |
| bogdando | so we'll need memory stats for that node, I think | 14:57 |
| bogdando | at least ovb reproduce script creates vbmc as another vm | 14:58 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: turn on tcpdump debug by default for ovb jobs https://review.openstack.org/610072 | 14:58 |
| weshay | bogdando, right.. bmc is it's own node.. I can increase that flavor too | 14:58 |
| bogdando | ah, never-mind, that was for ipmitool not for vbmc | 14:58 |
| derekh | weshay: I'm thinking its rdo-cloud, see logs for the env on RDO cloud I have running for about a week https://goodsquishy.com/upload/5bc0b6bd3e914 | 15:01 |
| derekh | weshay: ipmi errors started yesterday 2018-10-11 10:13:26.487 for 8 hours | 15:02 |
| derekh | weshay: then went away again | 15:02 |
| bogdando | weshay: > 30k feet and 500 mph | 15:02 |
| bogdando | I always suspected that rdo cloud is hosted on Flying Command Center | 15:02 |
| bogdando | to follow the sun for engineers and keeping it close 24/7 | 15:03 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: increase the available ram for the bmc node https://review.openstack.org/610078 | 15:04 |
| weshay | bogdando, lolz | 15:04 |
| rlandy | weshay: ^^ is the for the reproducer only or actual CI? | 15:04 |
| *** ansmith has quit IRC | 15:05 | |
| weshay | rlandy, heh.. that's why I pinged.. we need it for both I suppose | 15:05 |
| weshay | rlandy, please help me if I'm missing something | 15:05 |
| rlandy | weshay: for actual CI, you need to change te-broker | 15:06 |
| rlandy | and we can test that change on the instance itself | 15:06 |
| weshay | derekh, that is very cool man, interesting | 15:06 |
| * rlandy gets | 15:06 | |
| *** aufi has quit IRC | 15:06 | |
| rlandy | yesterday we had a few rdocloud issues that may be responsible | 15:06 |
| weshay | derekh, are you just running impmi commands directly to a bmc in your tenant or something and polling on a constant? | 15:06 |
| weshay | I'd love to get this data in front of kforde | 15:07 |
| derekh | weshay: I just grepped my ironic-conductor logs for " ERROR " | 15:07 |
| *** iurygregory has quit IRC | 15:07 | |
| weshay | derekh, you have a undercloud in place.. you just leave up I think.. right? | 15:08 |
| derekh | weshay: every so often I tear it down but ya I got into the habbit of reusing the undercloud multiple times if I can | 15:09 |
| rlandy | https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/te-broker/create-env#L50 | 15:09 |
| weshay | ya.. that's the pro way | 15:09 |
| bogdando | > ipmi errors started yesterday 2018-10-11 10:13:26.487 for 8 hours | 15:09 |
| bogdando | > derekh: weshay: then went away again | 15:09 |
| bogdando | Let's check if rdo cloud wasn't passing by North Korea air space or something by that time period, that might explain the interference | 15:09 |
| bogdando | it's Friday evening, so please excuse the way my root cause analysis mindset goes :) | 15:09 |
| *** leanderthal has quit IRC | 15:10 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 15:10 |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 15:10 |
| *** paramite has quit IRC | 15:10 | |
| bogdando | btw, it may be worth exploring co-locating vbmc on undercloud for ovb jobs, to make one remote to remote communication fail point less | 15:11 |
| bogdando | rlandy, weshay: ? | 15:11 |
| derekh | weshay: here is the full conductor logs, but your probably just interested in the errors above to see if they correlate to rdo-cloud problems https://goodsquishy.com/upload/5bc0b919e1877 | 15:11 |
| *** ksambor has quit IRC | 15:12 | |
| weshay | bogdando, probably worth discussing at a tripleo mtg | 15:12 |
| bogdando | yeah | 15:13 |
| derekh | hmm /me should have zipped that | 15:13 |
| * derekh checks to see what his hosting limit is | 15:13 | |
| *** hjensas|afk is now known as hjensas | 15:14 | |
| *** rpioso|afk is now known as rpioso | 15:14 | |
| *** abishop has quit IRC | 15:17 | |
| *** jtomasek has joined #tripleo | 15:18 | |
| *** ansmith has joined #tripleo | 15:18 | |
| rlandy | bogdando: ^^ I'm in agreement | 15:20 |
| rlandy | we should be sure the errors are not as a result of rdocloud troubles | 15:21 |
| bogdando | another "hack" I can think of is using "fake" nested qemu overcloud VMs on undercloud to check BM provisioning fully locally, given even local vbmc placement. Then switch those to pre-provisioned servers running nearby and continue as multinode | 15:23 |
| bogdando | that elliminates networking communication for hardware provisioning steps | 15:24 |
| bogdando | was the flying fortress idea better?.. | 15:24 |
| bogdando | as pre-provisioned I mean deployed servers | 15:25 |
| bogdando | derekh: WDYT? would that be a too ugly hack and affect the coverage badly? | 15:25 |
| bogdando | if we do that right, nothing changes but virtual networking moved to local undercloud host fully. We can still "break" things by wrong net configs or iptables rules | 15:26 |
| derekh | bogdando: sounds like too much effort to me, if it is networking problems between nodes then we'll hit it again later in the process | 15:27 |
| bogdando | right, maybe | 15:27 |
| *** Petersingh is now known as Petersingh|gone | 15:28 | |
| bogdando | but I think it may be L2 issues | 15:28 |
| *** Petersingh|gone has quit IRC | 15:28 | |
| bogdando | which we do not care of after the hw provisionin | 15:28 |
| bogdando | AFAIK | 15:28 |
| derekh | bogdando: this is most likely only happening during introspection because its one of the earliest communications between nodes we have when doing OVB | 15:28 |
| bogdando | moving L2 to local libvirt stack could help | 15:29 |
| bogdando | just guessing... | 15:29 |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo master: Implement Global Galera database https://review.openstack.org/609734 | 15:29 |
| bogdando | may be its L2 affected on RDO cloud and not above | 15:30 |
| bogdando | ok, I have no crazy ideas for now, gtg | 15:30 |
| bogdando | see you on | 15:30 |
| bogdando | Monday | 15:30 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: DNM Test Upgrade Ci jobs https://review.openstack.org/610049 | 15:32 |
| openstackgerrit | Merged openstack/ansible-role-tripleo-cookiecutter master: Change YAML file extensions to .yaml from .yml https://review.openstack.org/604463 | 15:32 |
| *** bogdando has quit IRC | 15:35 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: add port 623 to tcpdump for ironic debug https://review.openstack.org/610087 | 15:35 |
| weshay | thanks derekh for the adivce and help! | 15:35 |
| weshay | rlandy, we probably should refactor that tcpdump debug out of the jinja template and put it in ansible.. wdyt? | 15:36 |
| * rlandy looks | 15:36 | |
| rlandy | weshay: ack - probably more generic but not essential | 15:38 |
| derekh | weshay: no prob, if you manage to get it reproduced let me know and I'm happy to take another look, but for the moment it looks like underlying infrastructure problems | 15:38 |
| zzzeek_ | Hi tripleo-ers! As a Python veteran, I know absolutely nothing about Ruby, is there some simple idiot reason this happens when I try to run puppet-tripleo tests ? http://paste.openstack.org/show/731975/ | 15:39 |
| zzzeek_ | attempting to follow instruction at https://docs.openstack.org/puppet-openstack-guide/latest/contributor/testing.html#running-rspec | 15:39 |
| mwhahaha | zzzeek_: that's the way to run them | 15:40 |
| zzzeek_ | mwhahaha: welp, it's not working | 15:41 |
| zzzeek_ | this worked for me like a year ago i remember that | 15:41 |
| mwhahaha | zzzeek_: might try removing the Gemfile.lock and rerunning bundle install. then it's bundle exec rake spec to run the tests | 15:42 |
| zzzeek_ | mwhahaha: OK trying | 15:42 |
| mwhahaha | it appears to be working fine for me right now | 15:42 |
| *** huynq has joined #tripleo | 15:43 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: DNM: testing standalone upgrade package building. https://review.openstack.org/610091 | 15:43 |
| openstackgerrit | Merged openstack/instack master: Use openstack-tox-cover template https://review.openstack.org/600923 | 15:44 |
| chem | weshay: hey, by the way we've got a successful standalone upgrade there https://review.openstack.org/#/c/604706/ | 15:44 |
| weshay | I love chem | 15:44 |
| *** abishop has joined #tripleo | 15:44 | |
| chem | weshay: me too | 15:44 |
| *** rlandy is now known as rlandy|biab | 15:45 | |
| *** abishop has quit IRC | 15:45 | |
| chem | weshay: it's seems it's doing everything as expected like switching from rocky to master using emit release. | 15:45 |
| weshay | \0/ | 15:45 |
| * weshay is looking | 15:46 | |
| weshay | best news all day | 15:46 |
| chem | weshay: I'm testing that package building is happening at the rigth time as well, it should be, but I just want to make sure | 15:46 |
| * weshay writes a song about the standalone deployment | 15:46 | |
| chem | weshay: yeah, this stuff rocks | 15:46 |
| weshay | mwhahaha++ | 15:46 |
| *** huynq has quit IRC | 15:46 | |
| weshay | chem++ | 15:46 |
| *** abishop has joined #tripleo | 15:48 | |
| weshay | cool.. this looks sane http://logs.openstack.org/06/604706/23/check/tripleo-ci-centos-7-standalone-upgrade/2f3dbc8/logs/emit_releases_file.log | 15:48 |
| weshay | hashes are right | 15:48 |
| weshay | containers in deploy look right | 15:49 |
| nhicher | weshay, rlandy|biab: do you think we can merge https://review.openstack.org/#/c/596432/ or do we need 3rd +1 ? | 15:51 |
| *** ansmith has quit IRC | 15:51 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Adopt use of ansible_pkg_mgr instead of just yum https://review.openstack.org/610067 | 15:52 |
| zzzeek_ | mwhahaha: thanks, looks like Gemfile.lock was the magic talisman preventing all that is good and holy from proceeding | 15:52 |
| mwhahaha | zzzeek_: cool | 15:52 |
| *** boazel has quit IRC | 15:53 | |
| derekh | Trying to get a new ironic in overcloud job working if anybody has time to review https://review.openstack.org/#/c/579603/ https://review.openstack.org/#/c/509728/ | 15:53 |
| derekh | example job running here https://review.openstack.org/#/c/582294/ see http://logs.openstack.org/94/582294/13/experimental/tripleo-ci-centos-7-scenario012-multinode-oooq-container/0299941/ | 15:54 |
| weshay | nhicher, ya.. I want to merge this as well. master passed ovb which is awesome.. my only thought is, if we included a depends on that ran the job in vexx as well would we see the vexx results in this patch as well? | 15:54 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Add support for ODL-OVS IPv6 deployment https://review.openstack.org/586251 | 15:56 |
| *** ansmith has joined #tripleo | 15:57 | |
| nhicher | weshay: I can run this job https://review.rdoproject.org/r/#/c/15896/ to validate on rdocloud and vexxhost | 15:57 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Lint all the generated playbook after standalone deployment. https://review.openstack.org/604757 | 15:58 |
| nhicher | right now, https://review.rdoproject.org/r/#/c/16566/ is running (with periodic) | 15:58 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add provision to specify java arguments to ODL https://review.openstack.org/604750 | 15:58 |
| weshay | nhicher, right.. that is reporting to sf, do we have approval to light the candle on reporting to openstack.gerrit? | 15:59 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 16:00 |
| nhicher | weshay: not yet, we have only one nodepool slave capacity for now | 16:00 |
| *** ssbarnea has quit IRC | 16:00 | |
| nhicher | versus 150 on rdocloud | 16:00 |
| weshay | nhicher, ack | 16:01 |
| *** derekh has quit IRC | 16:01 | |
| nhicher | weshay: but I will run periodic to get metric, I can run this periodic on rdocloud and vexxhost providers | 16:01 |
| *** panda has quit IRC | 16:01 | |
| *** panda has joined #tripleo | 16:02 | |
| zzzeek_ | I like how in t-h-t the "tox pep8" is doing an enormous set of syntactical checks that not only have nohting to do with pep8 they have nothing to do with Python :) | 16:02 |
| chem | weshay: the lint jobs is nearly good to go (and catched another bug) check my last comment there https://review.openstack.org/#/c/604756/4 when you have time. | 16:02 |
| weshay | chem, nice 2018-10-11 17:12:42 | 2018-10-11 17:12:42.761 112410 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "2018-10-11 17:09:05,152 DEBUG: 157858 -- config_image docker.io/tripleomaster/centos-binary-cinder-api:965941f1e62cef16967e7a7cd6d98263e52acb62_0989b280",[00m | 16:02 |
| weshay | this is awesome | 16:03 |
| chem | weshay: yeah, a tear dropped from my eye when I saw this | 16:03 |
| weshay | :) | 16:03 |
| weshay | nhicher, btw.. we're about to bump export BMC_FLAVOR="ci.m1.small" to medium.. patch on top | 16:04 |
| *** janki has quit IRC | 16:04 | |
| nhicher | weshay: ok, I need to ask vexxhost got get this flavor | 16:04 |
| weshay | medium? | 16:05 |
| weshay | chem, wait a sec.. the upgrade role/playbook is in tqe | 16:07 |
| weshay | is that where you want it? | 16:07 |
| weshay | fine for now.. but wondering | 16:07 |
| Tengu | uhu, cgroups limits. One of the purposes of containers imho. | 16:07 |
| Tengu | we should set limits for all services, and analyse WHY they explode in case set limits aren't enough :D | 16:08 |
| *** dmacpher_ has joined #tripleo | 16:08 | |
| nhicher | weshay: yes, ci.m1.* flavors are specifics to rdocloud, on vexxhost, I asked mnaser to create ci.m1.small, ci.m1.large and ci.m1.nodepool, with same specs than what we have on rdocloud | 16:09 |
| weshay | nhicher, just saying you would want something one size bigger for the bmc | 16:09 |
| chem | weshay: I don't see the need for tripleo-upgarde integration as we don't need infrared integration and the role should move hand by hand with the standalone role, not the upgrade workflow | 16:09 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 16:10 |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 16:10 |
| weshay | chem, well.. we can figure it out later.. VERY WELL DONE! | 16:10 |
| *** dmacpher has quit IRC | 16:10 | |
| weshay | chem, let's talk about that though.. | 16:10 |
| weshay | chem, go celebrate | 16:10 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Fix documentation string for update roles and nodes option. https://review.openstack.org/608921 | 16:13 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Put create repo script into its own tasks file. https://review.openstack.org/605369 | 16:13 |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo master: Implement Global Galera database https://review.openstack.org/609734 | 16:13 |
| chem | weshay: red wine on its way, when https://review.openstack.org/#/c/610091/ proves that package upgrade happens at the right time then I open the champagne, and when everything is merged I get Lagavulin | 16:13 |
| chem | bye now. | 16:14 |
| weshay | chem, enjoy | 16:15 |
| *** akrivoka has quit IRC | 16:15 | |
| Tengu | chem: good choice with the laga :) | 16:15 |
| *** akrivoka has joined #tripleo | 16:15 | |
| Tengu | although... I know some other brands that are also really interesting :D | 16:16 |
| weshay | everyone hold your breath | 16:16 |
| openstackgerrit | Merged openstack/tripleo-ansible master: fix tox python3 overrides https://review.openstack.org/607754 | 16:17 |
| chem | Tengu: brough back so old connemara from the dublin ptg which which had a very short life span :) | 16:17 |
| Tengu | oh, the bot is back :D | 16:17 |
| Tengu | chem: have you ever heard of "Black Art" bottles? | 16:17 |
| *** jpich has quit IRC | 16:17 | |
| chem | Tengu: nope | 16:18 |
| Tengu | chem: https://www.whisky.fr/bruichladdich-black-art-part-6959.html that's from another world :) | 16:18 |
| openstackgerrit | Merged openstack/tripleo-ipsec master: fix tox python3 overrides https://review.openstack.org/607758 | 16:18 |
| Tengu | oh.. French. sorry. | 16:18 |
| chem | Tengu: I'm confortable with french :) | 16:18 |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Create a python2 venv https://review.openstack.org/610101 | 16:18 |
| Tengu | ;) | 16:19 |
| chem | Tengu: I'll try that when we have a 3ctl/3ceph upgrade gating somewhere :) | 16:19 |
| Tengu | hehehe | 16:19 |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: fix tox python3 overrides https://review.openstack.org/607753 | 16:19 |
| *** ykarel__ has quit IRC | 16:19 | |
| chem | bye for real now | 16:20 |
| *** chem is now known as chem_gone | 16:20 | |
| Tengu | ++ | 16:20 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Add python2/python3 handling in imagebuilding https://review.openstack.org/610102 | 16:20 |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: fix tox python3 overrides https://review.openstack.org/607753 | 16:20 |
| weshay | mwhahaha, any new thoughts on disabling ara on the tripleo roles? | 16:22 |
| *** rlandy|biab is now known as rlandy | 16:24 | |
| *** ssbarnea has joined #tripleo | 16:25 | |
| rlandy | nhicher: +2'ed https://review.openstack.org/#/c/596432 | 16:25 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Add python2/python3 handling in imagebuilding https://review.openstack.org/610102 | 16:25 |
| mwhahaha | weshay: i don't know that we can, i haven't figured out how to do that yet | 16:28 |
| weshay | for the love | 16:28 |
| *** colonwq has quit IRC | 16:29 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Add new featureset 056 for standalone upgarde. https://review.openstack.org/605363 | 16:31 |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient master: Honor blacklist during temp key injection https://review.openstack.org/610104 | 16:31 |
| weshay | dang it | 16:34 |
| weshay | time out | 16:34 |
| *** salmankhan has quit IRC | 16:35 | |
| *** shardy has quit IRC | 16:37 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: increase the available ram for the bmc node https://review.openstack.org/610108 | 16:45 |
| weshay | rlandy, like so ^? | 16:45 |
| rlandy | weshay: yes - relying on nhicher's change | 16:46 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: increase the available ram for the bmc node https://review.openstack.org/610113 | 16:52 |
| weshay | rlandy, ok.. sorry that will do it then ^ | 16:52 |
| rlandy | weshay: yeah - better | 16:54 |
| *** rdopiera has quit IRC | 16:54 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Fix stackviz and subunit2html report generation https://review.openstack.org/605419 | 17:02 |
| openstackgerrit | Michael Bayer proposed openstack/tripleo-heat-templates master: Implement Global Galera database https://review.openstack.org/609738 | 17:05 |
| *** dprince has quit IRC | 17:05 | |
| *** apevec has joined #tripleo | 17:07 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: standalone support for quickstart on libvirt https://review.openstack.org/591540 | 17:08 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: DNM: test change https://review.openstack.org/610119 | 17:09 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: DNM: test change https://review.openstack.org/610119 | 17:10 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 17:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797600 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 17:10 |
| openstack | Launchpad bug 1797600 in tripleo "CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"" [Critical,Incomplete] | 17:10 |
| *** trown is now known as trown|lunch | 17:16 | |
| *** artom is now known as temka | 17:18 | |
| nhicher | rlandy, weshay: thanks for the +2, do we need to wait for another core to merge it ? | 17:31 |
| weshay | nhicher, them's the rulz ya | 17:31 |
| nhicher | weshay: good to know, thanks =) | 17:32 |
| weshay | mwhahaha, may want to check it out | 17:32 |
| mwhahaha | Review what | 17:33 |
| nhicher | mwhahaha: https://review.openstack.org/#/c/596432/ | 17:33 |
| ssbarnea | weshay: is " Fixed interval looping call 'nova.virt.ironic.driver.IronicDriver._wait_for_active' failed: InstanceDeployFailure: Failed to provision instance 6b50aa1a-1a0f-4e39-86ec-b2f1f3e84d1a: Failed to deploy. Error: Failed to set node power state to power on." a better description for what happens? | 17:34 |
| apevec | weshay, why does 3rd party CI provider need to be defined in openstack-infra/tripleo-ci ? | 17:37 |
| apevec | it's confusing to have oooci split across multiple places | 17:37 |
| weshay | ssbarnea, ya.. but read the scroll back on the channel about hardware provisioning | 17:39 |
| *** mjturek has joined #tripleo | 17:39 | |
| mwhahaha | apevec: it's adding support for it in the CI scripts since assumptions about ovb we're in code for RDO cloud | 17:40 |
| weshay | apevec, it sure is.. however that's what the upstream pinned us into.. | 17:40 |
| *** akrivoka has quit IRC | 17:40 | |
| *** Chaserjim has joined #tripleo | 17:40 | |
| apevec | weshay, which upstream? | 17:40 |
| weshay | apevec, we are working on consolidating that to two repos | 17:40 |
| weshay | something stack | 17:40 |
| apevec | I mean which team | 17:41 |
| apevec | infra required that? | 17:41 |
| apevec | I'd like to explore that requirement, was it that it must be in openstack-infra namespace? | 17:41 |
| apevec | maybe that was historical | 17:42 |
| mwhahaha | It was | 17:42 |
| mwhahaha | That's where everything was originally defined | 17:42 |
| mwhahaha | And we have not evaluated if/how we should get rid of it | 17:43 |
| mwhahaha | I tend to lean on keeping the tripleo-ci repo for job configs and reduce what we carry around CI envs in quickstart | 17:43 |
| weshay | right | 17:44 |
| weshay | agree | 17:44 |
| apevec | shipit | 17:44 |
| weshay | apevec, note that third party ci and upstream ci are tied together via parent jobs now too.. so there is a significant link between the two | 17:47 |
| weshay | tripleo-ci seems like a good place to define third party vars to me | 17:48 |
| *** sai_p has quit IRC | 17:58 | |
| *** ssbarnea_ has joined #tripleo | 18:00 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 18:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797600 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 18:10 |
| openstack | Launchpad bug 1797600 in tripleo "Fixed interval looping call 'nova.virt.ironic.driver.IronicDriver._wait_for_active' failed: InstanceDeployFailure: Failed to set node power state to power on." [Critical,Incomplete] | 18:10 |
| *** trown|lunch is now known as trown | 18:14 | |
| *** rnoriega has joined #tripleo | 18:14 | |
| *** lhinds has joined #tripleo | 18:16 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Improve deep_compare code detection https://review.openstack.org/609998 | 18:24 |
| EvilienM | weshay, mwhahaha: do you want to go ahead with https://review.openstack.org/#/c/609963/ ? | 18:41 |
| EvilienM | or trying to get https://review.openstack.org/#/c/609586/ ? | 18:41 |
| mwhahaha | i'd go with the revert | 18:41 |
| EvilienM | ok | 18:41 |
| EvilienM | mwhahaha: sorry if I missed the info, but is it making gate stuck? | 18:42 |
| EvilienM | mwhahaha: or blocking promotion? | 18:42 |
| EvilienM | is tripleo-ci-centos-7-undercloud-containers timeouting or? | 18:42 |
| EvilienM | because the revert is only for one job | 18:44 |
| EvilienM | we didn't enable podman on all underclouds on master | 18:44 |
| *** larsks has joined #tripleo | 18:44 | |
| mwhahaha | EvilienM: it's labeled as a promotion blocker, i'm not sure though a 60% perf issue is revert-worthy | 18:44 |
| EvilienM | mhhh | 18:45 |
| EvilienM | weshay: insights here? | 18:45 |
| EvilienM | I don't want to revert for nothing | 18:45 |
| EvilienM | it's a step backward for container team | 18:45 |
| EvilienM | but we know the perf issue that we are addressing in the tripleo-common patch | 18:46 |
| EvilienM | i've +2, if it's an *actual* promotion blocker, weshay please approve | 18:48 |
| mwhahaha | EvilienM: https://review.openstack.org/#/c/609993/ | 18:48 |
| EvilienM | mwhahaha: approved | 18:49 |
| openstackgerrit | Harald Jensås proposed openstack/puppet-tripleo master: Fix Undercloud masquerading firewall rules https://review.openstack.org/609858 | 18:59 |
| weshay | sorry back | 19:04 |
| weshay | reading | 19:04 |
| weshay | EvilienM, bogdan measured things and said the undercloud was taking 60% longer to install | 19:05 |
| weshay | or was that skopeo | 19:05 |
| * weshay looks | 19:05 | |
| weshay | so many to choose from | 19:05 |
| EvilienM | weshay: on ONE job | 19:05 |
| weshay | don' | 19:05 |
| EvilienM | fs027 only | 19:05 |
| weshay | don't get all puffy w/ me frenchman | 19:05 |
| EvilienM | so is tripleo-ci-centos-7-undercloud-containers failing or timeouting anywhere? | 19:06 |
| EvilienM | is this job blocking the gate or promotion pipeline, currently? | 19:06 |
| * weshay gets some data and freedom fries | 19:06 | |
| EvilienM | weshay: I'm not puffy, I try to not revert the world :D | 19:07 |
| weshay | :) | 19:07 |
| weshay | that is the USA's job | 19:07 |
| weshay | http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen | 19:07 |
| weshay | I don't have a dog in this fight.. :) | 19:08 |
| weshay | I just want https://review.openstack.org/608589 to land | 19:08 |
| weshay | and rdo-cloud to work | 19:08 |
| EvilienM | I'm afk 1h | 19:09 |
| EvilienM | weshay: tripleo-ci-centos-7-containers-multinode was failing | 19:10 |
| itlinux_ | hello all.. in Pike running this command openstack container image prepare default --output-env-file containers-prep-parameter.yaml gives me an error openstack tripleo container image --help says unknow command | 19:10 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797600 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 19:10 |
| EvilienM | not tripleo-ci-centos-7-undercloud-containers | 19:10 |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 19:10 |
| openstack | Launchpad bug 1797600 in tripleo "Fixed interval looping call 'nova.virt.ironic.driver.IronicDriver._wait_for_active' failed: InstanceDeployFailure: Failed to set node power state to power on." [Critical,Incomplete] | 19:10 |
| itlinux_ | any suggestions? Thanks | 19:10 |
| EvilienM | so it's a wrong assumption to say we should revert the fs27 patch | 19:10 |
| EvilienM | anyway I'm out and back later | 19:10 |
| weshay | ya | 19:10 |
| weshay | aight :) | 19:10 |
| mwhahaha | itlinux_: that's the new command | 19:12 |
| mwhahaha | itlinux_: you need openstack overclodu container image prepare | 19:12 |
| itlinux_ | ahh ok. thanks | 19:13 |
| itlinux_ | looks like is pulling directly form docker.io what is the extra option I need to pass for local repo (PIKE LOCAL RPM Server) Thank you mwhahaha: | 19:15 |
| mwhahaha | itlinux_: containers are only from docker.io, you can upload them to the undercloud and use openstack overcloud container image upload afterwards | 19:15 |
| * mwhahaha tries to dig up the old docs | 19:15 | |
| itlinux_ | ok thank you.. | 19:16 |
| mwhahaha | itlinux_: https://github.com/openstack/tripleo-docs/blob/55478b83a4abed626e1a1dede3b00710c74af0a7/doc/source/install/containers_deployment/overcloud.rst | 19:16 |
| itlinux_ | found this https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html/director_installation_and_usage/configuring-registry_details | 19:17 |
| itlinux_ | thanks | 19:17 |
| mwhahaha | acutally i think https://github.com/openstack/tripleo-docs/blob/c02091115255ee804cf8a10c54bd77afb8e4afb9/doc/source/install/containers_deployment/overcloud.rst | 19:17 |
| mwhahaha | s/tripleomaster/tripleopike | 19:17 |
| itlinux_ | well I see DockerAodhApiImage: docker.io/tripleoupstream/centos-binary-aodh-api:latest | 19:18 |
| itlinux_ | so I guess I have to do some sed here :) | 19:18 |
| mwhahaha | yea | 19:18 |
| mwhahaha | itlinux_: try --help you might be able to specify the namespace | 19:18 |
| itlinux_ | k | 19:19 |
| mwhahaha | which would allow you to not have to sed | 19:19 |
| itlinux_ | yea I was looking at that.. | 19:19 |
| *** dpeacock has quit IRC | 19:19 | |
| itlinux_ | yeap it has namespace.. | 19:20 |
| itlinux_ | very nice.. thanks! | 19:20 |
| *** dpeacock has joined #tripleo | 19:22 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: DNM: Test python error https://review.openstack.org/610145 | 19:31 |
| *** dhill_ has quit IRC | 19:37 | |
| *** dhill_ has joined #tripleo | 19:41 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: DNM: Testing change https://review.openstack.org/610146 | 19:42 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: DNM; Testing https://review.openstack.org/610147 | 19:45 |
| *** apetrich has quit IRC | 19:48 | |
| *** apetrich has joined #tripleo | 19:49 | |
| weshay | dam it | 19:53 |
| weshay | timeout | 19:53 |
| EvilienM | weshay: which job? | 19:53 |
| weshay | not yours :) http://zuul.openstack.org/stream/e7006000e1b04a9d9b12d149d23f29f9?logfile=console.log | 19:53 |
| weshay | https://review.openstack.org/608589 | 19:53 |
| weshay | tripleo-ci-centos-7-scenario001-multinode-oooq-container | 19:54 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/tripleo-ci master: DNM: add upstream-centos-7-vexxhost host https://review.openstack.org/610150 | 20:02 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 20:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797600 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 20:10 |
| openstack | Launchpad bug 1797600 in tripleo "Fixed interval looping call 'nova.virt.ironic.driver.IronicDriver._wait_for_active' failed: InstanceDeployFailure: Failed to set node power state to power on." [Critical,Incomplete] | 20:10 |
| weshay | EvilienM, I think the details you seek are in https://bugs.launchpad.net/tripleo/+bug/1789680 | 20:13 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 20:13 |
| weshay | which of course has bled into more than one issue | 20:13 |
| EvilienM | weshay: not related to podman | 20:14 |
| EvilienM | (just making sure) | 20:14 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: DNM, testing https://review.openstack.org/#/c/609586/ https://review.openstack.org/610035 | 20:14 |
| *** ssbarnea_ has quit IRC | 20:17 | |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo master: Implement Global Galera database https://review.openstack.org/609734 | 20:25 |
| *** apevec has quit IRC | 20:34 | |
| *** sai_p has joined #tripleo | 20:39 | |
| *** ansmith has quit IRC | 20:40 | |
| *** dciabrin has quit IRC | 20:41 | |
| *** Vorrtex has quit IRC | 20:56 | |
| *** trown is now known as trown|outtypewww | 21:00 | |
| *** temka has quit IRC | 21:01 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 21:07 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 21:08 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797600 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 21:10 |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 21:10 |
| openstack | Launchpad bug 1797600 in tripleo "Fixed interval looping call 'nova.virt.ironic.driver.IronicDriver._wait_for_active' failed: InstanceDeployFailure: Failed to set node power state to power on." [Critical,Incomplete] | 21:10 |
| *** lblanchard has quit IRC | 21:14 | |
| *** mcornea has quit IRC | 21:16 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 21:16 |
| *** jamesdenton has quit IRC | 21:24 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 21:26 |
| *** salmankhan has joined #tripleo | 21:40 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: turn on tcpdump debug by default for ovb jobs https://review.openstack.org/610072 | 21:41 |
| *** ansmith has joined #tripleo | 21:43 | |
| *** agopi_ has joined #tripleo | 21:47 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 21:48 |
| *** agopi has quit IRC | 21:49 | |
| *** rlandy has quit IRC | 21:54 | |
| *** jistr has quit IRC | 21:55 | |
| *** jistr has joined #tripleo | 21:56 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 21:58 |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: fix tox python3 overrides https://review.openstack.org/607753 | 21:59 |
| *** tosky has quit IRC | 22:02 | |
| *** jistr has quit IRC | 22:05 | |
| *** agopi_ is now known as agopi | 22:05 | |
| *** jistr has joined #tripleo | 22:06 | |
| *** salmankhan has left #tripleo | 22:07 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 22:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797600 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 22:10 |
| openstack | Launchpad bug 1797600 in tripleo "Fixed interval looping call 'nova.virt.ironic.driver.IronicDriver._wait_for_active' failed: InstanceDeployFailure: Failed to set node power state to power on." [Critical,Incomplete] | 22:10 |
| *** agopi is now known as agopi|brb | 22:13 | |
| *** agopi|brb has quit IRC | 22:17 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 22:18 |
| *** pcaruana has quit IRC | 22:20 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 22:27 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 22:35 |
| *** toure is now known as toure|gone | 22:50 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 22:51 |
| mwhahaha | friday afternoon zuul v3ing | 22:51 |
| *** Chaserjim has quit IRC | 22:53 | |
| *** Chaserjim has joined #tripleo | 22:54 | |
| *** Chaserjim has quit IRC | 22:59 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 23:03 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Remove tripleo-ci-centos-7-3nodes-multinode job https://review.openstack.org/609710 | 23:06 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add nova-scheduler worker support https://review.openstack.org/609056 | 23:06 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/rocky: Add nova-scheduler worker support https://review.openstack.org/610183 | 23:06 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1789680 | 23:10 |
| openstack | Launchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797525 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1797600 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1797525 in tripleo "After the switch to podman & skopeo, undercloud deployment takes +60% longer" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 23:10 |
| openstack | Launchpad bug 1797600 in tripleo "Fixed interval looping call 'nova.virt.ironic.driver.IronicDriver._wait_for_active' failed: InstanceDeployFailure: Failed to set node power state to power on." [Critical,Incomplete] | 23:10 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 23:17 |
| *** owalsh_ has joined #tripleo | 23:18 | |
| *** owalsh has quit IRC | 23:21 | |
| *** agopi|brb has joined #tripleo | 23:25 | |
| *** slaweq has quit IRC | 23:32 | |
| *** Chaserjim has joined #tripleo | 23:33 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Update standalone container prep steps https://review.openstack.org/604487 | 23:34 |
| *** Chaserjim has quit IRC | 23:38 | |
| *** agopi|brb has quit IRC | 23:41 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Update buildimage playbook for zuul v3 https://review.openstack.org/610102 | 23:42 |
| *** artom has joined #tripleo | 23:54 | |
| *** artom has quit IRC | 23:54 | |
| *** EvilienM is now known as EmilienM | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!