| *** holser_ has joined #oooq | 00:27 | |
| *** tosky has quit IRC | 00:29 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (2 more messages) | 00:30 |
|---|---|---|
| *** holser_ has quit IRC | 00:59 | |
| *** dsneddon has quit IRC | 01:30 | |
| *** dsneddon has joined #oooq | 01:35 | |
| *** dsneddon has quit IRC | 01:47 | |
| *** dsneddon has joined #oooq | 02:05 | |
| *** dsneddon has quit IRC | 02:12 | |
| *** saneax has joined #oooq | 02:13 | |
| *** marios has quit IRC | 02:19 | |
| *** chandan_kumar has quit IRC | 02:19 | |
| *** chandan_kumar has joined #oooq | 02:20 | |
| *** ykarel has joined #oooq | 02:21 | |
| *** dsneddon has joined #oooq | 02:27 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (2 more messages) | 02:30 |
| *** dsneddon has quit IRC | 02:32 | |
| *** panda has quit IRC | 02:49 | |
| *** ykarel has quit IRC | 02:49 | |
| *** panda has joined #oooq | 02:51 | |
| *** dsneddon has joined #oooq | 02:59 | |
| *** ykarel has joined #oooq | 03:02 | |
| *** dsneddon has quit IRC | 03:04 | |
| *** dsneddon has joined #oooq | 03:05 | |
| *** dsneddon has quit IRC | 03:12 | |
| *** apetrich has quit IRC | 03:14 | |
| *** ykarel_ has joined #oooq | 03:22 | |
| *** ykarel has quit IRC | 03:25 | |
| *** dsneddon has joined #oooq | 03:33 | |
| *** dsneddon has quit IRC | 03:38 | |
| *** udesale has joined #oooq | 03:57 | |
| *** ykarel_ is now known as ykarel | 04:04 | |
| *** dsneddon has joined #oooq | 04:06 | |
| *** rlandy|bbl is now known as rlandy | 04:07 | |
| *** rlandy has quit IRC | 04:07 | |
| *** dsneddon has quit IRC | 04:17 | |
| *** udesale has quit IRC | 04:21 | |
| *** udesale has joined #oooq | 04:22 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ (1 more message) | 04:30 |
| *** dsneddon has joined #oooq | 04:39 | |
| *** dsneddon has quit IRC | 04:44 | |
| *** udesale has quit IRC | 04:47 | |
| *** udesale has joined #oooq | 04:47 | |
| *** jaosorior has joined #oooq | 05:15 | |
| *** dsneddon has joined #oooq | 05:17 | |
| *** dsneddon has quit IRC | 05:22 | |
| *** dsneddon has joined #oooq | 05:23 | |
| *** weshay has quit IRC | 05:29 | |
| *** udesale has quit IRC | 05:31 | |
| *** udesale has joined #oooq | 05:33 | |
| *** ratailor has joined #oooq | 05:56 | |
| *** ykarel has quit IRC | 06:03 | |
| *** ykarel has joined #oooq | 06:18 | |
| *** marios has joined #oooq | 06:18 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ (1 more message) | 06:30 |
| *** quiquell|off is now known as quiquell | 06:35 | |
| quiquell | sshnaidm|afk: couldn't use docker-compose secrets without docker-ce | 06:56 |
| quiquell | At least the mounting ones | 06:58 |
| quiquell | Have to test creating secrets with docker secret | 06:58 |
| quiquell | Teorically docker 1.13.1 supports secrets | 07:03 |
| quiquell | We should not need docker-ce | 07:03 |
| *** jfrancoa has joined #oooq | 07:29 | |
| *** ykarel is now known as ykarel|lunch | 07:32 | |
| *** apetrich has joined #oooq | 07:32 | |
| *** quiquell is now known as quiquell|brb | 07:47 | |
| *** saneax has quit IRC | 07:47 | |
| *** jtomasek has joined #oooq | 07:59 | |
| *** ratailor_ has joined #oooq | 08:05 | |
| *** ratailor has quit IRC | 08:07 | |
| *** udesale has quit IRC | 08:11 | |
| *** saneax has joined #oooq | 08:11 | |
| *** udesale has joined #oooq | 08:11 | |
| *** ccamacho has joined #oooq | 08:14 | |
| *** udesale has quit IRC | 08:16 | |
| *** udesale has joined #oooq | 08:16 | |
| *** udesale has quit IRC | 08:17 | |
| *** udesale has joined #oooq | 08:17 | |
| *** dsneddon has quit IRC | 08:17 | |
| *** ykarel|lunch is now known as ykarel | 08:22 | |
| *** chem has joined #oooq | 08:24 | |
| *** sanjayu_ has joined #oooq | 08:25 | |
| *** saneax has quit IRC | 08:28 | |
| *** kopecmartin|off is now known as kopecmartin | 08:28 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ (1 more message) | 08:30 |
| *** dsneddon has joined #oooq | 08:37 | |
| *** udesale has quit IRC | 08:39 | |
| *** udesale has joined #oooq | 08:39 | |
| *** jpena|off is now known as jpena | 08:39 | |
| *** tosky has joined #oooq | 08:40 | |
| *** brault has joined #oooq | 08:42 | |
| *** brault has quit IRC | 08:45 | |
| *** udesale has quit IRC | 08:49 | |
| *** udesale has joined #oooq | 08:49 | |
| chandan_kumar | marios: Hey | 09:01 |
| chandan_kumar | marios: https://github.com/openstack/ansible-config_template is needed by os_tempest | 09:02 |
| chandan_kumar | but it is ansible action plugin | 09:02 |
| chandan_kumar | marios: https://review.openstack.org/631214 I have proposed this so that it will be get discovered by tq https://github.com/openstack/tripleo-quickstart/blob/master/ansible.cfg | 09:03 |
| chandan_kumar | but it is not working here https://review.openstack.org/627500 | 09:03 |
| chandan_kumar | do i need to add action_plugins and library path for making it discover? | 09:04 |
| chandan_kumar | marios: http://logs.openstack.org/00/627500/46/check/tripleo-ci-centos-7-standalone-os-tempest/ce0c4bc/job-output.txt.gz#_2019-01-17_07_16_50_595151 | 09:05 |
| *** ccamacho has quit IRC | 09:07 | |
| *** udesale has quit IRC | 09:08 | |
| *** udesale has joined #oooq | 09:09 | |
| marios | chandan_kumar: maybe you need tempest_plugins on the job definition? otherwise not sure | 09:09 |
| *** bogdando has joined #oooq | 09:22 | |
| *** dtantsur|afk is now known as dtantsur | 09:25 | |
| *** skramaja has joined #oooq | 09:28 | |
| quiquell|brb | marios: What reviews do I have to focus on ? | 09:44 |
| marios | quiquell|brb: o/ | 09:45 |
| marios | quiquell|brb: https://review.openstack.org/#/c/631227/ needs some votes but will update the commit message in a sec | 09:45 |
| marios | and the depends on | 09:45 |
| marios | and parent | 09:45 |
| marios | please thanks | 09:45 |
| marios | :) | 09:45 |
| quiquell|brb | marios: puff I see a lot of duplicity here https://review.rdoproject.org/r/#/c/18079/10/zuul.d/standalone-jobs.yaml | 09:46 |
| marios | quiquell|brb: yeah its how it goes with the periodic jobs though right | 09:46 |
| quiquell|brb | marios: That's very bad... | 09:46 |
| quiquell|brb | marios: change upstrem will not be reflected at RDO promotions if we forget | 09:47 |
| marios | quiquell|brb: :( i'm sorry? | 09:47 |
| *** quiquell|brb is now known as quiquell | 09:47 | |
| marios | quiquell|brb: yeah its a 'known' thing like we have to track two places | 09:47 |
| quiquell | marios: maybe we override periodic stuff instead of featurset stuff | 09:47 |
| marios | quiquell: so weshay was saying maybe somekind of job scraper that alerts when the periodic<->upstream are out of sync | 09:47 |
| quiquell | marios: puff | 09:47 |
| marios | quiquell: in our "spare time" | 09:47 |
| marios | :D | 09:47 |
| quiquell | marios: My 2cents is to create periodics for scenario using upstream as parent | 09:49 |
| quiquell | marios: Can you point me to the tripleo-ci-base-standalone-periodic definition ? | 09:49 |
| marios | quiquell: https://codesearch.rdoproject.org/?q=tripleo-ci-base-standalone-periodic&i=nope&files=&repos= | 09:49 |
| marios | (https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo-rdo-base.yaml) | 09:49 |
| marios | chandan_kumar: ;) ^ | 09:50 |
| quiquell | marios: I know you are nog goint to like it | 09:50 |
| marios | quiquell: well that sounds nice the parenting i mean | 09:50 |
| marios | quiquell: but it sounds like a good task for next sprint | 09:50 |
| quiquell | marios: what about we create at "config" directly the periodic scenarios jobs without parenting tripleo-ci-base-standalone-periodic | 09:50 |
| quiquell | marios: but parenting upstream | 09:50 |
| quiquell | marios: and in those put the periodc stuff missing | 09:51 |
| quiquell | marios: ack | 09:51 |
| quiquell | marios: why do we need the double quote here --transport-url="'TRANSPORTURL'" ? | 09:54 |
| marios | quiquell: good question add acomment and i can update it with the comit message | 09:57 |
| marios | ykarel: ^ (https://review.openstack.org/#/c/631227/edit/docker/services/nova-api.yaml@496 | 09:57 |
| quiquell | marios: Do we exercise not default anywhere ? | 09:57 |
| marios | quiquell: you mean what we get from hiera? | 09:58 |
| marios | quiquell: like not rabbit? | 09:58 |
| marios | quiquell: (yeah in this scen3 which it fixes, we can point to the conf and find amqp instead of rabbit) | 09:58 |
| marios | in logs i mean | 09:58 |
| quiquell | marios: Do any of the jobs there us amqp to verify ? | 09:59 |
| ykarel | quiquell, marios because bash expansion to work | 09:59 |
| quiquell | marios: or different port ? | 09:59 |
| ykarel | quiquell, scenario003 uses amqp and port:31459 | 09:59 |
| ykarel | rest all rabbit and 5672 | 09:59 |
| quiquell | ykarel: ack | 10:00 |
| quiquell | ykarel: what log to look for ? | 10:00 |
| *** holser_ has joined #oooq | 10:00 | |
| ykarel | quiquell, for checking hiera? | 10:00 |
| marios | quiquell: sec (is on bug) | 10:00 |
| quiquell | marios: ack will look in the bag | 10:00 |
| quiquell | I was just bloody lazy | 10:01 |
| marios | http://logs.openstack.org/98/604298/176/check/tripleo-ci-centos-7-scenario003-standalone/b2d7fd7/logs/undercloud/var/log/config-data/nova/etc/nova/nova.conf.txt.gz | 10:01 |
| marios | e.g. here | 10:01 |
| marios | quiquell: in "broken" it looksed like transport_url=rabbit://guest:kSBh78C58WIEt0esxvEfkcRpM@centos-7-rax-iad-0001678323.internalapi.localdomain:5672/?ssl=0 | 10:01 |
| marios | (in oslo_messaging_notifications section) | 10:01 |
| marios | becuse it was hard coded | 10:01 |
| *** ccamacho has joined #oooq | 10:01 | |
| ykarel | marios, nope that's not the issue ^^ | 10:01 |
| marios | but it should looks like transport_url=amqp://guest:MxaYJlOW0NjPMNu3t8SkMhoov@centos-7-rax-iad-0001678323.internalapi.localdomain:31459/?ssl=0 | 10:01 |
| marios | ykarel: ? | 10:02 |
| quiquell | transport_url=amqp://guest:MxaYJlOW0NjPMNu3t8SkMhoov@centos-7-rax-iad-0001678323.internalapi.localdomain:31459/?ssl=0 | 10:02 |
| quiquell | yep | 10:02 |
| quiquell | Looks like working let check the others | 10:02 |
| ykarel | marios, actual issue is nova cell configuration | 10:02 |
| ykarel | nova.conf was correct | 10:02 |
| ykarel | from beginning | 10:02 |
| quiquell | ykarel: so where to look at ? | 10:02 |
| ykarel | quiquell, logs are missing in ci for that script | 10:02 |
| ykarel | they exist at /var/lib/docker-config-script | 10:03 |
| marios | ykarel: then why are we changing the hard coded rabbit to amqp | 10:03 |
| marios | ykarel: or it is still some ongoing issue in nova that we are working around here? | 10:04 |
| ykarel | marios, we are fetching from hiera | 10:04 |
| marios | ykarel: yes? | 10:04 |
| ykarel | marios, rabbit and 5672 is hardcoded during cell configuration, this is the issue | 10:04 |
| ykarel | in scenario003 we deploy qdrouterd which is amqp and 31459 | 10:05 |
| ykarel | so with patch we are not hardcoding, instead fetch from hiera | 10:05 |
| ykarel | which will work for both amqp and rabbit | 10:05 |
| marios | ykarel: yes | 10:05 |
| marios | ykarel: i posted the patch if you recall ;) to fetch from hiera | 10:05 |
| marios | ykarel: i am confused as i thought you were saying the issue was not the one we are fixing here | 10:06 |
| ykarel | marios, yes u posted hiera, but had issues, so i fixed it | 10:06 |
| ykarel | basically bash issues | 10:06 |
| ykarel | like u were using () instead of {} | 10:06 |
| ykarel | for setting default | 10:06 |
| marios | ykarel: yeah sorry for those :) i left before zuul reported so didn't get a chance to fixup and thanks for the update | 10:07 |
| marios | ykarel: ok then, maybe miscommunication here thanks | 10:08 |
| ykarel | marios, ack | 10:08 |
| ykarel | so we are basically doing u nova -s /bin/bash -c "/usr/bin/nova-manage cell_v2 update_cell --cell_uuid ca965ef0-5f69-46bd-9824-ab9d47feab8b --name=default --database_connection='{scheme}://{username}:{password}@{hostname}/nova?{query}' --transport-url='rabbit://guest:bJBSqdrupPe7IjG8dg9uRjltQ@subnode-0.internalapi.localdomain:5672/?ssl=0'" --> su nova -s /bin/bash -c "/usr/bin/nova-manage cell_v2 update_cell --cell_uuid ca965ef0-5f69-46bd-9824-ab9d47feab8b | 10:09 |
| ykarel | --name=default --database_connection='{scheme}://{username}:{password}@{hostname}/nova?{query}' --transport-url='$(hiera -c /etc/puppet/hiera.yaml oslo_messaging_rpc_scheme rabbit)://guest:bJBSqdrupPe7IjG8dg9uRjltQ@subnode-0.internalapi.localdomain:$(hiera -c /etc/puppet/hiera.yaml oslo_messaging_rpc_port 5672)/?ssl=0'" | 10:09 |
| ykarel | "" are used for bash expansion of $ | 10:10 |
| ykarel | quiquell, is fedora standalone issue knownw? | 10:11 |
| marios | ykarel: ack so we'll keep the "" quiquell | 10:11 |
| marios | ykarel: oh no thats unrelated, i thought that is what you were commenting on here | 10:12 |
| marios | 12:10 < ykarel> "" are used for bash expansion of $ | 10:12 |
| ykarel | marios, this is related :) | 10:12 |
| ykarel | the fedora one is not | 10:13 |
| ykarel | quiquell, http://logs.openstack.org/27/631227/6/check/tripleo-ci-fedora-28-standalone/a7c2d88/logs/tempest.html.gz | 10:13 |
| marios | ykarel: oh its the TRANSPORTURL you pointed to there | 10:13 |
| ykarel | quiquell, and job is taking 3 hours | 10:13 |
| ykarel | marios, yes | 10:13 |
| marios | yeah it is going to time out i think | 10:13 |
| marios | thats what we're waiting for | 10:13 |
| ykarel | it already timedout | 10:13 |
| ykarel | so i asked quiquell as i remember he used to take care of this job | 10:13 |
| quiquell | ykarel: So we have break something, still don't know if we want to make the job voting so it does not break | 10:14 |
| quiquell | ykarel: Now that it's release is a ruck/rover thing I suppose, normal flow open lp and ask them | 10:15 |
| quiquell | ssbarnea|rover, arxcruz|ruck: ^ f28 job is broken | 10:15 |
| ykarel | quiquell, ack may be they are already aware | 10:15 |
| ykarel | so it's passing sometimes: http://zuul.openstack.org/builds?job_name=tripleo-ci-fedora-28-standalone | 10:16 |
| ssbarnea|rover | quiquell: if is failing randomly on tempest I have reasons to believe is not ci related, right? | 10:17 |
| quiquell | ssbarnea|rover: what about noop review is failing there too ? | 10:18 |
| ssbarnea|rover | lets try to do a logstash query to see exactly on which jobs is happening. | 10:18 |
| ssbarnea|rover | quiquell: ykarel : please have a look at this http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%20tempest.common.compute%20tempest.lib.exceptions.TimeoutException%3A%20Request%20timed%20out%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1 | 10:23 |
| quiquell | ssbarnea|rover: thanks | 10:24 |
| ssbarnea|rover | i am not sure if the search term is ok, double check it. | 10:24 |
| ssbarnea|rover | but based on the results I would say: is not f28 fault, is likely tempest related. | 10:25 |
| *** sshnaidm|afk is now known as sshnaidm | 10:25 | |
| quiquell | sshnaidm: o/ | 10:26 |
| sshnaidm | quiquell, hey | 10:26 |
| quiquell | sshnaidm: Found a way to inject keys without secrets (this means without docker-ce) | 10:27 |
| quiquell | sshnaidm: https://review.rdoproject.org/r/#/c/18352 | 10:27 |
| quiquell | sshnaidm: Have to fix something for CI but it's working locally | 10:28 |
| quiquell | sshnaidm: Let me know what you think | 10:28 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode- (1 more message) | 10:30 |
| *** dsneddon has quit IRC | 10:33 | |
| marios | quiquell: vote please ?https://review.openstack.org/#/c/631228/ | 10:35 |
| marios | panda: ^ ? please sshnaidm | 10:36 |
| marios | is depends-on for https://review.openstack.org/#/c/631227/ | 10:36 |
| marios | (so the job runs on nova changes) | 10:36 |
| quiquell | marios: done | 10:36 |
| marios | thanks | 10:36 |
| panda | marios: +W | 10:37 |
| quiquell | marios: what else ? | 10:38 |
| marios | quiquell: i think good for now thanks | 10:38 |
| ykarel | ssbarnea|rover, the tempest.lib.exceptions.TimeoutException can be different reason, more likely we are seeing http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22within%20the%20required%20time%20(500%20s).%20Current%20status%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1 | 10:40 |
| ssbarnea|rover | ykarel: this msg is too generic to be trusted, needs to be combined with something more specific. | 10:41 |
| ykarel | ssbarnea|rover, agree, just wanted to avoid regex so pointed minimal | 10:42 |
| marios | thanks panda | 10:42 |
| marios | quiquell: revote? https://review.openstack.org/#/c/631227/7 | 10:42 |
| ssbarnea|rover | ykarel: you are not allowed to use regex or even wilcards, but you can use multiple conditions. | 10:42 |
| ykarel | ssbarnea|rover, ack | 10:42 |
| sshnaidm | quiquell, why do you base64 them though? | 10:42 |
| ssbarnea|rover | ykarel: mostly due to extreme performance costs. | 10:42 |
| ykarel | ssbarnea|rover, ack | 10:43 |
| quiquell | ssbarnea|rover: there issue with multiline env values with docker-compose | 10:45 |
| quiquell | sshnaidm: I mean | 10:45 |
| quiquell | :-) | 10:45 |
| sshnaidm | quiquell, I see | 10:46 |
| sshnaidm | quiquell, commented | 10:46 |
| *** sanjayu_ has quit IRC | 10:49 | |
| quiquell | sshnaidm: ok, answered | 10:50 |
| sshnaidm | quiquell, if we take 3 files, it's really doesn't matter to take another one | 10:52 |
| sshnaidm | quiquell, so we'll need to have on host upstream and rdo gerrit keys files in addition? | 10:53 |
| quiquell | sshnaidm: well matters at naming, maybe user don't have id_rsa.pub as pub key and we have to do mappins and the like | 10:53 |
| quiquell | sshnaidm: Can be the same | 10:53 |
| sshnaidm | quiquell, why would user not have id_rsa.pub if he has id_rsa? | 10:53 |
| quiquell | sshnaidm: Is not mandatory | 10:54 |
| sshnaidm | quiquell, I think 100$ of our dev have it :) | 10:54 |
| quiquell | sshnaidm: The lest we take from host the better | 10:54 |
| quiquell | s/lest/less/ | 10:54 |
| quiquell | sshnaidm: what the issue using ssh-keygen ? | 10:54 |
| sshnaidm | quiquell, not really the issue, but just complicating things instead of taking just another file, and not really secret one - unlike id_rsa | 10:55 |
| sshnaidm | quiquell, I don't think it matters how much we take from host also.. | 10:55 |
| quiquell | sshnaidm: It matters we reduce the requisites from users | 10:56 |
| sshnaidm | quiquell, and what is value? | 10:56 |
| quiquell | sshnaidm: even if 99.9% have them, this 00.1% is going to fail | 10:56 |
| *** udesale has quit IRC | 10:56 | |
| sshnaidm | quiquell, I think it's full 100$ :D | 10:56 |
| sshnaidm | s/$/% | 10:56 |
| quiquell | sshnaidm: you don't know | 10:56 |
| quiquell | sshnaidm: It's not that much of complexity come one | 10:57 |
| quiquell | come on | 10:57 |
| sshnaidm | quiquell, I really don't understand why we need this and what the problem to take even all files from host :) | 10:57 |
| sshnaidm | quiquell, anyway, seems like it failed.. | 10:58 |
| *** dsneddon has joined #oooq | 10:58 | |
| quiquell | sshnaidm: Yep that the main issue, let's fix it first | 10:58 |
| sshnaidm | quiquell, so you will require from users to have separate files for upstream and rdo keys? | 10:58 |
| quiquell | sshnaidm: Locally it works, but do you see this approach correct ? | 10:58 |
| sshnaidm | quiquell, yeah, totally | 10:58 |
| quiquell | sshnaidm: nope, they can ust set upstream_key_name and rdo_key_name to the same | 10:59 |
| quiquell | sshnaidm: In my case is id_rsa | 10:59 |
| quiquell | sshnaidm: in fact by defaul they all are id_rsa | 10:59 |
| quiquell | sshnaidm: but inside the container they are diffent files with same content | 10:59 |
| *** jtomasek has quit IRC | 11:00 | |
| sshnaidm | quiquell, I see | 11:00 |
| *** dsneddon has quit IRC | 11:02 | |
| *** jtomasek has joined #oooq | 11:07 | |
| quiquell | sshnaidm: I have being able to reproduce the CI issue with CI(CI) \o/ :-) | 11:19 |
| *** sanjayu_ has joined #oooq | 11:19 | |
| *** dsneddon has joined #oooq | 11:31 | |
| sshnaidm | quiquell, so CI in CI reproduces CI? :D | 11:36 |
| *** dsneddon has quit IRC | 11:36 | |
| quiquell | sshnaidm: Yep I have the schduler dying here as in RDO jobs | 11:37 |
| quiquell | sshnaidm: Still don't know why | 11:37 |
| quiquell | sshnaidm: Locally it works so maybe it's realted to the gerrit tripleo user I created | 11:37 |
| panda | riiight, but are we able to reproduce the reproducer now ? | 11:54 |
| panda | we may want to repoduce the CI reproducer on the CI too | 11:55 |
| quiquell | panda: We can reproducer the reproducer CI in the reproducer itself | 11:55 |
| quiquell | panda: It's failing consistent with RDO | 11:56 |
| quiquell | panda: so I can autohold the node and poke around | 11:56 |
| quiquell | Pufff we may have hit a limit at openstack infra :-( | 11:57 |
| sshnaidm | quiquell, which limits? | 11:59 |
| *** dsneddon has joined #oooq | 12:00 | |
| *** holser_ is now known as holser|burger | 12:01 | |
| quiquell | sshnaidm: 2019-01-17 12:01:16,859 - paramiko.transport - INFO - Disconnect (code 12): Too many concurrent connections (64) - max. allowed: 64 | 12:02 |
| quiquell | sshnaidm: Maybe they have take down our tripleo gerrit user :-( | 12:02 |
| *** jtomasek has quit IRC | 12:05 | |
| sshnaidm | quiquell, worth to check in #openstack-infra | 12:05 |
| sshnaidm | quiquell, but how is it 64 connections? | 12:05 |
| *** ratailor__ has joined #oooq | 12:05 | |
| *** jtomasek has joined #oooq | 12:05 | |
| quiquell | sshnaidm: going to try with the user locally at my laptop | 12:07 |
| quiquell | To see if we have the issue | 12:07 |
| sshnaidm | quiquell, what is name of user? | 12:08 |
| *** ratailor_ has quit IRC | 12:08 | |
| quiquell | sshnaidm: it's failing accessing project-config so upstream, user tripleo.ci | 12:10 |
| quiquell | sshnaidm: going to check zuul code maybe we can reduce concurrency | 12:11 |
| *** ratailor__ has quit IRC | 12:17 | |
| quiquell | sshnaidm: I have dump scheduler logs at job post | 12:19 |
| quiquell | sshnaidm: The error looks different | 12:19 |
| quiquell | :-/ | 12:19 |
| quiquell | http://logs.rdoproject.org/52/18352/12/check/tripleo-ci-reproducer-fedora-28/0aa3569/tripleo-ci-reproducer/docker-compose.log | 12:20 |
| quiquell | sshnaidm: argg fuck | 12:21 |
| quiquell | sshnaidm: forget about it | 12:21 |
| quiquell | I have break gerritconfig | 12:22 |
| quiquell | :-/ | 12:22 |
| *** udesale has joined #oooq | 12:23 | |
| *** jtomasek has quit IRC | 12:23 | |
| *** jtomasek has joined #oooq | 12:24 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/560445, stable/queens: (1 more message) | 12:30 |
| marios | folks review on this please https://review.openstack.org/#/c/631227/ needed for the bug in the commit message (and you can see green scenario003-standalone on it thanks) | 12:40 |
| *** radez has joined #oooq | 12:44 | |
| *** jpena is now known as jpena|lunch | 12:45 | |
| *** irclogbot_0 has quit IRC | 12:47 | |
| *** irclogbot_0 has joined #oooq | 12:57 | |
| sshnaidm | quiquell, this one? scheduler_1 | failed: [localhost] (item={'name': 'gerrit', 'type': 'ssh-ed25519'}) => {"changed": false, "item": {"name": "gerrit", "type": "ssh-ed25519"}, "msg": "Host parameter does not match hashed host field in supplied key"} | 12:59 |
| sshnaidm | quiquell, let's separate all log files, it's difficult to investigate so.. | 13:00 |
| quiquell | sshnaidm: this is fixed look at latest run | 13:04 |
| *** dsneddon has quit IRC | 13:04 | |
| *** agopi has joined #oooq | 13:11 | |
| *** dsneddon has joined #oooq | 13:17 | |
| quiquell | sshnaidm: same place with my user and keys is working fine :-/ | 13:18 |
| *** rlandy has joined #oooq | 13:19 | |
| quiquell | sshnaidm: maybe there is a process stuck somewhere with this user tripleo.ci | 13:19 |
| quiquell | sshnaidm: and this is affecting CI | 13:19 |
| rlandy | quiquell: hello ... when you have a moment, let me know what you think of https://review.openstack.org/#/c/631067/2/roles/create-zuul-based-reproducer/templates/reproducer-zuul-based-quickstart.sh.j2 | 13:20 |
| rlandy | questions in the file | 13:20 |
| *** dsneddon has quit IRC | 13:21 | |
| quiquell | rlandy: Will take a look after lunch | 13:22 |
| quiquell | sshnaidm: Are you firing up the CI(CI) somewhere ? | 13:22 |
| rlandy | quiquell: no worries - we will discuss at meeting | 13:22 |
| sshnaidm | quiquell, no | 13:22 |
| *** jtomasek has quit IRC | 13:22 | |
| *** trown|outtypewww is now known as trown | 13:23 | |
| *** zul has joined #oooq | 13:31 | |
| *** jtomasek has joined #oooq | 13:32 | |
| *** vinaykns has joined #oooq | 13:36 | |
| *** quiquell is now known as quiquell|lunch | 13:38 | |
| *** holser|burger is now known as holser_ | 13:39 | |
| *** jpena|lunch is now known as jpena | 13:41 | |
| *** irclogbot_0 has quit IRC | 13:45 | |
| quiquell|lunch | rlandy, sshnaidm: c7,f28 repro CI passing https://review.rdoproject.org/r/#/c/18352/ | 13:46 |
| sshnaidm | quiquell|lunch, so what was the problem? | 13:47 |
| quiquell|lunch | sshnaidm: Have just recreated the user | 13:48 |
| quiquell|lunch | sshnaidm: We will have to monitor in case it happend again | 13:49 |
| sshnaidm | quiquell|lunch, recreate? | 13:49 |
| quiquell|lunch | sshnaidm: delete user at ubuntu one and create it again with same ssh pub key | 13:49 |
| quiquell|lunch | sshnaidm: I suspenct the CI(CI) thing using this user is not a good thing | 13:49 |
| quiquell|lunch | sshnaidm: We have to use our personal users for that | 13:50 |
| sshnaidm | hmm, weird | 13:50 |
| quiquell|lunch | sshnaidm: yep | 13:50 |
| quiquell|lunch | sshnaidm: good thing centos is working now | 13:50 |
| quiquell|lunch | Damn I have to finish lunch | 13:50 |
| sshnaidm | \o/ | 13:50 |
| *** weshay has joined #oooq | 13:51 | |
| *** irclogbot_0 has joined #oooq | 13:54 | |
| *** dsneddon has joined #oooq | 13:55 | |
| rlandy | nice | 13:59 |
| weshay | marios, quiquell|lunch panda rlandy ssbarnea|rover sshnaidm mtg in 1min | 13:59 |
| *** quiquell|lunch is now known as quiquell | 13:59 | |
| quiquell | I am back | 13:59 |
| *** dsneddon has quit IRC | 13:59 | |
| quiquell | rlandy: Now that I see maybe you are right I we want a template for the zuul.yaml in the repro so we keep things there | 14:01 |
| rlandy | quiquell: design is for discussion | 14:01 |
| rlandy | want to narrow that down so I can work on the launcher role | 14:01 |
| quiquell | rlandy: Also all the checks about the cloud can be done in the role | 14:03 |
| rlandy | quiquell; let's see if we cab get the team's overall basica ik on eth script | 14:03 |
| weshay | chandan_kumar, can you join | 14:03 |
| rlandy | can | 14:03 |
| quiquell | rlandy: ack | 14:03 |
| weshay | quiquell, where you at? | 14:03 |
| rlandy | and then nail down the details | 14:03 |
| quiquell | rlandy: So for now focus in the workflow more than where to put the logic, is that it ? | 14:05 |
| rlandy | quiquell: ack - pls comment on the review and I'll move it | 14:06 |
| rlandy | just wanted to put the workflow somewhere | 14:06 |
| rlandy | for top to bottom review | 14:06 |
| quiquell | rlandy: ack | 14:06 |
| chandan_kumar | weshay: joined | 14:06 |
| quiquell | rlandy: ok, commented | 14:12 |
| quiquell | holy sh libvirt + fedora + centos passing https://review.rdoproject.org/r/#/c/18121/ | 14:12 |
| quiquell | :-) | 14:12 |
| sshnaidm | ssbarnea|rover, arxcruz|ruck do you know about ovb problem "No more IP addresses available on network 1f6dd7c9-1c87-4310-a1c3-bd7c0346d6ad."? | 14:13 |
| ssbarnea|rover | sshnaidm: nope. | 14:14 |
| ssbarnea|rover | sshnaidm: arx is on pto today. | 14:14 |
| *** ykarel is now known as ykarel|away | 14:15 | |
| weshay | ssbarnea|rover, start reviewing marios's promotion job reviews in rdo | 14:16 |
| weshay | thanks | 14:16 |
| ssbarnea|rover | sshnaidm: do you have links? i wonder if it run out of addresses or if the network config was messed. got something similar when the tenant run out of resources (net addrs) | 14:16 |
| sshnaidm | ssbarnea|rover, https://review.rdoproject.org/zuul/stream/fd6c54c3de5442e7add7c082a4b428c7?logfile=console.log | 14:17 |
| sshnaidm | ssbarnea|rover, let's see, maybe just transient issue.. | 14:17 |
| ssbarnea|rover | sshnaidm: yep, lets see what happens on retry | 14:18 |
| sshnaidm | ssbarnea|rover, I see in cockpit 20 stack failed to create.. | 14:18 |
| marios | weshay: ssbarnea|rover https://review.rdoproject.org/r/#/q/topic:standalone-scenario-promotion | 14:22 |
| marios | quiquell: did you add https://tree.taiga.io/project/tripleo-ci-board/task/616 by mistake under the centralise layout story ? | 14:24 |
| quiquell | marios: could be | 14:30 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/560445, stable/queens: (1 more message) | 14:30 |
| quiquell | marios: fixed | 14:31 |
| marios | quiquell: ack | 14:32 |
| *** dsneddon has joined #oooq | 14:33 | |
| *** ykarel|away has quit IRC | 14:33 | |
| *** brault has joined #oooq | 14:33 | |
| *** brault has quit IRC | 14:33 | |
| ssbarnea|rover | sshnaidm: i think is a graph issue, I do not see more than once occasional create failure. at least his is what the popup displays. | 14:35 |
| ssbarnea|rover | this graph is very hard to read, not sure what the overlapping lines mean. | 14:36 |
| *** chem has quit IRC | 14:36 | |
| *** chem has joined #oooq | 14:37 | |
| *** dsneddon has quit IRC | 14:37 | |
| ssbarnea|rover | exported data show no CREATE_FAILED | 14:38 |
| weshay | ssbarnea|rover, fyi.. making you qe on https://tree.taiga.io/project/tripleo-ci-board/us/571?milestone=215191 as you had an interest in the api being stable | 14:59 |
| ssbarnea|rover | weshay: ok! thanks. | 14:59 |
| ssbarnea|rover | weshay: bj or later? | 15:05 |
| *** dsneddon has joined #oooq | 15:06 | |
| weshay | ssbarnea|rover, ya.. sorry for the delay .. mtg is running over | 15:07 |
| *** jfrancoa has quit IRC | 15:10 | |
| *** dsneddon has quit IRC | 15:11 | |
| sshnaidm | ssbarnea|rover, what | 15:14 |
| sshnaidm | is the problem with graph? | 15:14 |
| sshnaidm | ssbarnea|rover, you can see "CREATE_FAILED" is 20 | 15:15 |
| sshnaidm | ssbarnea|rover, I think it's pretty explaining | 15:15 |
| ssbarnea|rover | sshnaidm: it was a problem, but after doing a refresh it does render correctly now, with clear failures. | 15:16 |
| ssbarnea|rover | but weshould change the graph config to be stacked instead of overlapped. | 15:16 |
| sshnaidm | ssbarnea|rover, clear your browser cache | 15:16 |
| ssbarnea|rover | we already know that we have a limit of 150 stacks, so i am not surprised they fail. | 15:18 |
| weshay | ssbarnea|rover, need a couple min, then blue | 15:18 |
| rlandy | quiquell: any time left to talk about launcher role? | 15:18 |
| quiquell | rlandy: yep, let me go to the cave | 15:18 |
| sshnaidm | ssbarnea|rover, what do you mean?? | 15:19 |
| sshnaidm | ssbarnea|rover, how is that related at all | 15:19 |
| ssbarnea|rover | sshnaidm: i think you told me that the stack limit was increased to 150, yesterday right. | 15:19 |
| sshnaidm | ssbarnea|rover, yeah, and? | 15:19 |
| quiquell | rlandy: I am at your blue now | 15:20 |
| rlandy | joining | 15:20 |
| ssbarnea|rover | the graph goes up to 200, which > 150. | 15:20 |
| sshnaidm | ssbarnea|rover, it goes only now, because of a lot of "create_failed" stacks, you confuse between the reason and result | 15:22 |
| sshnaidm | ssbarnea|rover, I'm going to lower it to 100, but only for help to clean up everything there, it's not the reason for stack failures | 15:23 |
| weshay | ssbarnea|rover, ok.. ready | 15:23 |
| ssbarnea|rover | i do not have access to the tenant yet. | 15:23 |
| sshnaidm | weshay, jpena please: https://softwarefactory-project.io/r/14794 | 15:23 |
| sshnaidm | ssbarnea|rover, which tenant?? | 15:24 |
| *** dsneddon has joined #oooq | 15:24 | |
| sshnaidm | ssbarnea|rover, do you know how OVB jobs work? | 15:26 |
| *** jfrancoa has joined #oooq | 15:27 | |
| *** dsneddon has quit IRC | 15:29 | |
| *** ykarel|away has joined #oooq | 15:37 | |
| ska | Is an oooq system similar enough to a bare-metal ooo for testing purposes? | 15:38 |
| panda | ska: oooq drives installation and testing of bare-metal ooo. | 15:40 |
| panda | ska: OOO is the installer. OOOQ emulates the admin that would launch a OOO installation in a selection of environments | 15:42 |
| ska | We just need a single OOO system for testing purposes only. We don't need any performance. What do you reocmmend for an easy ooo deployment? | 15:46 |
| chandan_kumar | panda: is there a way to do something like this in runtime for config_template plugin http://git.openstack.org/cgit/openstack/openstack-ansible-ops/tree/osquery/tests/functional.yml#n29 | 15:50 |
| chandan_kumar | ? | 15:50 |
| chandan_kumar | panda: It is also used by ceph-ansible also | 15:50 |
| chandan_kumar | config_template -> https://github.com/openstack/ansible-config_template | 15:51 |
| chandan_kumar | weshay: marios for neutron tempest tests except two tests all passing http://logs.openstack.org/41/631441/2/check/tripleo-ci-centos-7-scenario001-standalone/dbea82c/logs/tempest.html.gz | 15:55 |
| chandan_kumar | weshay: marios we need some one from neutron team to make them passing | 15:55 |
| *** jaosorior has quit IRC | 15:56 | |
| panda | chandan_kumar: runtime ? | 15:56 |
| chandan_kumar | panda: yes, it is needed during run time | 15:57 |
| panda | ska: a single OOO system you mean at least an undercloud and an all-in-one controller ? | 15:57 |
| weshay | sshnaidm, ssbarnea|rover https://logs.rdoproject.org/52/631152/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/6e4f70f/job-output.txt.gz | 15:58 |
| chandan_kumar | panda: I tried to mess with ansible.cfg here https://review.openstack.org/#/c/628421/ but it lead to os_Tempest not found | 15:58 |
| panda | ska: you can try the oooq installing TripleO via libvirt in two VMs | 15:58 |
| panda | ska: I don't know what you're really trying to test though | 15:59 |
| chandan_kumar | panda: can you show me a nest ansible playbook example in tqe then I can implement it | 15:59 |
| chandan_kumar | the above same thing which is used in OSA | 16:00 |
| rlandy | quiquell: with https://review.rdoproject.org/r/#/c/18121/71/tasks/libvirt/main.yaml, we don;t needlibvirt from quickstart at all?? | 16:00 |
| * rlandy is very confused | 16:00 | |
| rlandy | - name: Setup libvirt nodes | 16:01 |
| *** hamzy_ has quit IRC | 16:01 | |
| panda | chandan_kumar: mmhh, not sure I'm following. You need thos plugins available to the tempest role ? | 16:02 |
| chandan_kumar | panda: yes | 16:03 |
| *** brault has joined #oooq | 16:04 | |
| *** skramaja has quit IRC | 16:04 | |
| panda | chandan_kumar: if we are calling a role we are not calling a nested ansible, the role will use the main ansible config | 16:04 |
| *** dsneddon has joined #oooq | 16:05 | |
| panda | chandan_kumar: where are the action plugins defined ? | 16:05 |
| chandan_kumar | panda: I have added this change https://review.openstack.org/#/c/631214/ | 16:05 |
| chandan_kumar | panda: https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tasks/tempest_post_install.yml#L31 here | 16:06 |
| *** quiquell is now known as quiquell|off | 16:06 | |
| chandan_kumar | panda: on ceph-ansible side they just copied this file https://github.com/ceph/ceph-ansible/blob/master/plugins/actions/config_template.py | 16:07 |
| chandan_kumar | panda: which I donot want to do that | 16:07 |
| chandan_kumar | panda: I wanted something cleaner and usable | 16:07 |
| panda | chandan_kumar: how is the role installed ? | 16:08 |
| quiquell|off | rlandy: The libvirt/main.yaml calls tq libvirt roles and do the nodepool setup | 16:08 |
| *** brault has quit IRC | 16:08 | |
| quiquell|off | rlandy: you can take a look at the tripleo-ci-fedora-28-libvirt | 16:08 |
| quiquell|off | rlandy: Try to execute it with a role | 16:08 |
| quiquell|off | rlandy: you can see the libvirt jobs here https://review.rdoproject.org/r/#/c/18121/ | 16:09 |
| chandan_kumar | panda: on OSA side they use nested ansible to do that http://git.openstack.org/cgit/openstack/openstack-ansible-ops/tree/osquery/tests/functional.yml#n29 | 16:09 |
| chandan_kumar | but I am not getting How an enduser can use it locally | 16:10 |
| panda | chandan_kumar: I mean how is the role installed so it can be used with quickstart. You may just need to specify a different path for the plugins | 16:12 |
| rlandy | quiquell|off: I want to change the defaults | 16:12 |
| rlandy | from a workflow perspective | 16:12 |
| rlandy | trying to figure out where that would go - that's all | 16:13 |
| quiquell|off | rlandy: ack, commenting at your review, play a little with the role first trying the three types of nodepool_providers | 16:13 |
| rlandy | the entry points here are not clear | 16:13 |
| rlandy | again - thinking of the user | 16:13 |
| chandan_kumar | panda: for changing the path I have added this https://review.openstack.org/#/c/628421/9/ansible.cfg but it is not picked here http://logs.openstack.org/00/627500/50/check/tripleo-ci-centos-7-standalone-os-tempest/94ee319/job-output.txt.gz#_2019-01-17_14_34_41_180286 | 16:13 |
| chandan_kumar | panda: I am not sure I am doing something wrong here https://review.openstack.org/628415 and https://review.openstack.org/627500 | 16:14 |
| sshnaidm | ssbarnea|rover, weshay https://review.rdoproject.org/r/18386 | 16:15 |
| quiquell|off | rlandy: commented https://review.openstack.org/#/c/631067 | 16:16 |
| quiquell|off | rlandy, sshnaidm, weshay, team: please review/merge https://review.rdoproject.org/r/#/c/18352/ and https://review.rdoproject.org/r/#/c/18121/ | 16:17 |
| panda | chandan_kumar: 628415 looks fine | 16:17 |
| quiquell|off | so we have the fix for reproducer CI and libvirt in place | 16:17 |
| quiquell|off | panda: ^ | 16:17 |
| quiquell|off | Drop now read you tomorrow people | 16:18 |
| quiquell|off | marios: We can talk tomorrow about repro if you want to help | 16:18 |
| panda | quiquell|off: marios include me | 16:18 |
| *** dsneddon has quit IRC | 16:19 | |
| panda | chandan_kumar: for 628421 I thnk the path may end up being in usr/local/share. | 16:19 |
| panda | chandan_kumar: but I see the problem now, the role is not even installed correctly | 16:19 |
| marios | quiquell|off: yes | 16:19 |
| chandan_kumar | panda: Oh, can you comment on the review I can fix it. thanks :-) | 16:21 |
| *** brault has joined #oooq | 16:24 | |
| panda | chandan_kumar: yep, getting the right steps for you | 16:24 |
| *** bogdando has quit IRC | 16:25 | |
| *** hamzy_ has joined #oooq | 16:26 | |
| *** brault has quit IRC | 16:29 | |
| panda | chandan_kumar: your test doesn't depend on 628421 | 16:30 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/560445, stable/queens: (1 more message) | 16:30 |
| panda | chandan_kumar: the log you showed me, it's from a job that doesn't depend on the quickstart-extras-requirements.txt change | 16:31 |
| panda | chandan_kumar: so os_tempest does not get installed | 16:31 |
| panda | chandan_kumar: because it's not in the requirements | 16:31 |
| chandan_kumar | oh no, | 16:32 |
| rlandy | sshnaidm: maybe you understand this - I am also struggling to understand the diff between host and libvirt options | 16:33 |
| rlandy | is host only a CI thing? | 16:33 |
| sshnaidm | rlandy, host - only for CI, yes | 16:34 |
| rlandy | ok - commenting on that | 16:34 |
| sshnaidm | rlandy, libvirt - will be for libvirt reproducer | 16:34 |
| chandan_kumar | panda: fixed, may be I was trying 4 depends on some time today that's why missed | 16:35 |
| chandan_kumar | panda++ | 16:36 |
| hubbot1 | chandan_kumar: panda's karma is now 12 | 16:36 |
| panda | hubbot1: where's costello ? | 16:38 |
| hubbot1 | panda: Error: "where's" is not a valid command. | 16:38 |
| *** jfrancoa has quit IRC | 16:41 | |
| ska | panda: We're testing our monitoring system called Zenoss, we typically need read-only access to the API to do our testing, but sometimes we need to configure features for test requirements etc. . | 16:42 |
| chandan_kumar | hubbot1: source | 16:43 |
| hubbot1 | chandan_kumar: My source is at https://github.com/ProgVal/Limnoria | 16:43 |
| *** derekh has joined #oooq | 16:44 | |
| ska | panda: tyically we are testing functional requirement, not performance. when we need to test performance we often create some simulators: We don't have the memory/cpu to test say 10000 vms for example. | 16:44 |
| ska | panda: so oooq is not a stand-alone Triple0? | 16:45 |
| rlandy | sshnaidm: have you tried the libvirt workflow outside of CI? | 16:45 |
| sshnaidm | rlandy, no | 16:45 |
| rlandy | yep | 16:45 |
| sshnaidm | rlandy, not sure it's ready yet | 16:46 |
| rlandy | the real concern is that this work has become very CI focused and forgotten about the end user | 16:46 |
| panda | ska: oooq drives and tests an installation of openstack though TripleO | 16:47 |
| panda | ska: it can be used to instruct TripleO to install a standalone node | 16:48 |
| panda | ska: but not probably the standalone you mean. THe standalone topology is not for production, it's more to test containerized services that are used by TripleO | 16:48 |
| sshnaidm | rlandy, I see a few patches in oooq repo too about libvirt | 16:49 |
| sshnaidm | rlandy, so I suppose it should work | 16:49 |
| rlandy | sshnaidm: will be working on testing that out today | 16:49 |
| sshnaidm | rlandy, but need to check if it doesn't break all other libvirt cases we have | 16:49 |
| sshnaidm | rlandy, we reuse this libvirt part in a thousand use cases.. :/ | 16:50 |
| *** ccamacho has quit IRC | 16:54 | |
| *** dsneddon has joined #oooq | 16:55 | |
| weshay | so which nodepool is this referring to? - libvirt: Start up a pair of libvirt nodes at install and connects nodepool | 17:01 |
| weshay | the container running on a host? | 17:01 |
| *** dsneddon has quit IRC | 17:01 | |
| ska | panda: does anyone have test systems for public use? | 17:05 |
| ssbarnea|rover | out of curiosity, openstack cli is not able to load config using just OS_CLOUD var alone and the clouds.yml file? or I just missed to configure the clouds file correctly. | 17:10 |
| weshay | ssbarnea|rover, the cix call on monday at 2pm utc can you go to that for me? | 17:10 |
| *** dsneddon has joined #oooq | 17:11 | |
| ssbarnea|rover | weshay: i think so | 17:11 |
| weshay | ssbarnea|rover, thanks | 17:11 |
| weshay | ska, what is your email addr? | 17:11 |
| sshnaidm | ssbarnea|rover, did you check it out? https://review.rdoproject.org/r/#/c/18386/ are we ready to merge? | 17:11 |
| panda | ska: not that I know of, but I still don't understand what part of zenoss you need to test with what part of TripleO. You are installing Zenoss in openstack ? use TripleO to install Zenoss to monitor Openstack ? | 17:12 |
| sshnaidm | ssbarnea|rover, clouds.yaml, not yml | 17:12 |
| sshnaidm | ssbarnea|rover, and it should pick it up if in ~/.config/openstack or /etc/openstack | 17:12 |
| ssbarnea|rover | sshnaidm: is not that is not loaded, but i got a Expecting to find domain in user. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) | 17:13 |
| sshnaidm | ssbarnea|rover, check if it's v2 or v3 | 17:13 |
| ssbarnea|rover | sshnaidm: i have progres... i will narrow it down. thanks. | 17:14 |
| ssbarnea|rover | having url/email/pass used to be enough in the past... | 17:15 |
| ssbarnea|rover | sshnaidm: fixed, i had to add project_name, project_domain_name and user_domain_name | 17:19 |
| ssbarnea|rover | no need to mention api version | 17:19 |
| weshay | sshnaidm, so re: /tmp/delorean_logs/home/{{ undercloud_user }}/DLRN/data/repos/*/build.log | 17:21 |
| weshay | that can be /tmp/*/DLRN/data/repos/*/build.log ? | 17:22 |
| sshnaidm | weshay, yes | 17:22 |
| weshay | sshnaidm++ | 17:22 |
| hubbot1 | weshay: sshnaidm's karma is now 11 | 17:22 |
| weshay | thanks | 17:22 |
| sshnaidm | ssbarnea|rover, it's only for v3, that's why I asked what version | 17:23 |
| sshnaidm | ssbarnea|rover, if using v2 you don't need all these | 17:23 |
| sshnaidm | ssbarnea|rover, did you check out a patch above? | 17:23 |
| *** dsneddon has quit IRC | 17:24 | |
| *** bogdando has joined #oooq | 17:27 | |
| *** bogdando has quit IRC | 17:27 | |
| weshay | ssbarnea|rover, fyi https://review.rdoproject.org/r/#/c/18388/ | 17:32 |
| *** panda is now known as panda|off | 17:34 | |
| ssbarnea|rover | sshnaidm: can i update it? i have a better approach. | 17:35 |
| sshnaidm | ssbarnea|rover, you can comment, it's free | 17:35 |
| ssbarnea|rover | give me few minutes... | 17:35 |
| *** dtantsur is now known as dtantsur|afk | 17:37 | |
| *** kopecmartin is now known as kopecmartin|off | 17:37 | |
| *** udesale has quit IRC | 17:41 | |
| sshnaidm | ssbarnea|rover, replied, please revote | 17:42 |
| *** jpena is now known as jpena|off | 17:43 | |
| *** trown is now known as trown|lunch | 17:49 | |
| *** dsneddon has joined #oooq | 17:53 | |
| *** ccamacho has joined #oooq | 17:54 | |
| ssbarnea|rover | sshnaidm: commented, shortly: openstack server list -f value -c Name --name {{ idnum }} | 17:54 |
| ssbarnea|rover | sshnaidm: avoid grep and the need to ignore_errors. | 17:55 |
| sshnaidm | ssbarnea|rover, you're wrong in your patch.. | 17:56 |
| *** dsneddon has quit IRC | 17:57 | |
| *** derekh has quit IRC | 17:58 | |
| sshnaidm | ssbarnea|rover, hmm.. actually --name always returns 0, so maybe it's ok | 18:01 |
| ssbarnea|rover | sshnaidm: yep, and is a regex. i used in the past. works quite nice. it wasn't always there, but is old enough for us to use. | 18:02 |
| ssbarnea|rover | there are other workaround like using awk/sed to avoid grep exit code 1. | 18:02 |
| sshnaidm | ssbarnea|rover, ok, but how are credentials related? | 18:02 |
| ssbarnea|rover | sshnaidm: i didn't express correcly: i wanted to say that if listing itself fails, we missing to spot this error. (could credentials, 400-ish errors,...) | 18:03 |
| ssbarnea|rover | we are talking with a server, anything could happen. | 18:04 |
| sshnaidm | ssbarnea|rover, then it will fail later, no way to pass | 18:04 |
| ssbarnea|rover | sshnaidm: yep, but i prefer to not to miss errors with this, i can give you one example that happened to me few times | 18:05 |
| ssbarnea|rover | the command line become invalid due to some var being expanded incorrectly or something like this. nobody observed that for weeks if not months, because of ignore_errors which ignores even if ansible module crashes. | 18:05 |
| ssbarnea|rover | this is why i try to avoid it, whenever is possible. | 18:06 |
| ssbarnea|rover | i am more likely to prefer adding a "|| true" than using this ignore_errors. | 18:07 |
| sshnaidm | ssbarnea|rover, if nobody observed this it means nobody needed this | 18:10 |
| *** dsneddon has joined #oooq | 18:10 | |
| ssbarnea|rover | sshnaidm: hehe. not sure, we didn't oberserve that the cleanup was not working until the issue started to affect new builds. this does not really mean we didn't need the cleanup. many hidden issues hit you later, when you don't expect. | 18:12 |
| ssbarnea|rover | sshnaidm: i am working now to fix the script, i will raise a CR before tomorrow. | 18:13 |
| sshnaidm | ssbarnea|rover, that's good in zuul that you don't care about cleanup | 18:13 |
| ssbarnea|rover | yep | 18:13 |
| *** hamzy_ has quit IRC | 18:14 | |
| ssbarnea|rover | i have to say that I prefer to let CI to take care of cleanup, i never fully trust a job to be able to clean after itself. | 18:14 |
| *** hamzy has joined #oooq | 18:16 | |
| *** ykarel|away has quit IRC | 18:27 | |
| sshnaidm | weshay, this can possible break OVB: https://review.rdoproject.org/r/#/c/18390/ | 18:29 |
| weshay | sshnaidm, ok.. shall we put up a change window? | 18:30 |
| sshnaidm | weshay, window..? | 18:30 |
| weshay | sunday would be fine by me w/ self merge if needed | 18:30 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/560445, stable/queens: (1 more message) | 18:30 |
| weshay | https://www.google.com/search?q=define+change+window&oq=define+change+window&aqs=chrome..69i57.2806j1j1&sourceid=chrome&ie=UTF-8 | 18:30 |
| weshay | :) | 18:30 |
| *** ccamacho has quit IRC | 18:31 | |
| weshay | a time where we inform folks of a possible issue and when the potential damage is minimal | 18:31 |
| weshay | know what I mean vern? | 18:31 |
| weshay | sorry... eighties kid here.. https://goo.gl/images/B21d34 | 18:32 |
| Tengu | oh. wow. | 18:32 |
| Tengu | that's.... wow. | 18:32 |
| Tengu | Ernst. | 18:32 |
| Tengu | (at least in French, so I guess his name is Vern in English :D) | 18:33 |
| *** chem has quit IRC | 18:33 | |
| sshnaidm | weshay, let's do it in Sun | 18:33 |
| sshnaidm | weshay, will not affect many people | 18:33 |
| weshay | sshnaidm, sounds good time | 18:35 |
| sshnaidm | weshay, if merging this, worth to keep eye too: https://review.rdoproject.org/r/#/c/18386/ | 18:36 |
| sshnaidm | weshay, but it shouldn't affect in theory | 18:37 |
| weshay | sshnaidm, ssbarnea|rover more failed stacks, taking care of it | 18:48 |
| *** jbadiapa has quit IRC | 18:48 | |
| sshnaidm | weshay, so I'm merging https://review.rdoproject.org/r/#/c/18386 ? | 18:49 |
| weshay | sshnaidm, yes please | 18:50 |
| *** brault has joined #oooq | 18:52 | |
| sshnaidm | weshay, merged | 18:54 |
| ssbarnea|rover | weshay: sshnaidm: do you know the reason behind "don't overwhelm the tenant with mass delete" ? | 19:00 |
| sshnaidm | ssbarnea|rover, no, what is that? | 19:00 |
| ssbarnea|rover | sshnaidm: a comment in the code and a sleep of 20s between delete. i was wondering... | 19:01 |
| sshnaidm | ssbarnea|rover, maybe rlandy knows ^ | 19:02 |
| ssbarnea|rover | my personal approach was to run loop: delete &; wait :D | 19:02 |
| sshnaidm | ssbarnea|rover, it could cause more problems, than it solves | 19:03 |
| *** trown|lunch is now known as trown | 19:03 | |
| *** jbadiapa has joined #oooq | 19:03 | |
| * rlandy looks | 19:04 | |
| rlandy | sshnaidm: the tenant used to fall over when we deleted too many stacks concurrently | 19:04 |
| sshnaidm | ssbarnea|rover, ^^ | 19:05 |
| ssbarnea|rover | rlandy: is this still a limitation? it does not feel right to nurture the cloud so much. | 19:07 |
| weshay | ssbarnea|rover, in the script? | 19:09 |
| rlandy | ssbarnea|rover: that is what we needed when we wrote it | 19:09 |
| weshay | ssbarnea|rover, ya.. if you run heat stack commands too quickly.. it gets all confused | 19:09 |
| rlandy | feel free to try remove it - and see what happens :) | 19:09 |
| weshay | ssbarnea|rover, you have a patch for the script? | 19:13 |
| ssbarnea|rover | weshay: testing it right now, running it several times, minor tunnings. | 19:14 |
| ssbarnea|rover | what was the original wrong condition? | 19:14 |
| weshay | k.. we have several in delete_failed and create_failed | 19:14 |
| weshay | so it's a good time | 19:14 |
| weshay | if we are unable to remove delete_failed then we have to contact rhos-ops | 19:15 |
| ssbarnea|rover | nevermind, i seen it. only looking for create_failed instead of both. | 19:16 |
| rlandy | ssbarnea|rover: the create_failed should be able to be deleted quickly | 19:18 |
| rlandy | even multiple at a time | 19:18 |
| weshay | ci.centos sucks | 19:18 |
| rlandy | failed? | 19:19 |
| weshay | rlandy, got the not enough hosts error.. meaning most likely we got a bad host | 19:21 |
| weshay | on run three | 19:21 |
| weshay | rlandy, honestly.. we could switch that job to singlenode | 19:21 |
| rlandy | shoot | 19:22 |
| weshay | it has worked https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-extras-gate-master-delorean-full-featureset052-2265/console.txt.gz | 19:22 |
| ssbarnea|rover | perfect time to test the script changes :) .... on production. | 19:23 |
| rlandy | worth doing one more run? | 19:23 |
| ssbarnea|rover | i am doing multiple runs, it will take some time to finish. | 19:23 |
| weshay | oh I see why it's failing | 19:23 |
| weshay | pip errors :( | 19:23 |
| weshay | friggin JENKINS!!! | 19:24 |
| rlandy | welcome to my world | 19:24 |
| *** jtomasek has quit IRC | 19:25 | |
| *** rfolco has joined #oooq | 19:28 | |
| *** holser_ has quit IRC | 19:38 | |
| weshay | ssbarnea|rover, the clean up script was on what ip? 38.145.34.41 | 19:46 |
| weshay | ? | 19:46 |
| *** rfolco has quit IRC | 19:49 | |
| ssbarnea|rover | i don't know, i called my local copy. | 19:51 |
| ssbarnea|rover | rlandy: weshay: regarding pip/venv issues with jenkins, i made a fix few days ago: https://review.openstack.org/#/c/630300/ | 19:52 |
| ssbarnea|rover | if the problem you encountered was "IOError: [Errno 26] Text file busy" | 19:53 |
| weshay | ssbarnea|rover, was not not picking up all the files from the repos | 19:53 |
| ssbarnea|rover | ahh, different issue. | 19:54 |
| weshay | 2019-01-17 19:31:00.258 | + VERBOSITY=vv | 19:54 |
| weshay | 2019-01-17 19:31:00.258 | + '[' '!' -f /home/jenkins/workspace/tripleo-quickstart-extras-gate-master-delorean-full-featureset052/playbooks/quickstart-extras-standalone.yml ']' | 19:54 |
| weshay | 2019-01-17 19:31:00.258 | + printf '\n !! execute quickstart.sh --clean to ensure the dependencies are installed !!' | 19:54 |
| weshay | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-extras-gate-master-delorean-full-featureset052-2290/console.txt.gz | 19:54 |
| weshay | https://ci.centos.org/view/rdo/view/tripleo-gate/job/tripleo-quickstart-extras-gate-master-delorean-full-featureset052/2290/ | 19:54 |
| ssbarnea|rover | i find the message a bit unfortunate, --clean to install.. not really the most obvious choice. also the condition for displaying this message seems bit weird. If I give wrong playbook as param, i endup with really weird message. maybe we can improve that a little bit. | 20:00 |
| weshay | ssbarnea|rover, it's fucking jenkins and not having a clean workspace afaict | 20:01 |
| ssbarnea|rover | weshay: you can enforce cleaning of workspace with jjb | 20:03 |
| ssbarnea|rover | weshay: something is weird here, i know that jenkins is full of problems but I am not sure if its fault here. do we run jobs in parallel? | 20:06 |
| ssbarnea|rover | $W/playbooks/quickstart-extras-standalone.yml is missing, so who is creating this file? | 20:09 |
| *** sanjayu_ has quit IRC | 20:18 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (2 more messages) | 20:31 |
| *** irclogbot_0 has quit IRC | 20:41 | |
| weshay | ssbarnea|rover, ya we do | 20:46 |
| weshay | ssbarnea|rover, do you need access to the tebroker box where the script runs, I'm in | 20:47 |
| ssbarnea|rover | i do not need, but I tried to read the cred doc and assure that what is there is is usable. | 20:47 |
| ssbarnea|rover | probably you seen that I did some "css" on it. | 20:48 |
| *** irclogbot_0 has joined #oooq | 20:51 | |
| weshay | ssbarnea|rover, do anything on the promoter server? | 21:02 |
| weshay | or anyone? | 21:02 |
| rlandy | remove the cache | 21:14 |
| rlandy | ^^ for jenkins | 21:14 |
| weshay | rlandy, jeeeez... | 21:23 |
| weshay | dlrn_api craziness | 21:23 |
| weshay | rlandy, check this out https://bugs.launchpad.net/tripleo/+bug/1812261 | 21:24 |
| openstack | Launchpad bug 1812261 in tripleo "invalid dlrn_hash created for tripleo current-tripleo" [Critical,Triaged] | 21:24 |
| rlandy | oh gosh | 21:26 |
| rlandy | weshay: did you manually promote/update | 21:27 |
| weshay | rlandy, I couldn't get the command to work, 404's | 21:27 |
| weshay | david simard had to do it | 21:27 |
| rlandy | what did he do that we could not? | 21:28 |
| rlandy | the api was down yesterday and then it was not? | 21:28 |
| rlandy | I am very confused as to what is really happening here | 21:29 |
| * rlandy reads bug again | 21:29 | |
| rlandy | weshay: I am trying to understand the bug | 21:30 |
| rlandy | if the master reported has is pointing to rocky we have a bug - not the dlrn api | 21:30 |
| ssbarnea|rover | weshay: oops, i think i made a mistake few minutes ago.... i called the example from the DLRN_API document.... | 21:34 |
| weshay | ssbarnea|rover, got it fixed | 21:38 |
| weshay | just keeping in the loop | 21:38 |
| weshay | ignore my pings when ur offline :) | 21:38 |
| ssbarnea|rover | weshay: ohh, thanks! and sorry. | 21:38 |
| *** hamzy has quit IRC | 21:38 | |
| ssbarnea|rover | i added a o --do-not-copypaste-me param, to the example, hopefully enough to protect another accident. | 21:40 |
| weshay | ugh | 21:42 |
| weshay | ssbarnea|rover, arxcruz|ruck fyi.. rlandy I've shutdown the promoter server until dlrn is fixed | 21:42 |
| weshay | rlandy, ok.. | 21:46 |
| * weshay headed to blue | 21:46 | |
| rlandy | ok | 21:46 |
| *** trown is now known as trown|outtypewww | 22:05 | |
| weshay | rlandy, HA.. of course rdo phase 1 is passing now and the promoter is off | 22:15 |
| rlandy | weshay: at least it's passing | 22:16 |
| * rlandy is grateful for all good news | 22:16 | |
| weshay | very zen | 22:18 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario003-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (2 more messages) | 22:31 |
| *** agopi has quit IRC | 22:54 | |
| *** vinaykns has quit IRC | 22:59 | |
| *** agopi has joined #oooq | 23:23 | |
| *** dsneddon has quit IRC | 23:45 | |
| *** dsneddon has joined #oooq | 23:48 | |
| *** tosky has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!