| *** dsneddon has joined #oooq | 00:18 | |
| *** dsneddon has quit IRC | 00:22 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages) | 00:47 |
|---|---|---|
| *** dsneddon has joined #oooq | 00:51 | |
| *** dsneddon has quit IRC | 00:55 | |
| *** dsneddon has joined #oooq | 01:24 | |
| *** dsneddon has quit IRC | 01:29 | |
| *** dsneddon has joined #oooq | 02:00 | |
| *** dsneddon has quit IRC | 02:04 | |
| *** jtomasek has joined #oooq | 02:05 | |
| *** jtomasek has quit IRC | 02:10 | |
| *** dsneddon has joined #oooq | 02:33 | |
| *** dsneddon has quit IRC | 02:45 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages) | 02:47 |
| *** jtomasek has joined #oooq | 02:47 | |
| *** jtomasek has quit IRC | 02:53 | |
| *** skramaja has joined #oooq | 02:53 | |
| *** dsneddon has joined #oooq | 03:13 | |
| *** dsneddon has quit IRC | 03:26 | |
| *** dsneddon has joined #oooq | 03:30 | |
| *** ykarel|away has joined #oooq | 04:21 | |
| *** ykarel|away is now known as ykarel | 04:35 | |
| *** ykarel has quit IRC | 04:42 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages) | 04:47 |
| *** ykarel has joined #oooq | 04:58 | |
| *** ratailor has joined #oooq | 05:27 | |
| quiquell|off | Good morning | 06:20 |
| *** quiquell|off is now known as quiquell | 06:20 | |
| *** sshnaidm|off is now known as sshnaidm | 06:30 | |
| sshnaidm | hi | 06:30 |
| quiquell | sshnaidm: o/ | 06:33 |
| quiquell | sshnaidm: Abou vm_password, how dangerous it is ? | 06:34 |
| quiquell | sshnaidm: Is a public server at RDO so everyone can access right ? | 06:34 |
| quiquell | sshnaidm: Maybe we can random generate it and show it to user first time | 06:34 |
| quiquell | sshnaidm: here https://github.com/istio/istio/pull/11129 | 06:35 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages) | 06:47 |
| *** jbadiapa has joined #oooq | 06:48 | |
| sshnaidm | quiquell, I left it only for console, ssh_pwauth is false and nobody can access by ssh with password | 06:52 |
| sshnaidm | quiquell, only people that can log in to rdo cloud web interface and open console there | 06:53 |
| quiquell | sshnaidm: Ahh that's right | 06:53 |
| sshnaidm | quiquell, I think it's fine, could be useful when something wrong with connections and you can't ssh | 06:53 |
| quiquell | sshnaidm: no ip access with password since we don't have ssh password thing ok | 06:53 |
| sshnaidm | quiquell, yeah, ssh only by key | 06:54 |
| quiquell | quiquell: wrong link | 06:54 |
| quiquell | sshnaidm: random generate it is needed ? | 06:54 |
| sshnaidm | quiquell, maybe, not sure right now | 06:55 |
| quiquell | sshnaidm: docs talk about something like mkpasswd --method=SHA-512 --rounds=4096 | 06:55 |
| sshnaidm | quiquell, don't have ansible some password module? | 06:56 |
| quiquell | looks like https://docs.ansible.com/ansible/latest/plugins/lookup/password.html | 06:56 |
| quiquell | very difficult module name | 06:56 |
| quiquell | sshnaidm: btw, libvirt is very unreliable :-/ | 06:58 |
| quiquell | sshnaidm: At list the implementation we have of calling libvirt | 06:59 |
| sshnaidm | quiquell, what's failing? | 06:59 |
| quiquell | sshnaidm: I have see like different unrelated stuff at CI with it | 06:59 |
| quiquell | sshnaidm: Sometimes is centos7 virt-resize | 06:59 |
| quiquell | sshnaidm: others is like in your review node does not go up after reboot | 06:59 |
| quiquell | sshnaidm: and other at centos7 there is no IP at undercloud | 07:00 |
| sshnaidm | quiquell, maybe we can try twice on some fragile tasks? | 07:01 |
| *** jtomasek has joined #oooq | 07:01 | |
| sshnaidm | or it won't help much | 07:01 |
| quiquell | sshnaidm: no kidding ? | 07:01 |
| quiquell | sshnaidm: Maybe we can reimplement the libvirt roles from tq | 07:01 |
| sshnaidm | quiquell, if it depends on weather.. | 07:01 |
| sshnaidm | quiquell, I'm not sure it's roles fault, even I'd like to redo them | 07:02 |
| quiquell | sshnaidm: But what I have see the most is virt-resize on centos7 :-/ | 07:02 |
| sshnaidm | quiquell, I think we need to remove this stuff with environment variables, that we see broke it | 07:02 |
| quiquell | sshnaidm: have same feelings, don't have prove of them being wrong, but want to reimplent | 07:02 |
| quiquell | sshnaidm: what do you mean ? | 07:03 |
| sshnaidm | quiquell, I'd say that libvirt itself worth to be rewritten.. | 07:03 |
| quiquell | sshnaidm: What can be the alternatives of libvirt ? | 07:03 |
| quiquell | sshnaidm: let's fix linting at https://review.rdoproject.org/r/#/c/18515 and merge it | 07:03 |
| sshnaidm | quiquell, I meant this shit: https://github.com/openstack/tripleo-quickstart/blob/364c4ecce906f34fcc3eed7e16d2b79b25497509/playbooks/libvirt-nodepool.yml#L22-L26 | 07:04 |
| quiquell | sshnaidm: repro don't use the playbook | 07:05 |
| arxcruz|ruck | quiquell: hey, we are having a problem with fedora 28 job | 07:05 |
| quiquell | sshnaidm: We are not affected by that | 07:05 |
| sshnaidm | quiquell, ah, ok | 07:05 |
| quiquell | arxcruz|ruck: don't know | 07:05 |
| quiquell | sshnaidm: we reuse roles, it's the way we should reuse stuff at ansible I think | 07:05 |
| quiquell | arxcruz|ruck: do you see something there ? | 07:05 |
| arxcruz|ruck | quiquell: just a sec, verifying again, now it seems it's no longer failing on that part | 07:06 |
| quiquell | sshnaidm: virt-resize failure http://logs.rdoproject.org/75/18475/9/check/tripleo-ci-reproducer-centos-7-libvirt/7d5ac5e/job-output.txt.gz | 07:07 |
| quiquell | sshnaidm: It's stuck there | 07:07 |
| *** chandankumar has quit IRC | 07:07 | |
| sshnaidm | quiquell, maybe let's do "-vx" in task and redirect logs to file in home dir so that we'll collect it | 07:09 |
| sshnaidm | quiquell, at least to see what's problem | 07:09 |
| sshnaidm | quiquell, although I tend to think it's hardware problem | 07:10 |
| sshnaidm | quiquell, maybe let's publish disk that don't need to be resized :) | 07:10 |
| quiquell | sshnaidm: Do we need to resize ? | 07:10 |
| sshnaidm | quiquell, I think so, it's 8gb disks | 07:11 |
| quiquell | sshnaidm: We are doing a lot of not needed stuff there, like for example starting up two nodes for a one node run | 07:11 |
| *** jfrancoa has joined #oooq | 07:11 | |
| sshnaidm | quiquell, oh, yeah, I didn't like this too | 07:11 |
| quiquell | sshnaidm: It's like complicated to bypass that | 07:11 |
| sshnaidm | quiquell, well, with current role - yes | 07:12 |
| quiquell | sshnaidm: ok will dump info so we debug that | 07:12 |
| quiquell | sshnaidm: maybe we can use libvirt/setup/undercloud for standalone jobs instead of libvirt/setup/overcloud | 07:16 |
| quiquell | sshnaidm: it would be usable ? | 07:17 |
| *** panda|off is now known as panda | 07:20 | |
| quiquell | sshnaidm: damn since virt-resize is failing at timeout we cannot dump the logs | 07:21 |
| quiquell | sshnaidm: ahh we can | 07:21 |
| sshnaidm | quiquell, post playbook? | 07:21 |
| *** chandan_kumar has joined #oooq | 07:21 | |
| quiquell | sshnaidm: normal bash stuff will do | 07:21 |
| *** gkadam has joined #oooq | 07:22 | |
| *** kopecmartin|off is now known as kopecmartin | 07:27 | |
| quiquell | sshnaidm: to debug virt-resize https://review.rdoproject.org/r/18558 | 07:27 |
| quiquell | panda: o/ | 07:27 |
| sshnaidm | quiquell, great | 07:28 |
| panda | morningz | 07:29 |
| quiquell | panda: have you being able to run the reproducer ? | 07:30 |
| marios | sup folks | 07:33 |
| panda | quiquell: nope, couldn't get past the 500 internal server error, even after adding the keys. I'll try again in a couple hours, have something to do first | 07:33 |
| quiquell | panda: ack, let me know if you need help | 07:34 |
| quiquell | marios: you there ? | 07:34 |
| quiquell | marios: This task https://tree.taiga.io/project/tripleo-ci-board/task/613 depends on dryrun I think | 07:35 |
| quiquell | marios: If we do do the dryrun we just run reproducer CI for libvirt roles changes | 07:35 |
| quiquell | marios: If we do the dryrun we have to also run the reproducer CI on toci changes | 07:35 |
| quiquell | marios: Don't know if we want to add new review for that and keep this just for libvirt roles changes | 07:35 |
| quiquell | marios: what do you think | 07:36 |
| * quiquell feels the coffe kicking in | 07:36 | |
| marios | quiquell: o/ so dryrun would be the default | 07:36 |
| quiquell | marios: Should be | 07:37 |
| quiquell | marios: but we are near sprint end so maybe we can add repro CI for tripleo-ci project next sprint and keep this for libvirt | 07:37 |
| marios | quiquell: k, well add a note on the card and lets discuss some more on the phone either this afternoon or earlier if a need arises | 07:37 |
| marios | quiquell: i mean if it is depends on | 07:38 |
| quiquell | marios: ack | 07:38 |
| marios | quiquell: but same time is a bit late to be identifying issues like this (say the depends on merges today | 07:38 |
| marios | still very tight to fix this by wed | 07:38 |
| quiquell | marios: yep | 07:38 |
| marios | not your fault just saying (I know this is complex and changing all the time) | 07:38 |
| quiquell | marios: Let me know if you need help with dryrun | 07:39 |
| marios | quiquell: k gonna go see what happend there :) | 07:39 |
| marios | quiquell: on the review | 07:39 |
| marios | and i need to setup a beaker box | 07:39 |
| marios | for the libvirt | 07:39 |
| quiquell | marios: You can launch libvirt at RDO | 07:39 |
| marios | quiquell: (I mean https://review.rdoproject.org/r/18539 | 07:39 |
| quiquell | marios: Humm there is no rebase button for me :-/ | 07:40 |
| marios | quiquell: which means you have to use meld probably | 07:40 |
| quiquell | marios: nope there is "merge conflict" have to be something with project config | 07:41 |
| marios | quiquell: yeah thats what i mean | 07:41 |
| quiquell | marios: like "only owner can hit rebase" | 07:41 |
| marios | quiquell: there is merge conflict | 07:41 |
| marios | merld is conflict tool | 07:41 |
| marios | meld | 07:41 |
| quiquell | I know | 07:41 |
| marios | quiquell: but sometimes git can't do it | 07:41 |
| marios | quiquell: hence the conflict | 07:41 |
| marios | :D | 07:41 |
| marios | k | 07:41 |
| quiquell | marios: but gerrit show it to you | 07:41 |
| quiquell | marios: that's weird | 07:41 |
| marios | quiquell: ah i see | 07:41 |
| marios | quiquell: but even then | 07:41 |
| marios | quiquell: the button will likely fail | 07:41 |
| marios | quiquell: and you need to do it locally and push a version | 07:42 |
| quiquell | marios: thing is button is not there for me | 07:42 |
| *** jtomasek has quit IRC | 07:42 | |
| quiquell | marios: I mean does not appear | 07:42 |
| marios | quiquell: yeah i understood that | 07:42 |
| marios | quiquell: :) | 07:42 |
| quiquell | ack | 07:42 |
| marios | quiquell: ack | 07:42 |
| marios | rst | 07:42 |
| quiquell | marios: so mirror issues | 07:43 |
| quiquell | http://logs.rdoproject.org/39/18539/2/check/tripleo-ci-reproducer-fedora-28-libvirt/0355fd4/tripleo-ci-reproducer/logs/01/1001/1/check/tripleo-ci-centos-7-standalone-reprozuul-dryrun/66b1f16/job-output.txt.gz | 07:43 |
| *** skramaja has quit IRC | 07:43 | |
| quiquell | http://mirror.none.none.rdoproject.org/centos/7/os/x86_64/repodata/repomd.xml | 07:43 |
| quiquell | marios: you have to play with var "mirror_fqdn" | 07:43 |
| quiquell | marios: I think | 07:44 |
| marios | quiquell: thanks noted | 07:44 |
| marios | quiquell: i'll have a closer look in a sec | 07:44 |
| marios | quiquell: finish some reviews | 07:44 |
| *** rascasoft has joined #oooq | 07:49 | |
| *** chandan_kumar is now known as chandankumar | 08:02 | |
| *** jtomasek has joined #oooq | 08:02 | |
| *** ykarel is now known as ykarel|lunch | 08:20 | |
| *** ccamacho has joined #oooq | 08:25 | |
| arxcruz|ruck | ykarel|lunch: around ? | 08:25 |
| arxcruz|ruck | ykarel|lunch: everytime i try to run the reproducer on fedora, i'm getting this http://paste.openstack.org/show/744004/ | 08:26 |
| *** apetrich has joined #oooq | 08:27 | |
| ykarel|lunch | arxcruz|ruck, looks like you are using old repos | 08:31 |
| arxcruz|ruck | ykarel|lunch: that's what the reproducer gave to me | 08:31 |
| ykarel|lunch | arxcruz|ruck, link | 08:31 |
| marios | quiquell: ykarel|lunch panda any idea why skipped ? https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic&job_name=periodic-tripleo-ci-centos-7-scenario001-standalone-master | 08:36 |
| marios | same for all of them afaics scen 4/3/2 | 08:36 |
| ykarel|lunch | marios, dependency failed | 08:36 |
| ykarel|lunch | container-build job | 08:36 |
| marios | ykarel|lunch: ah | 08:36 |
| marios | ykarel|lunch: ack thanks | 08:36 |
| quiquell | marios: yep | 08:36 |
| quiquell | marios: is skipped if the depedant job fails | 08:36 |
| quiquell | marios: It's how zuul marks it | 08:36 |
| ykarel|lunch | marios, quiquell fix: https://review.openstack.org/#/c/633434/ | 08:36 |
| marios | quiquell: ykarel|lunch thanks was worried i missed something | 08:36 |
| ykarel|lunch | let's get it merged | 08:36 |
| marios | cos we merged it late friday (for me anyway ) | 08:36 |
| marios | thanks ykarel|lunch checking | 08:37 |
| quiquell | ykarel|lunch: We don't need this thing to be backward compatible ? | 08:38 |
| ykarel|lunch | quiquell, nope, | 08:39 |
| ykarel|lunch | bacward compatible was required until promotion | 08:39 |
| quiquell | ykarel|lunch: ok, I am going to workflow that | 08:39 |
| marios | ykarel|lunch: its a revert of https://review.openstack.org/#/c/633014/2/container-images/tripleo_kolla_template_overrides.j2 | 08:39 |
| ykarel|lunch | ack | 08:39 |
| marios | er quiquell i meant | 08:39 |
| marios | quiquell: i just wf it | 08:39 |
| quiquell | marios: ack | 08:39 |
| *** tosky has joined #oooq | 08:41 | |
| arxcruz|ruck | ykarel|lunch: http://logs.openstack.org/39/633039/2/check/tripleo-ci-fedora-28-standalone/6455149/logs/reproducer-quickstart.sh | 08:44 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages) | 08:47 |
| *** ykarel|lunch is now known as ykarel | 08:51 | |
| ykarel | arxcruz|ruck, looking | 08:51 |
| arxcruz|ruck | ykarel: thanks | 08:51 |
| *** jpena|off is now known as jpena | 08:54 | |
| ykarel | arxcruz|ruck, at which step it's failing? | 08:58 |
| ykarel | from fedora python2- packages are removed | 08:58 |
| arxcruz|ruck | ykarel: install_packages.sh | 08:59 |
| ykarel | seems some issue with reproducer | 08:59 |
| arxcruz|ruck | the first step basically | 08:59 |
| ykarel | arxcruz|ruck, it's trying to use centos repo | 08:59 |
| sshnaidm | quiquell, docker pull fails sometimes, need to use docker proxy.. | 08:59 |
| quiquell | sshnaidm: yep Have just commented in the taiga user story of dryrun | 08:59 |
| arxcruz|ruck | ykarel: i blame quiquell | 08:59 |
| quiquell | arxcruz|ruck: why why !!! | 09:00 |
| arxcruz|ruck | quiquell: in the reproducer, fedora 28 are trying to use centos repo | 09:00 |
| quiquell | arxcruz|ruck: current reproducer you mean ? | 09:00 |
| arxcruz|ruck | quiquell: yes | 09:00 |
| quiquell | arxcruz|ruck: I think there is not much love of f28 for the repro | 09:01 |
| quiquell | arxcruz|ruck: sure we have miss some duplication at toci non jinja scripts | 09:01 |
| arxcruz|ruck | quiquell: is there a way to use the new reproducer already ? | 09:01 |
| quiquell | arxcruz|ruck: yep | 09:01 |
| quiquell | arxcruz|ruck: but you need to get some patience, as is not out of the box thing yet | 09:02 |
| quiquell | sshnaidm: can we merge the cloudinit thing ? or do you want to add more changes there ? | 09:02 |
| sshnaidm | quiquell, CI failed, I rerun it | 09:03 |
| quiquell | sshnaidm: I removed "state: present" and replaced it with ansible tags mechanism | 09:03 |
| quiquell | sshnaidm: I see you have add it again | 09:03 |
| quiquell | sshnaidm: Looks like a rebase issue | 09:04 |
| sshnaidm | quiquell, yeah, leftovers | 09:04 |
| quiquell | sshnaidm: ack, let's fix that and try to merge, with enough rechecks :-/ | 09:04 |
| *** bogdando has joined #oooq | 09:07 | |
| quiquell | arxcruz|ruck: Want to try the repro ? | 09:08 |
| arxcruz|ruck | quiquell: sure | 09:08 |
| quiquell | arxcruz|ruck: https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/README.md | 09:09 |
| quiquell | arxcruz|ruck: More or less is that | 09:09 |
| sshnaidm | chandankumar, http://logs.openstack.org/00/627500/74/check/tripleo-ci-centos-7-standalone-os-tempest/937e4eb/job-output.txt.gz#_2019-01-28_08_41_16_018007 | 09:09 |
| *** jaosorior has joined #oooq | 09:10 | |
| chandankumar | sshnaidm: fixing that | 09:11 |
| chandankumar | sshnaidm: thanks :-) Done | 09:12 |
| arxcruz|ruck | quiquell: so, just need to run that playbook ? | 09:16 |
| arxcruz|ruck | and done ? | 09:16 |
| quiquell | arxcruz|ruck: Do you have docker in the system ? | 09:17 |
| arxcruz|ruck | quiquell: yup | 09:17 |
| *** chem has joined #oooq | 09:17 | |
| quiquell | arxcruz|ruck: try to run the playbook and see if it works | 09:17 |
| quiquell | arxcruz|ruck: if nope then run pre.yaml playbook first | 09:17 |
| quiquell | arxcruz|ruck: do you have docker-ce or normal docker ? | 09:18 |
| sshnaidm | taiga removes "http" from everywhere, even from code and scripts.. | 09:18 |
| quiquell | arxcruz|ruck: you need the clouds.yaml thing that is described in the README | 09:18 |
| sshnaidm | so stupid interface.. | 09:18 |
| sshnaidm | why the heck someone would remove "http" from urls?? | 09:19 |
| arxcruz|ruck | quiquell: and there's no way to generate this right ? | 09:21 |
| quiquell | arxcruz|ruck: nope | 09:21 |
| quiquell | arxcruz|ruck: just put it there | 09:21 |
| quiquell | arxcruz|ruck: we don't want to mess with peoples clouds.yaml | 09:21 |
| *** dtantsur|afk is now known as dtantsur | 09:32 | |
| *** derekh has joined #oooq | 09:37 | |
| arxcruz|ruck | quiquell: this only works on libvirt right ? | 09:37 |
| quiquell | arxcruz|ruck: openstack and libvirt | 09:38 |
| quiquell | arxcruz|ruck: that's why you need the clouds.yaml | 09:38 |
| arxcruz|ruck | how do i set to run on rdocloud ? | 09:38 |
| quiquell | arxcruz|ruck: to point to your personal tenant | 09:38 |
| quiquell | arxcruz|ruck: you need to have at your ~/.config/openstack/clouds.yaml file something like this https://docs.openstack.org/python-openstackclient/pike/configuration/index.html | 09:39 |
| quiquell | arxcruz|ruck: for your tenant | 09:39 |
| sshnaidm | quiquell, merging? https://review.rdoproject.org/r/#/c/18515/ | 09:41 |
| sshnaidm | quiquell, look at this too: https://review.rdoproject.org/r/#/c/18559/ | 09:44 |
| quiquell | sshnaidm: +w both | 09:47 |
| quiquell | weshay, panda: This week is kind of difficult for me have to leave early monday/wednesday/friday around 4pm my time, will have to leave retro earlier | 09:59 |
| quiquell | Damn floating ips disappear from my tenant after creating them | 09:59 |
| panda | lol | 10:03 |
| panda | quiquell: ok, no worries, ruck/rover shift for you as punishment | 10:04 |
| quiquell | panda: With pleasure | 10:05 |
| quiquell | panda: I was going to sacrifice myself for the shake of reproducer though | 10:05 |
| quiquell | panda: So I don't get too much involve with it | 10:06 |
| quiquell | sshnaidm: virt-resize works now in the debug review :-( | 10:15 |
| quiquell | arxcruz|ruck: cloudinit review merged, you can update your clone and try again | 10:16 |
| quiquell | sshnaidm: ^ | 10:16 |
| *** ratailor has quit IRC | 10:26 | |
| *** ratailor has joined #oooq | 10:27 | |
| quiquell | panda: I am going to workflow your stuff here https://review.openstack.org/#/c/618780/ | 10:29 |
| quiquell | panda: it's ok ? | 10:29 |
| quiquell | sshnaidm, ykarel: It's ok to merge the 'config' related to run repro CI at libvirt role changes ? https://review.rdoproject.org/r/#/c/18297/ | 10:30 |
| sshnaidm | quiquell, +w | 10:35 |
| ykarel | ack | 10:36 |
| sshnaidm | quiquell, seems like now virt-resize is stuck.. let's see | 10:37 |
| quiquell | sshnaidm: let's see reproducer ci as third party CI here https://review.openstack.org/#/c/565839 | 10:38 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages) | 10:47 |
| *** dsneddon has quit IRC | 10:47 | |
| *** chem has quit IRC | 10:48 | |
| sshnaidm | quiquell, this patch will have a birthday soon.. | 10:51 |
| * sshnaidm is preparing a cake | 10:51 | |
| quiquell | sshnaidm: It's like dogs openstack review years is *7 or something like that | 10:52 |
| quiquell | sshnaidm: as perception of human years | 10:52 |
| quiquell | sshnaidm: at least now we can test some libvirt at CI :-) | 10:52 |
| sshnaidm | quiquell, what did you mean to say about openstack devs..? | 10:53 |
| sshnaidm | :D | 10:53 |
| quiquell | woot ? | 10:53 |
| quiquell | damn cloudflare issue again | 10:54 |
| quiquell | sshnaidm: how do we do the docker proxy to RDO ? | 10:54 |
| quiquell | sshnaidm: something like this ? https://github.com/openstack-infra/zuul-jobs/blob/master/roles/install-docker/tasks/mirror.yaml | 10:56 |
| *** rfolco has joined #oooq | 10:56 | |
| sshnaidm | quiquell, we have it in /etc/ci/mirrors, need just to set a proxy | 11:01 |
| quiquell | sshnaidm: suppose we do similar for sova and rrockpit | 11:02 |
| chandankumar | sshnaidm: http://logs.openstack.org/00/627500/75/check/tripleo-ci-centos-7-standalone-os-tempest/a1ef2e4/logs/stestr_results.html | 11:04 |
| chandankumar | sshnaidm: it passed | 11:04 |
| chandankumar | panda: sshnaidm: https://review.openstack.org/633185 and https://review.openstack.org/627500 now good to go | 11:05 |
| *** dsneddon has joined #oooq | 11:12 | |
| *** dsneddon has quit IRC | 11:17 | |
| sshnaidm | quiquell, not too much info: https://logs.rdoproject.org/58/18558/3/check/tripleo-ci-reproducer-centos-7-libvirt/e955706/tripleo-ci-reproducer/virt-resize.log | 11:18 |
| quiquell | sshnaidm: it appears something else at a working one ? | 11:24 |
| quiquell | sshnaidm: it should | 11:24 |
| *** ratailor has quit IRC | 11:27 | |
| *** ratailor has joined #oooq | 11:28 | |
| sshnaidm | quiquell, https://logs.rdoproject.org/58/18558/3/check/tripleo-ci-reproducer-fedora-28-libvirt/7b779c8/tripleo-ci-reproducer/virt-resize.log | 11:28 |
| quiquell | sshnaidm: :/ | 11:29 |
| quiquell | sshnaidm: not much | 11:29 |
| sshnaidm | quiquell, maybe let's collect journal too | 11:31 |
| quiquell | sshnaidm: journal https://review.rdoproject.org/r/18558 | 11:40 |
| panda | ok, ready to move to reproducer again. | 11:40 |
| panda | or you can hit me with review | 11:41 |
| panda | s | 11:41 |
| quiquell | panda: not yet, let me know if you find something with the repro | 11:45 |
| panda | ok | 11:45 |
| quiquell | panda: cloudinit images mechanism is merged you can update the role | 11:45 |
| quiquell | panda: and remove thte images specification part from the playbook no need to share images anymore | 11:45 |
| *** dsneddon has joined #oooq | 11:52 | |
| *** dsneddon has quit IRC | 11:56 | |
| quiquell | arxcruz|ruck: https://review.rdoproject.org/r/18569 | 12:07 |
| *** panda is now known as panda|lunch | 12:07 | |
| quiquell | arxcruz|ruck: can you try with that ? just pass user_key: id_rsa_no_password | 12:07 |
| quiquell | arxcruz|ruck: generate new one without password | 12:07 |
| quiquell | arxcruz|ruck: btw depending on openssh version you need -m PEM at ssh-keygen | 12:08 |
| ykarel | quiquell, sshnaidm re. the virt-resize it might be related to the nested kernel bug | 12:17 |
| sshnaidm | ykarel, what is a bug? | 12:17 |
| quiquell | ykarel: man you know everything :-) | 12:17 |
| ykarel | sshnaidm, https://bugs.launchpad.net/tripleo/+bug/1743749 | 12:18 |
| openstack | Launchpad bug 1743749 in tripleo "Task modify-image : Run virt-customize on the provided image fails while uploading image" [Critical,Won't fix] - Assigned to yatin (yatinkarel) | 12:18 |
| ykarel | sshnaidm, do you have a reproducer vm? if so u can try running: libguestfs-test-tool | 12:18 |
| ykarel | or just for testing try modifying:- http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/playbooks/libvirt-nodepool.yml#n26 | 12:19 |
| quiquell | ykarel: argg I remember that a bug at RHEL was pending | 12:19 |
| ykarel | like LIBGUESTFS_BACKEND_SETTINGS: "{{ lookup( 'env', 'LIBGUESTFS_BACKEND_SETTINGS')|default('force_tcg', true) }}" | 12:20 |
| sshnaidm | quiquell, ykarel yeah, we can add running test-tool | 12:20 |
| quiquell | ykarel: but that's super slow, isn't it ? | 12:20 |
| ykarel | quiquell, yup it's slow | 12:20 |
| quiquell | ykarel: so we can do a fallback in case test-tool fails ? | 12:20 |
| ykarel | i think's it sused for setup nodes, so should not effect much | 12:21 |
| quiquell | sshnaidm, ykarel: Ok going to add just the call to test-tool and see if it fails | 12:22 |
| sshnaidm | quiquell, did it | 12:23 |
| quiquell | sshnaidm: ack, where ? | 12:24 |
| quiquell | ykarel: so it depends on the hypervisor RHEL version ? | 12:25 |
| *** ratailor has quit IRC | 12:25 | |
| quiquell | arxcruz|ruck: It's working like a charm | 12:26 |
| quiquell | arxcruz|ruck: you can check the progress at http://localhost:9000 | 12:26 |
| arxcruz|ruck | quiquell++ | 12:27 |
| hubbot1 | arxcruz|ruck: quiquell's karma is now 18 | 12:27 |
| quiquell | arxcruz|ruck: ssh the running node and hold it in case of failure | 12:27 |
| arxcruz|ruck | quiquell: how can i hold it ? | 12:27 |
| quiquell | arxcruz|ruck: now you can use --skip-tags start and it will just launch again the job without restart zuul | 12:27 |
| quiquell | arxcruz|ruck: hold is not automatize yet | 12:28 |
| ykarel | quiquell, yes it's hypervisor dependent, kvm nested bug | 12:28 |
| quiquell | arxcruz|ruck: you have to run "docker-compose exec scheduler zuul autohold --tenant tripleo-ci-reproducer --reason whatever --project test1 --job "<job name>" | 12:28 |
| arxcruz|ruck | shit | 12:28 |
| quiquell | arxcruz|ruck: yes I know... | 12:29 |
| quiquell | arxcruz|ruck: panda is automatizing it | 12:29 |
| panda|lunch | ?gear.Client.b'unknown' - ERROR - Exception while connecting to <gear.Connection 0x7fd2b5b54898 host: scheduler port: 4730> socket.gaierror: [Errno -2] Name does not resolve | 12:30 |
| quiquell | panda|lunch: this usually means scheduler is not starting up | 12:31 |
| *** dsneddon has joined #oooq | 12:31 | |
| quiquell | panda|lunch: arx have launch job without problem | 12:31 |
| *** skramaja has joined #oooq | 12:31 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| panda|lunch | ... | 12:31 |
| quiquell | panda|lunch: tmate ? | 12:31 |
| chandankumar | arxcruz|ruck: Hey | 12:32 |
| *** panda|lunch is now known as panda | 12:33 | |
| quiquell | sshnaidm: btw how much people is using centos7 to launch repro libvirt ? | 12:33 |
| quiquell | skramaja: usualy the have fedoras | 12:33 |
| sshnaidm | quiquell, I think more than fedora | 12:33 |
| quiquell | sshnaidm: ups ok | 12:33 |
| sshnaidm | quiquell, people launches this on their minidells, servers, etc | 12:34 |
| sshnaidm | quiquell, not on their laptops | 12:34 |
| panda | I would never launch this on my laptop | 12:35 |
| arxcruz|ruck | chandankumar: ho, let's go | 12:35 |
| *** dsneddon has quit IRC | 12:35 | |
| panda | quiquell: I don't know, if arxcruz|ruck had this working on the first try, maybe I should just start from scratch | 12:35 |
| arxcruz|ruck | have what? | 12:36 |
| panda | arxcruz|ruck: a working reproducer on the first try ? | 12:36 |
| panda | arxcruz|ruck: did I dream it ? | 12:37 |
| quiquell | panda: well he has the know ssh key issues | 12:37 |
| quiquell | panda: after that was working fine | 12:37 |
| quiquell | panda: directly launching the job | 12:37 |
| chandankumar | arxcruz|ruck: need some help here to debug this http://logs.openstack.org/67/631967/5/check/openstack-ansible-functional-centos-7/292e9e3/logs/openstack/tempest1/stestr_results.html.gz | 12:38 |
| chandankumar | ssh timedout issue | 12:38 |
| chandankumar | arxcruz|ruck: I tried my hooks and crooks but no hope | 12:38 |
| quiquell | panda: try just an RDO fedora28 image | 12:39 |
| quiquell | panda: and run it there | 12:39 |
| chandankumar | arxcruz|ruck: there might be some floating ip issue, I am not sure what is wrong, | 12:40 |
| arxcruz|ruck | chandankumar: http://logs.openstack.org/67/631967/5/check/openstack-ansible-functional-centos-7/292e9e3/logs/openstack/openstack1/nova/nova-api-wsgi.log.txt.gz#_2019-01-22_05_34_12_136 | 12:40 |
| arxcruz|ruck | i'm seeing this errors | 12:40 |
| arxcruz|ruck | chandankumar: also i'm not seeing nova-compute logs | 12:40 |
| chandankumar | arxcruz|ruck: nova compute logs http://logs.openstack.org/67/631967/5/check/openstack-ansible-functional-centos-7/292e9e3/logs/host/nova/ | 12:41 |
| arxcruz|ruck | chandankumar: http://logs.openstack.org/67/631967/5/check/openstack-ansible-functional-centos-7/292e9e3/logs/openstack/openstack1/neutron/neutron-server.log.txt.gz#_2019-01-22_04_59_33_532 | 12:41 |
| arxcruz|ruck | neutron can't connect to mysql | 12:41 |
| arxcruz|ruck | access denied | 12:42 |
| arxcruz|ruck | chandankumar: also on nova side http://logs.openstack.org/67/631967/5/check/openstack-ansible-functional-centos-7/292e9e3/logs/host/nova/nova-compute.log.txt.gz#_2019-01-22_05_16_46_210 | 12:42 |
| chandankumar | arxcruz|ruck: let me collect stuff at one place, and look | 12:45 |
| chandankumar | heading home | 12:45 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 12:47 |
| sshnaidm | quiquell, what do mirrors roles do in libvirt mode? https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/61c685245f6d1d5b615e9548940c73e070ff987c/files/projects/zuul-config/playbooks/base/pre.yaml#L27 | 12:52 |
| *** rlandy has joined #oooq | 12:54 | |
| quiquell | sshnaidm: To use RDO mirror at docker in CI https://review.rdoproject.org/r/18571 | 12:55 |
| quiquell | sshnaidm: all the pre.yaml is still needed and also isntalling libvirt packages | 12:56 |
| quiquell | rlandy: o/ | 12:56 |
| sshnaidm | quiquell, yeah, but what does this role do when running libvirt? | 12:57 |
| quiquell | sshnaidm: you mean CI(CI) ? | 12:57 |
| sshnaidm | quiquell, no | 12:57 |
| quiquell | sshnaidm: Don't se the relation between this role and libvirt | 12:58 |
| sshnaidm | quiquell, it's in base job, right? | 12:58 |
| rlandy | quiquell: hey | 12:58 |
| quiquell | sshnaidm: yep same as upstream and RDO jobs | 12:58 |
| quiquell | rlandy: commented at the reproducer script review | 12:58 |
| rlandy | quiquell: worked with weshay last week and we redid the libvirt clone piece | 12:58 |
| rlandy | yeah - thanks | 12:58 |
| sshnaidm | quiquell, so it's running when we run libvirt reproducer? | 12:58 |
| quiquell | sshnaidm: yep | 12:59 |
| rlandy | quiquell: weshay wants to setup to look like ci | 12:59 |
| sshnaidm | quiquell, and what mirrors does it set? | 12:59 |
| quiquell | rlandy: what do you mean ? | 12:59 |
| rlandy | so if tq id there in ci, clone it before we start | 12:59 |
| rlandy | is | 12:59 |
| quiquell | sshnaidm: mirrof_fqdn variable | 12:59 |
| quiquell | sshnaidm: it's set up by zuul at startup I think | 12:59 |
| sshnaidm | quiquell, hmm.. and what is it in libvirt case | 13:00 |
| rfolco | panda, o/ | 13:00 |
| quiquell | sshnaidm: we can look in the inventory I think | 13:00 |
| rfolco | panda, looks like f28 jobs are failing on undecloud install... https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-fedora-28-master-containers-build | 13:01 |
| sshnaidm | quiquell, do you have some from libvirt run? because I didn't run libvirt yet.. | 13:01 |
| rlandy | quiquell:he wants us to copy roles and playbooks as the ansible user is used to | 13:01 |
| *** chem has joined #oooq | 13:01 | |
| rlandy | will explain more in meeting | 13:01 |
| rlandy | that' why we have the copy | 13:01 |
| rfolco | panda, please update me when you get a breath | 13:01 |
| quiquell | rlandy: ack | 13:01 |
| quiquell | sshnaidm: not now | 13:02 |
| sshnaidm | rlandy, do you have log files from libvirt run maybe? | 13:02 |
| rlandy | sshnaidm: not for a while | 13:02 |
| sshnaidm | ok, let's see later | 13:03 |
| quiquell | sshnaidm: what's the isse, btw the mirror thing is wrong at the internal CI | 13:03 |
| quiquell | sshnaidm: at RDO http://mirror.none.none.rdoproject.org/pypi/simple | 13:03 |
| quiquell | rlandy: I remember you were seting mirror_fqdn at jobs, is that right ? | 13:04 |
| rlandy | quiquell: yes | 13:04 |
| marios | quiquell: trying mirror.regionone.rdo-cloud-tripleo.rdoproject.org https://review.rdoproject.org/r/18539 | 13:04 |
| sshnaidm | quiquell, wrong mirror breaks jobs | 13:04 |
| rlandy | quiquell: for libvirt it's not set | 13:04 |
| sshnaidm | quiquell, it should be parametrized in reproducer I think | 13:04 |
| weshay | morning | 13:04 |
| rlandy | quiquell: we never tested libvirt in ci | 13:04 |
| marios | rlandy: quiquell i just updated my review to use that mirror.regionone.rdo-cloud-tripleo.rdoproject.org | 13:04 |
| quiquell | rlandy: I think we have to setup zuul somewhere so it's global indepdendent of nodepool_provider | 13:04 |
| quiquell | marios: Let's take a look | 13:05 |
| marios | rlandy: cos of failing with 2019-01-25 14:37:14.193690 | primary | "msg": "Failure talking to yum: failure: repodata/repomd.xml from base: [Errno 256] No more mirrors to try.\nhttp://mirror.none.none.rdoproject.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - \"Could not resolve host: mirror.none.none.rdoproject.org; Unknown error\"" | 13:05 |
| marios | http://logs.rdoproject.org/39/18539/2/check/tripleo-ci-reproducer-fedora-28-libvirt/0355fd4/tripleo-ci-reproducer/logs/01/1001/1/check/tripleo-ci-centos-7-standalone-reprozuul-dryrun/66b1f16/job-output.txt.gz | 13:05 |
| panda | rfolco: no need, the US is done | 13:05 |
| sshnaidm | marios, rdo cloud mirror works for vms on rdo cloud only | 13:05 |
| quiquell | rlandy: well now we run hello reproducer but it's installing nothing, after acivating toci dryrun is failing | 13:05 |
| marios | sshnaidm: so what do i need for that host: mirror.none.none.rdoproject.org; Unknown error\ | 13:06 |
| weshay | sshnaidm, quiquell have you guys seen the zuul container zuul/zuul scheduler_1 go down in the job? during deploy or later? | 13:06 |
| rlandy | we used to use NODEPOOL_MIRROR_HOST=mirror.mtl01.inap.openstack.org | 13:06 |
| rlandy | quiquell: ^^ | 13:06 |
| rlandy | but that was outside ci | 13:06 |
| sshnaidm | quiquell, either that ^^ or not to set it at all | 13:06 |
| quiquell | weshay: o/ nope | 13:06 |
| weshay | hrm.. | 13:06 |
| quiquell | weshay: also I reconnect to my laptop in the morning and it's still there no need to restart | 13:07 |
| quiquell | weshay: fedora28 | 13:07 |
| weshay | hrm | 13:07 |
| weshay | zuul/zuul? | 13:07 |
| sshnaidm | quiquell, rdo cloud better to run with its mirrors, all the rest should be parameter | 13:07 |
| quiquell | weshay: arx has job running there with cloudinit | 13:07 |
| weshay | where? | 13:07 |
| weshay | quiquell, we missed some reqs btw https://review.rdoproject.org/r/#/c/18545/ | 13:08 |
| weshay | meh.. white space | 13:08 |
| sshnaidm | marios, need to parametrize the mirror stuff.. | 13:08 |
| * weshay fixes it | 13:08 | |
| quiquell | weshay: docker-compose is isntalled with pip | 13:08 |
| quiquell | weshay: should we uses package ? | 13:08 |
| weshay | quiquell, oh.. so you source | 13:08 |
| weshay | quiquell, if there is a package.. yes | 13:08 |
| weshay | unless someone can tell me a compelling reason not to | 13:09 |
| quiquell | weshay: well at CI export $PATH=~/.local/bin:$PATH | 13:09 |
| quiquell | weshay: ansible and source at command is no good | 13:09 |
| quiquell | weshay: have give me problems | 13:09 |
| quiquell | weshay: version I suppose | 13:09 |
| weshay | quiquell, if we're not using the same version of packages we're not testing the same thing | 13:09 |
| quiquell | weshay: can you remove the pip install from your review and we check there ? | 13:10 |
| weshay | aye | 13:10 |
| quiquell | weshay: is with pip but with version | 13:10 |
| quiquell | weshay: same version everyone | 13:10 |
| weshay | rlandy, explains the docker-compose logs issue probably | 13:10 |
| weshay | ^ | 13:10 |
| quiquell | docker-compose==1.23.2 | 13:10 |
| weshay | rpm > pip | 13:10 |
| quiquell | weshay: pin the packages | 13:11 |
| *** dsneddon has joined #oooq | 13:11 | |
| weshay | quiquell, hrm.. for rpms? | 13:12 |
| quiquell | weshay: newgrp is problematic | 13:12 |
| weshay | quiquell, it doesn't work otherwise | 13:13 |
| quiquell | weshay: it changes default group | 13:13 |
| weshay | your ci is not clean | 13:13 |
| quiquell | weshay: could be, what did you detect ? | 13:13 |
| weshay | quiquell, if you need a blank system... or fire up a vm in rdo | 13:13 |
| quiquell | weshay: let's fix it so we can rely on that | 13:13 |
| weshay | and you'll find the same | 13:13 |
| weshay | quiquell, agree | 13:13 |
| quiquell | weshay: do you know if we have clean f28 nodesets from zuul ? | 13:13 |
| weshay | I'm not suggesting it has to be newgrp | 13:14 |
| quiquell | weshay: and c7 so you can do CI on clean nodes ? | 13:14 |
| weshay | however this setup fails as is on a new f28 install | 13:14 |
| quiquell | weshay: would be nice to test that at a clean f28 in CI :-/ | 13:14 |
| weshay | sure would | 13:14 |
| weshay | quiquell, -1 on pinning rpm packages btw | 13:16 |
| *** dsneddon has quit IRC | 13:16 | |
| quiquell | weshay: how so ? | 13:16 |
| weshay | it's going to be a disaster when users try to install | 13:16 |
| quiquell | weshay: Don't we have to ensure at least a minimal version ? | 13:16 |
| weshay | quiquell, if we need to pin rpm packages ... let's review and add that config to /etc/yum | 13:17 |
| quiquell | weshay: peoples /etc/yum ? | 13:17 |
| *** jfrancoa has quit IRC | 13:18 | |
| quiquell | weshay: what's the issue with pinning at pip ? | 13:19 |
| quiquell | weshay: and maybe using virtualenv to run the whole zuul enchilada ? | 13:19 |
| weshay | let's raise it for discussion.. | 13:20 |
| weshay | for mvp I don't think it matters much, but in the long run it does | 13:20 |
| quiquell | weshay: ack, btw have to drop earlier at meetings this week monday, wednesday and friday | 13:21 |
| weshay | k | 13:21 |
| arxcruz|ruck | weshay: hey boss, found the problem on fedora jobs, but don't know how to fix it, i add my comments on the lp | 13:24 |
| *** jfrancoa has joined #oooq | 13:24 | |
| weshay | arxcruz|ruck, it's ovn? | 13:24 |
| *** derekh has quit IRC | 13:24 | |
| arxcruz|ruck | weshay: openvswitch | 13:27 |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** agopi has quit IRC | 13:31 | |
| weshay | arxcruz|ruck, so can you check in w/ ykarel re: that package in fedora vs. centos | 13:32 |
| arxcruz|ruck | ykarel: https://bugs.launchpad.net/tripleo/+bug/1813224 | 13:32 |
| openstack | Launchpad bug 1813224 in tripleo "fedora28 standalone failing on tempest" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 13:32 |
| weshay | arxcruz|ruck, or go compare it to the centos version itself | 13:32 |
| weshay | perhaps it needs an update | 13:32 |
| arxcruz|ruck | i'm on ci scalation | 13:32 |
| weshay | panda, did the issue w/ the promoter script get resolved? | 13:35 |
| ykarel | arxcruz|ruck, weshay, same version of openvswitch in both Fedora and Centos | 13:37 |
| weshay | hrm.. | 13:37 |
| ykarel | 2.10.1 | 13:37 |
| weshay | it's causing an additional 10% of the jobs to fail | 13:38 |
| weshay | and it's a good comparison | 13:38 |
| arxcruz|ruck | so, something is wrong because it's failing, so the vm doesn't get network, or stays in build status, or get error | 13:38 |
| weshay | because both jobs are using the same containers | 13:38 |
| weshay | arxcruz|ruck, actually check that please | 13:38 |
| weshay | arxcruz|ruck, make sure the centos-7 and fedora container hashes are at the same level.. first run a test by hand | 13:38 |
| weshay | that's the next step | 13:39 |
| panda | weshay: the one in friday ? I had to stop and start the systemd service, and also cleaned up old containers. I've seen it working and promoting. | 13:40 |
| weshay | panda, thanks | 13:41 |
| marios | rlandy: was that https://github.com/sshnaidm/sova/commit/d18e0454dd33b7a3a0deaa5d9bed17a716673501 mistake (i saw it was reverted also the job names aren't periodic..? ) | 13:43 |
| marios | sshnaidm: how do i add periodic-tripleo-ci-centos-7-scenario001-standalone-master (and scenario002 003 004) to http://cistatus.tripleo.org/promotion/ please i can't see periodic anywahere under https://github.com/sshnaidm/sova thanks | 13:45 |
| sshnaidm | marios, yes, I'd prefer not to merge w/o my review, it's just complicated there | 13:48 |
| sshnaidm | rlandy, weshay ^ | 13:48 |
| weshay | panda, arxcruz|ruck needs to make sure fedora mixed and centos are at the same dlrn hash | 13:48 |
| weshay | https://trunk.rdoproject.org/fedora/ | 13:48 |
| sshnaidm | marios, I hope soon it will be much more clear | 13:48 |
| weshay | https://trunk.rdoproject.org/centos7-master/ | 13:48 |
| weshay | panda, can you give him a hand w/ that? | 13:48 |
| arxcruz|ruck | weshay: pinging you on cix | 13:48 |
| weshay | arxcruz|ruck, don't change the fedora hash w/o a manual test | 13:48 |
| sshnaidm | marios, I'm adding them right now.. | 13:49 |
| *** dsneddon has joined #oooq | 13:49 | |
| marios | sshnaidm: thanks mate | 13:49 |
| weshay | arxcruz|ruck, try and answer for me please | 13:49 |
| marios | sshnaidm: i'll check the commit /me curious :) | 13:49 |
| panda | arxcruz|ruck: why you need fedora and centos hashes to be the same ? | 13:52 |
| arxcruz|ruck | panda: not exactly, i need to check if both hashes are the same right now | 13:52 |
| arxcruz|ruck | because fedora 28 are failing with openvswitch | 13:52 |
| arxcruz|ruck | failing in tempest* | 13:52 |
| arxcruz|ruck | but centos is not failing, both distros, have the package in the same version | 13:52 |
| *** dsneddon has quit IRC | 13:53 | |
| weshay | arxcruz|ruck, better stated as.. centos is passing at 93% in check, fedora is 80'ish in check | 13:55 |
| arxcruz|ruck | weshay: they are not same hashe btw | 13:55 |
| arxcruz|ruck | hash | 13:56 |
| weshay | arxcruz|ruck, ya.. so let's get the jobs 100% the same | 13:56 |
| weshay | as much as possible, then tear apart why fedora may be failing more often | 13:56 |
| arxcruz|ruck | panda: how do i do that? :D | 13:56 |
| arxcruz|ruck | it seems fedora needs to be changed manually | 13:56 |
| weshay | so run a recreate w/ the same hash a centos-7 | 13:57 |
| weshay | if that works.. | 13:57 |
| arxcruz|ruck | weshay: on fedora it doesn't | 13:57 |
| arxcruz|ruck | i was talking with ykarel about it this morning | 13:57 |
| weshay | you run the dlrnapi command to promote the fedora hash | 13:57 |
| arxcruz|ruck | i'm trying the new reproducer with quiquell | 13:57 |
| weshay | arxcruz|ruck, meh.. that may be too new :) | 13:58 |
| weshay | but if it works for you ok | 13:58 |
| panda | arxcruz|ruck: too soon | 13:58 |
| weshay | arxcruz|ruck, run the old one too manybe | 13:58 |
| arxcruz|ruck | weshay: the old one doesn't work on fedora | 13:58 |
| rlandy | sshnaidm: marios: ok - can you explain then what we need to do to add periodics? weshay and I tried two approaches | 13:59 |
| rlandy | neither worked | 13:59 |
| panda | arxcruz|ruck: where do you see the fedora job failing ? | 13:59 |
| panda | arxcruz|ruck: in periodic ? | 13:59 |
| sshnaidm | rlandy, it was wrong branch | 13:59 |
| marios | rlandy: /me watching https://github.com/sshnaidm/sova/commits/master | 13:59 |
| rlandy | sshnaidm: which is the correct branch? | 14:00 |
| sshnaidm | rlandy, marios https://github.com/sshnaidm/sova/tree/promtest | 14:00 |
| sshnaidm | rlandy, marios I'll do branch names more clear.. | 14:00 |
| rlandy | sshnaidm: what is promstat for? | 14:00 |
| sshnaidm | rlandy, deleted it | 14:00 |
| marios | rfolco: o/ | 14:01 |
| arxcruz|ruck | panda: on check | 14:01 |
| marios | sshnaidm: yeah i saw a million branches on the sova github :D | 14:02 |
| sshnaidm | marios, yeah, made some cleanup there | 14:02 |
| marios | weshay: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:02 |
| *** agopi has joined #oooq | 14:03 | |
| *** trown|outtypewww is now known as trown | 14:03 | |
| weshay | arxcruz|ruck, ssbarnea|bkp2 can you guys join the scrum for a minute | 14:04 |
| weshay | we need to talk ptg | 14:04 |
| panda | arxcruz|ruck: is fedora28 still out of promotion critaria ? | 14:04 |
| arxcruz|ruck | panda: yes | 14:04 |
| weshay | panda, I thought the mixed job had to be manual promote? | 14:04 |
| weshay | rlandy, mtg | 14:04 |
| *** derekh has joined #oooq | 14:13 | |
| *** agopi has quit IRC | 14:13 | |
| *** agopi has joined #oooq | 14:13 | |
| *** agopi_ has joined #oooq | 14:15 | |
| *** agopi_ has quit IRC | 14:16 | |
| *** agopi_ has joined #oooq | 14:16 | |
| marios | weshay: from https://tree.taiga.io/project/tripleo-ci-board/task/543?kanban-status=1447276 review.openstack.org/#/q/topic:replace-scen2+(status:open+OR+status:merged) - only panko review.openstack.org/#/c/628251 still left (been pinging irc chans, sending email and generally harrassing people) - has +2A but looks like the py37 is borked there and its not gonna merge until that is fixed or removed. | 14:16 |
| *** agopi has quit IRC | 14:18 | |
| *** agopi_ is now known as agopi | 14:18 | |
| *** dsneddon has joined #oooq | 14:23 | |
| marios | thanks sshnaidm (cc rlandy weshay https://github.com/sshnaidm/sova/commit/0b2a4c97aa516b12d080405626f803de0c7fd7a4 ) | 14:27 |
| *** dsneddon has quit IRC | 14:28 | |
| marios | sshnaidm: do we stil need to reset something for the jobs to appear in http://cistatus.tripleo.org/promotion/ or is it a matter of time(on next runs?) | 14:31 |
| weshay | rfolco, I'm going to share | 14:37 |
| rfolco | weshay, please do so | 14:37 |
| *** ykarel is now known as ykarel|away | 14:43 | |
| *** quiquell is now known as quiquell|off | 14:47 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 14:47 |
| *** dsneddon has joined #oooq | 14:55 | |
| marios | sshnaidm: see them now thanks review.openstack.org/#/q/topic:replace-scen2+(status:open+OR+status:merged) - only panko review.openstack.org/#/c/628251 still left (been pinging irc chans, sending email and generally harrassing people) - has +2A but looks like the py37 is borked there and its not gonna merge until that is fixed or removed. | 15:00 |
| marios | sshnaidm: oops meant this http://cistatus.tripleo.org/promotion/ (wrong paste) | 15:00 |
| marios | sorry | 15:00 |
| *** dsneddon has quit IRC | 15:00 | |
| marios | sshnaidm: the periodic-tripleo-ci-centos-7-scenario001-standalone-master and friends | 15:00 |
| chandankumar | arxcruz|ruck: marios: sshnaidm panda weshay https://review.openstack.org/#/c/633185/ and https://review.openstack.org/#/c/627500/ finishes os_Tempest integration greenlight by ci | 15:01 |
| sshnaidm | marios, not sure I understand about panko.. | 15:02 |
| *** ykarel|away has quit IRC | 15:07 | |
| marios | sshnaidm: 17:00 < marios> sshnaidm: oops meant this http://cistatus.tripleo.org/promotion/ (wrong paste) | 15:08 |
| marios | sshnaidm: (was wrong paste sorry the panko was from earlier i pasted to weshay) | 15:08 |
| *** chandankumar is now known as chkumar|out | 15:08 | |
| weshay | thanks chkumar|out | 15:09 |
| rlandy | sshnaidm: weshay: pls see quiquell|off comment on gerrit_user ... https://review.openstack.org/#/c/631067/23/roles/create-zuul-based-reproducer/templates/reproducer-zuul-based-quickstart.sh.j2 | 15:12 |
| rlandy | that we will need both | 15:12 |
| rlandy | this is to run pre | 15:13 |
| weshay | sshnaidm, so for https://tree.taiga.io/project/tripleo-ci-board/task/644?kanban-status=1447276 | 15:15 |
| sshnaidm | marios, so everything is ok with sova promotion jobs? | 15:15 |
| weshay | you want to change the compose to use the rdo container reg. vs docker.io? | 15:15 |
| weshay | rdoci / zuul-base | 15:15 |
| weshay | rdoci / zuul-migrate | 15:15 |
| weshay | rdoci / zuul-merger | 15:15 |
| weshay | rdoci / zuul-bwrap | 15:15 |
| weshay | rdoci / zuul-fingergw | 15:15 |
| weshay | rdoci / zuul-executor | 15:15 |
| weshay | rdoci / zuul-scheduler | 15:15 |
| weshay | rdoci / zuul-web | 15:15 |
| weshay | rdoci / zuul | 15:15 |
| weshay | rdoci / nodepool-base | 15:15 |
| weshay | rdoci / nodepool-launcher | 15:15 |
| weshay | rdoci / nodepool | 15:15 |
| sshnaidm | weshay, rdoci is namespace on docker.io | 15:15 |
| weshay | rdoci / nodepool-builder | 15:15 |
| weshay | or the rdoci namespace I guess.. either registry | 15:15 |
| marios | sshnaidm: yes thanks i see them in cistatus.tripleo.org | 15:16 |
| weshay | sshnaidm, k.. see it https://hub.docker.com/u/rdoci | 15:16 |
| weshay | sshnaidm, ah.. k.. and the defaults in ansible-role-t-r are updated.. | 15:17 |
| weshay | sshnaidm, so.. we just need someone else to run through it once.. and we can move to complete | 15:17 |
| sshnaidm | weshay, yeah | 15:18 |
| weshay | sshnaidm, k cool | 15:18 |
| chkumar|out | weshay: I wanted to keep it voting as os_tempest has all the stuff we needed | 15:18 |
| weshay | chkumar|out, ok.. let's chat tomorrow | 15:19 |
| weshay | go be out :) | 15:19 |
| chkumar|out | hehe | 15:19 |
| chkumar|out | :-) | 15:19 |
| weshay | sshnaidm, the zuul container has been crashing on me | 15:19 |
| weshay | so maybe that will fix it | 15:19 |
| weshay | I'll give it a go | 15:19 |
| weshay | sshnaidm, we can move https://tree.taiga.io/project/tripleo-ci-board/task/642?kanban-status=1447276 to done right | 15:20 |
| weshay | not used now that we have cloud-init | 15:20 |
| sshnaidm | weshay, yeah, not needed anymore | 15:20 |
| weshay | k.. /me moving to done | 15:20 |
| weshay | sshnaidm, k.. 3/4 moved to done | 15:22 |
| weshay | will rerun a full run now w/ rdoci containers | 15:22 |
| weshay | thanks | 15:22 |
| *** dsneddon has joined #oooq | 15:27 | |
| weshay | sshnaidm, do you have an opinion on install things like docker and docker-compose via pip vs rpm? | 15:27 |
| sshnaidm | weshay, you can't install docker via pip | 15:28 |
| sshnaidm | weshay, it's only rpm | 15:28 |
| weshay | sshnaidm, was going test this change as well https://review.rdoproject.org/r/#/c/18545/2/playbooks/tripleo-ci-reproducer/pre.yaml | 15:28 |
| weshay | ssbarnea|bkp2, you here? or pto today | 15:28 |
| weshay | rfolco, panda we have a planning mtg | 15:29 |
| rfolco | joining | 15:29 |
| sshnaidm | weshay, and docker-compose is not so important I think, pip will give a newest, but I don't think we use some special features there, maybe better rpm | 15:29 |
| weshay | 1 sec for me | 15:30 |
| *** dsneddon has quit IRC | 15:32 | |
| *** ykarel|away has joined #oooq | 15:36 | |
| *** dsneddon has joined #oooq | 15:59 | |
| *** gkadam has quit IRC | 16:00 | |
| *** skramaja has quit IRC | 16:02 | |
| *** dsneddon has quit IRC | 16:04 | |
| *** dsneddon has joined #oooq | 16:27 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 16:47 |
| *** agopi is now known as agopi|lunch | 16:58 | |
| *** jfrancoa has quit IRC | 16:59 | |
| *** kopecmartin is now known as kopecmartin|off | 17:18 | |
| *** trown is now known as trown|lunch | 17:42 | |
| *** bogdando has quit IRC | 17:45 | |
| rlandy | stack failures in ovb | 17:47 |
| rlandy | No more IP addresses available on network 1f6dd7c9-1c87-4310-a1c3-bd7c0346d6ad. | 17:47 |
| rlandy | arxcruz|ruck: ^^ seen that? | 17:47 |
| weshay | hrm.. rlandy we may still be in outtage | 17:55 |
| * weshay looks | 17:55 | |
| *** agopi|lunch is now known as agopi | 17:55 | |
| rlandy | some stack started | 17:56 |
| rlandy | fs10 was able to run | 17:56 |
| *** derekh has quit IRC | 18:00 | |
| weshay | rlandy, I have a clean system setup again | 18:06 |
| rlandy | weshay:k - great | 18:06 |
| weshay | so going to test a bit.. and start writing some doc I think | 18:07 |
| weshay | not sure if RDO will cooperate.. | 18:07 |
| weshay | is libvirt ready for just -l at this point? | 18:07 |
| *** jpena is now known as jpena|off | 18:13 | |
| *** ykarel|away has quit IRC | 18:13 | |
| weshay | rlandy, oh noes.. http://logs.openstack.org/67/631067/23/check/tripleo-ci-centos-7-standalone/646e627/logs/reproducer-quickstart/ | 18:35 |
| weshay | do you have a rlandy version? | 18:35 |
| *** trown|lunch is now known as trown | 18:36 | |
| rlandy | weshay: https://logs.rdoproject.org/67/631067/27/openstack-check/tripleo-ci-centos-7-multinode-1ctlr-featureset010/c7802aa/logs/zuul-based-reproducer-quickstart/ | 18:36 |
| weshay | woot | 18:37 |
| weshay | thanks! | 18:37 |
| rlandy | weshay: with your install ansible | 18:37 |
| weshay | nice | 18:37 |
| rlandy | just generated ... will need to be tested | 18:37 |
| weshay | k.. got ur back there | 18:38 |
| rlandy | weshay: did you ever add a secret cred to jenkins | 18:38 |
| rlandy | I see the old one matt added | 18:38 |
| rlandy | I can modify that one | 18:38 |
| rlandy | but I would prefer to add a new one for upstream reporting | 18:38 |
| weshay | a secret cred.. no I have not | 18:39 |
| rlandy | on credentials, there is supposed to be ab add button but I don't see it | 18:39 |
| rlandy | oh well, I guess I will just have to modify the one there | 18:39 |
| weshay | so.. do we even need it? | 18:40 |
| weshay | those jobs will no longer be reporting to review.openstack.org? | 18:40 |
| weshay | maybe I can save you some work by clarifying? | 18:40 |
| rlandy | weshay: the old job used to report to dlrn api | 18:41 |
| weshay | ya.. oh.. reporting to dlrn | 18:41 |
| weshay | yes++ | 18:41 |
| weshay | thought you meant to review.openstack | 18:41 |
| rlandy | with DLRN_PASSWORD | 18:41 |
| weshay | aye.. that did change after some joker leaked the passwd | 18:42 |
| weshay | weshay <----- | 18:42 |
| rlandy | lol | 18:42 |
| * rlandy updates | 18:42 | |
| weshay | it's in the infra doc | 18:42 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message) | 18:47 |
| rlandy | weshay:sorry to be an idiot - which infra doc - looking in the DFG drive | 18:48 |
| rlandy | I could pull it off the reporting server | 18:48 |
| weshay | ha.. I can never find it either.. sec | 18:48 |
| rlandy | if it's the same one | 18:48 |
| weshay | sec | 18:49 |
| *** rfolco has quit IRC | 19:00 | |
| *** dtantsur is now known as dtantsur|afk | 19:32 | |
| *** rfolco has joined #oooq | 19:34 | |
| weshay | rlandy++ | 19:52 |
| hubbot1 | weshay: rlandy's karma is now 43 | 19:52 |
| weshay | rlandy, cp -n is no-clobbler | 19:54 |
| weshay | just in case someone does the boneheaded thing | 19:54 |
| rlandy | weshay: sure - pls comment on review and I'll fix it | 19:55 |
| weshay | k | 19:55 |
| rlandy | weshay: could tar the three files | 19:59 |
| rlandy | then the user would only wget the tar'ed file and untar | 19:59 |
| rlandy | may be easier | 20:00 |
| weshay | rlandy, ah.. that's a good idea | 20:11 |
| weshay | rlandy, fyi https://review.openstack.org/#/c/631067/27/roles/create-zuul-based-reproducer/templates/reproducer-zuul-based-quickstart.sh.j2 | 20:11 |
| rlandy | weshay: ok - fixing that and adding tar | 20:14 |
| rlandy | weshay: question about comment ... | 20:14 |
| weshay | aye | 20:15 |
| rlandy | looks like we need to install git too :) | 20:15 |
| rlandy | if ! ansible --version; then | 20:15 |
| rlandy | sudo $(command -v dnf || command -v yum) install -y git | 20:15 |
| rlandy | fi | 20:15 |
| rlandy | if no ansible install git?? | 20:15 |
| rlandy | check for git, install git? correct? | 20:15 |
| weshay | oh CRUD.. | 20:16 |
| weshay | sorry cut paste error | 20:16 |
| weshay | if no ansible, install | 20:16 |
| weshay | if no git, install | 20:16 |
| rlandy | k - got it | 20:16 |
| weshay | http://pastebin.test.redhat.com/703305 | 20:17 |
| rlandy | panda: you around? | 20:17 |
| rlandy | panda: I'm permission denied reporting to dlrn_api - with what I think is correct password | 20:17 |
| rlandy | there are two possible users | 20:18 |
| rlandy | both permission denied | 20:18 |
| rlandy | could use some help there | 20:18 |
| weshay | hrm | 20:18 |
| weshay | tmate? | 20:18 |
| * weshay just got | 20:19 | |
| weshay | TASK [Add user to docker group] ************************************************************************** | 20:19 |
| weshay | task path: /var/tmp/RECREATE/playbooks/pre.yaml:70 | 20:19 |
| weshay | fatal: [localhost]: FAILED! => {"changed": false, "msg": "useradd: user 'root' already exists\n", "name": "$USER", "rc": 9} | 20:19 |
| rlandy | weshay: yep - hit that as well | 20:22 |
| rlandy | new | 20:22 |
| weshay | k.. fixing | 20:24 |
| weshay | ansible suucks some times | 20:24 |
| weshay | TASK [Clone repos needed for reproducer] ***************************************************************** | 20:24 |
| weshay | ok: [localhost] => (item=https://git.openstack.org/openstack/tripleo-quickstart.git) | 20:24 |
| weshay | failed: [localhost] (item=https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer.git) => {"before": "ec6f13b5bfc5e730c5975bcb7bbae5f5aa386a6c", "changed": false, "item": "https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer.git", "msg": "Local modifications exist in repository (force=no)."} | 20:24 |
| weshay | to retry, use: --limit @/var/tmp/RECREATE/launcher-env-setup-playbook.retry | 20:24 |
| weshay | think we'll need to dir check | 20:24 |
| weshay | things get complicated when you try to make it easy | 20:24 |
| rlandy | weshay: where is the change? | 20:24 |
| *** zenpac has quit IRC | 20:25 | |
| weshay | rlandy, putting it up, not done | 20:25 |
| rlandy | weshay: also, usability question .. when adding the tar file, keep the zuul-based-reproducer-quickstart dir with the separate files or ditch the dir and out one tar file in main dir? | 20:26 |
| rlandy | put one | 20:26 |
| weshay | https://review.rdoproject.org/r/18580 | 20:27 |
| weshay | rlandy, I think you can ditch the dir | 20:28 |
| rlandy | not sure that https://review.rdoproject.org/r/#/c/18580/ should be necessary | 20:29 |
| rlandy | ansible_user should be defined correctly | 20:29 |
| rlandy | or I am using it wrong | 20:29 |
| rlandy | besides this used to work | 20:30 |
| rlandy | something has changed | 20:30 |
| weshay | ya.. not 100% sure either | 20:30 |
| panda | rlandy: use the user/password in the promoter server | 20:32 |
| rlandy | panda: should this work from any machine? | 20:32 |
| weshay | rlandy, ya | 20:38 |
| weshay | rlandy, note the example in the doc has options to prevent it from running | 20:38 |
| weshay | updated https://review.rdoproject.org/r/18580 fyi rlandy | 20:40 |
| weshay | works now | 20:40 |
| rlandy | weshay: lol - oh those empty spaces ... https://review.rdoproject.org/r/#/c/18580/2/playbooks/tripleo-ci-reproducer/pre.yaml | 20:41 |
| weshay | lolz | 20:41 |
| weshay | vi | 20:41 |
| * weshay needs to get a real .vimrc | 20:41 | |
| rlandy | weshay: git mine from marios ... copying | 20:42 |
| rlandy | got | 20:42 |
| rlandy | weshay: http://pastebin.test.redhat.com/703311 | 20:43 |
| weshay | thanks! | 20:44 |
| rlandy | weshay: https://review.openstack.org/631067 updated | 20:45 |
| rlandy | let's see what ci does with it | 20:45 |
| weshay | nice | 20:45 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message) | 20:47 |
| weshay | rlandy, we may need to create a docker group and newgrp in the shell file | 20:52 |
| weshay | I don't think ansible can do it | 20:52 |
| rlandy | weshay: what about if you just run launcher? | 20:54 |
| * rlandy can add to the script | 20:56 | |
| panda | rlandy: it should work from everywhere, the promoter server has only the latest working credentials | 20:58 |
| rlandy | panda: k - got distracted - checking | 20:59 |
| weshay | oh dam | 21:14 |
| rlandy | panda: hmm ... still HTTP response body: Unauthorized Access | 21:15 |
| rlandy | using password from server | 21:16 |
| panda | rlandy: can you paste the command ? | 21:27 |
| rlandy | panda: see pvt | 21:27 |
| *** jtomasek has quit IRC | 21:29 | |
| rlandy | weshay: how goes it now? | 21:29 |
| weshay | rlandy, so.. ur stuff is working fine :) | 21:30 |
| weshay | rlandy, having issues w/ docker logs, user, group stuff on a new install | 21:30 |
| rlandy | panda++ | 21:34 |
| hubbot1 | rlandy: panda's karma is now 14 | 21:34 |
| panda | \o/ | 21:34 |
| panda | I can sleep tonight. | 21:35 |
| *** panda is now known as panda|off | 21:35 | |
| weshay | rlandy, think I have to try w/ libvirt | 21:37 |
| weshay | rdo cloud is still hosed | 21:37 |
| rlandy | good luck with that | 21:37 |
| *** panda|off has quit IRC | 21:54 | |
| *** panda has joined #oooq | 21:56 | |
| rlandy | weshay: wget https://logs.rdoproject.org/67/631067/28/openstack-check/tripleo-ci-centos-7-multinode-1ctlr-featureset010/7985848/logs/reproducer-zuul-based-quickstart.tar | 22:24 |
| weshay | nice | 22:24 |
| rlandy | tar -xvf reproducer-zuul-based-quickstart.tar | 22:24 |
| rlandy | files are in +x mode | 22:25 |
| weshay | I hit the libvirt module import issue | 22:25 |
| rlandy | what is that??? | 22:25 |
| rlandy | libvirt thing? | 22:25 |
| rlandy | weshay: going to start adding some lines to https://logs.rdoproject.org/67/631067/28/openstack-check/tripleo-ci-centos-7-multinode-1ctlr-featureset010/7985848/logs/README-reproducer-zuul-based-quickstart.html | 22:27 |
| rlandy | feel free to edit/change | 22:27 |
| rlandy | unless you have a review somewhere?? | 22:27 |
| weshay | k | 22:27 |
| weshay | rlandy, I have a clone of your review | 22:27 |
| weshay | that I'm adding doc to, but all I have thus far is the wget :) | 22:27 |
| weshay | lolz | 22:27 |
| weshay | I think the ansible rpm may not have it | 22:29 |
| weshay | lolz | 22:29 |
| rlandy | weshay: k - well, I'll add some doc to my review and we can combine it | 22:29 |
| weshay | OH, it's undercloud | 22:30 |
| weshay | it's under... cloud | 22:30 |
| weshay | lolz | 22:30 |
| * weshay tries | 22:30 | |
| rlandy | what? what? | 22:30 |
| weshay | https://github.com/ansible/ansible-modules-extras/search?q=libvirt&unscoped_q=libvirt | 22:31 |
| weshay | hrm.. | 22:32 |
| weshay | wrong | 22:32 |
| weshay | rpm -q --whatprovides $PWD/virt_pool.py | 22:32 |
| weshay | ansible-2.7.5-1.fc28.noarch | 22:32 |
| rlandy | under... cloud, over ... cloud reminds me of wombles https://www.youtube.com/watch?v=XWQMMPFtoG4 - probably never got that show in America | 22:33 |
| weshay | oooh | 22:33 |
| weshay | EXEC /bin/sh -c '/usr/bin/python3 /home/weshayutin/.ansible/tmp/ansible-tmp-1548714751.77-138242198197363/AnsiballZ_virt_pool.py && sleep 0' | 22:33 |
| weshay | rpm -qa | grep ansible | 22:33 |
| weshay | ansible-2.7.5-1.fc28.noarch | 22:33 |
| weshay | ansible-openstack-modules-0-20140907git79d751a.fc28.noarch | 22:33 |
| weshay | in ansible-python3 it's under cloud | 22:36 |
| weshay | hrm.. didn't help | 22:38 |
| weshay | trying ansible-playbook-3 | 22:38 |
| * rlandy should look at that again | 22:38 | |
| rlandy | trying | 22:38 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message) | 22:47 |
| rlandy | k - running now | 22:49 |
| weshay | python3-libvirt-4.1.0-1.fc28.x86_64 | 22:49 |
| weshay | is not installed | 22:49 |
| weshay | rlandy, ya.. that was it | 22:53 |
| weshay | and python3-lxml | 22:53 |
| weshay | yup | 22:53 |
| * weshay updates https://review.rdoproject.org/r/#/c/18545/4/playbooks/tripleo-ci-reproducer/pre.yaml | 22:54 | |
| rlandy | weshay: so we need to install that | 22:54 |
| rlandy | python3-openstacksdk already installed | 22:56 |
| weshay | https://review.rdoproject.org/r/#/c/18545/5/playbooks/tripleo-ci-reproducer/pre.yaml | 22:57 |
| rlandy | ah | 22:57 |
| rlandy | getting | 22:57 |
| rlandy | weshay: the $USER thing needs to change | 23:02 |
| weshay | rlandy, ya.. that I have working | 23:02 |
| rlandy | changing to lookup | 23:02 |
| weshay | https://review.rdoproject.org/r/#/c/18580/ | 23:02 |
| weshay | lookup env thing? | 23:02 |
| rlandy | giving me more problems | 23:02 |
| rlandy | yeah - that thing | 23:03 |
| rlandy | doesn't always resolve | 23:03 |
| rlandy | will fix | 23:03 |
| weshay | same issue here | 23:04 |
| weshay | http://pastebin.test.redhat.com/703342 | 23:04 |
| rlandy | hmmm ... TASK [libvirt/setup/overcloud : Set hostname correctly for subnode-0] ******************************************************************************************************************************** | 23:04 |
| rlandy | task path: /tmp/reproduce-tmp.vhxER/roles/libvirt/setup/overcloud/tasks/libvirt_nodepool.yml:198 | 23:04 |
| rlandy | fatal: [localhost -> 192.168.122.64]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 192.168.122.64 closed.\r\n", "module_stdout": "/bin/sh: /usr/bin/python3: No such file or directory\r\n", "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127} | 23:04 |
| rlandy | weshay: I'll fix that in the script ... ^^ my next error | 23:04 |
| rlandy | probably missing another install | 23:05 |
| weshay | so right off the bat.. I think we need python3, ansible-3 | 23:06 |
| rlandy | hostnamectl | 23:07 |
| weshay | rlandy, https://review.rdoproject.org/r/#/c/18580/7..8/tasks/libvirt/main.yaml | 23:08 |
| weshay | fyi.. I think that will fix my error | 23:08 |
| weshay | rlandy, that looks like ur just missing python3 to me | 23:08 |
| rlandy | I have python3 | 23:08 |
| rlandy | -e ansible_python_interpreter="/usr/bin/python3" | 23:09 |
| rlandy | that error is on the node | 23:09 |
| weshay | hrm.. maybe ur further along than I am | 23:09 |
| weshay | probably | 23:09 |
| rlandy | maybe because I only have one user | 23:09 |
| rlandy | something to look forward to | 23:09 |
| * weshay keeps adding to https://review.rdoproject.org/r/#/c/18545/6/playbooks/tripleo-ci-reproducer/pre.yaml | 23:09 | |
| rlandy | yeah -- I am getting your review as I go | 23:10 |
| weshay | ok.. getting further | 23:11 |
| weshay | injecting key into image... adding zuul | 23:14 |
| rlandy | and here we go again | 23:14 |
| * weshay getting there | 23:14 | |
| weshay | uploading to volume pool | 23:14 |
| weshay | rlandy, yup.. just hit ur error | 23:15 |
| rlandy | TASK [libvirt/setup/overcloud : Resize undercloud image (call virt-resize)] | 23:15 |
| rlandy | I have your latest change | 23:15 |
| rlandy | think I am getting further | 23:16 |
| rlandy | TASK [libvirt/setup/overcloud : Upload the volume to storage pool] | 23:16 |
| rlandy | actually ha - maybe not | 23:16 |
| rlandy | yep- same error | 23:16 |
| weshay | hrm.. it's not installed on the image | 23:17 |
| rlandy | [zuul@localhost ~]$ which hostnamectl | 23:17 |
| rlandy | usr/bin/hostnamectl | 23:17 |
| rlandy | found it | 23:17 |
| rlandy | node doesn't have python3? | 23:18 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/f8fefa51436785cec8fa420506e40193cce4d607/roles/libvirt/setup/overcloud/tasks/libvirt_nodepool.yml#L198 | 23:18 |
| weshay | I don't think it's that.. | 23:19 |
| weshay | it may not have the hostnamectl command installed | 23:19 |
| weshay | say what | 23:19 |
| weshay | rpm -q --whatprovides /usr/bin/hostnamectl | 23:19 |
| weshay | systemd-238-10.git438ac26.fc28.x86_64 | 23:19 |
| weshay | - name: Set hostname correctly for subnode-0 | 23:20 |
| weshay | delegate_to: subnode-0 | 23:20 |
| weshay | shell: > | 23:20 |
| weshay | echo "127.0.0.1 subnode-0 localhost" > /etc/hosts; | 23:20 |
| weshay | echo "HOSTNAME=subnode-0" >> /etc/sysconfig/network; | 23:20 |
| weshay | echo "subnode-0" > /etc/hostname; | 23:20 |
| weshay | hostnamectl set-hostname subnode-0; | 23:20 |
| weshay | echo "nameserver {{ custom_nameserver|default('8.8.8.8') }} " >> /etc/resolv.conf; | 23:20 |
| weshay | echo "append domain-name-servers {{ custom_nameserver|default('8.8.8.8') }};" >> /etc/dhcp/dhclient.conf | 23:20 |
| weshay | become: true | 23:20 |
| weshay | rlandy, it's just running shell | 23:20 |
| rlandy | it works on the node itself | 23:20 |
| rlandy | I am on the node | 23:20 |
| weshay | OH.. "module_stdout": "/bin/sh: /usr/bin/python3: No such file or directory\r\n", | 23:20 |
| weshay | I see what ur saying | 23:20 |
| rlandy | delegate_to | 23:21 |
| rlandy | not sure what voodoo happens there | 23:21 |
| weshay | rlandy, wait wait.. | 23:21 |
| weshay | look at the task above this | 23:21 |
| weshay | hrml | 23:21 |
| weshay | maybe not | 23:21 |
| weshay | ansible_python_interpreter: "{{ python_interpreter|default('/usr/bin/python') }}" | 23:21 |
| weshay | OH crud.. | 23:22 |
| weshay | so if you have ansible-playbook-3 you need python3 on all the targets? | 23:22 |
| rlandy | that is what I am thinking | 23:22 |
| rlandy | although ci does just fine | 23:22 |
| weshay | rlandy, for centos that's a pita | 23:22 |
| rlandy | I am not running ansible-playbook-3 | 23:23 |
| rlandy | still an error | 23:23 |
| * rlandy checks install on node | 23:23 | |
| rlandy | [root@localhost ~]# which python3 | 23:24 |
| rlandy | usr/bin/which: no python3 in (/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin) | 23:24 |
| rlandy | weshay: centos nodes | 23:25 |
| rlandy | not going to work | 23:25 |
| rlandy | as above <weshay> rlandy, for centos that's a pita | 23:25 |
| rlandy | epel? | 23:25 |
| weshay | rlandy https://logs.rdoproject.org/48/625648/8/openstack-check/tripleo-ci-reproducer-centos-7-libvirt/edd1f5b/job-output.txt.gz#_2019-01-28_11_22_48_165849 | 23:26 |
| weshay | yum install centos-release-scl; yum install rh-python36 | 23:26 |
| weshay | not sure if that is polluting the node or not | 23:27 |
| weshay | trying it | 23:27 |
| rlandy | we don;t have a choice | 23:27 |
| rlandy | weshay: it's just a matter of the fact that we force use python3 on the local machine | 23:28 |
| rlandy | I don;t know what ci installs | 23:28 |
| * rlandy fixes $USER problem in the mean time | 23:30 | |
| weshay | ok.. maybe I have this fixed.. | 23:42 |
| weshay | crosses fingers | 23:42 |
| weshay | rlandy, we may need to run ci w/ -vvvv to see really wtf it's doing | 23:42 |
| weshay | or get clean nodes | 23:42 |
| weshay | I suspect nodepool nodes have both python2.7 and python3.6 preinstalled | 23:43 |
| rlandy | weshay: afaict, the nodes only have python2 | 23:44 |
| rlandy | I mean by default | 23:44 |
| weshay | so are we killing ourselves by setting python3? | 23:44 |
| rlandy | we can fix that in nodepool setup | 23:44 |
| rlandy | weshay: I ma not sure we have a choice | 23:44 |
| rlandy | if I run w/o python3, it fails way before that | 23:44 |
| * rlandy gets | 23:45 | |
| weshay | hrm | 23:45 |
| rlandy | we can fix this | 23:45 |
| rlandy | weshay: https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/tasks/libvirt/prepare.yaml | 23:46 |
| rlandy | packages_list | 23:47 |
| weshay | hrm.. that happens after this though | 23:47 |
| weshay | maybe it doesnt | 23:48 |
| weshay | rlandy, oh ya.. it's after | 23:49 |
| weshay | rlandy, https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/tasks/libvirt/main.yaml#L48 | 23:49 |
| weshay | rlandy, atm, we're failing at https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/tasks/libvirt/main.yaml#L42 | 23:49 |
| rlandy | weshay: yeah - we may need to insert this task before | 23:51 |
| weshay | working on it | 23:51 |
| *** agopi is now known as agopi|brb | 23:52 | |
| *** agopi|brb is now known as agopi|off | 23:52 | |
| *** agopi|off has quit IRC | 23:53 | |
| weshay | rlandy, your pre-setup yaml playbook makes this easy to develop :) | 23:53 |
| weshay | imho | 23:53 |
| * weshay udpates the git dir... | 23:54 | |
| weshay | and reruns | 23:54 |
| * rlandy is happy | 23:56 | |
| weshay | rlandy, woot.. got it | 23:56 |
| weshay | oh.. wait | 23:56 |
| weshay | running very verbose | 23:57 |
| rlandy | cool - change up? | 23:58 |
| rlandy | removing local edit ... | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!