| *** tosky has quit IRC | 00:01 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (1 more message) | 00:39 |
|---|---|---|
| *** rlandy has quit IRC | 01:00 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 02:39 |
| *** ykarel|away has joined #oooq | 03:12 | |
| *** apetrich has quit IRC | 03:15 | |
| *** chem has quit IRC | 03:59 | |
| *** chem has joined #oooq | 04:00 | |
| *** udesale has joined #oooq | 04:06 | |
| *** skramaja has joined #oooq | 04:11 | |
| *** dsneddon has quit IRC | 04:22 | |
| *** ykarel|away has quit IRC | 04:29 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 04:39 |
| *** dsneddon has joined #oooq | 04:46 | |
| *** ykarel|away has joined #oooq | 04:46 | |
| *** dsneddon has quit IRC | 04:54 | |
| *** ykarel|away is now known as ykarel | 05:23 | |
| *** dsneddon has joined #oooq | 05:27 | |
| *** saneax has joined #oooq | 05:28 | |
| *** udesale has quit IRC | 05:32 | |
| *** dsneddon has quit IRC | 05:32 | |
| *** dsneddon has joined #oooq | 05:33 | |
| *** udesale has joined #oooq | 05:36 | |
| *** ratailor has joined #oooq | 05:41 | |
| *** saneax has quit IRC | 05:51 | |
| *** quiquell|off is now known as quiquell | 06:26 | |
| *** udesale has quit IRC | 06:28 | |
| *** udesale has joined #oooq | 06:29 | |
| quiquell | marios, sshnaidm: as a temporal solution generate Gerrit keys from within the container | 06:29 |
| quiquell | I have to find what are the differences but that works | 06:29 |
| sshnaidm | quiquell, it's weird it works for me | 06:30 |
| quiquell | I suppose OpenSSH version | 06:30 |
| quiquell | For me too with my key | 06:30 |
| sshnaidm | quiquell, let's compare with my setup maybe | 06:30 |
| sshnaidm | quiquell, so what does not work? marios' key? | 06:30 |
| quiquell | Have to be that the version of OpenSSH was similar when we generated it | 06:30 |
| quiquell | Yep and tripleo ci key | 06:30 |
| *** udesale has quit IRC | 06:30 | |
| marios | sshnaidm: i think the trouble might be that the key i used initially had passphrase | 06:31 |
| quiquell | Also all the keys that I generate after updating my laptop | 06:31 |
| *** udesale has joined #oooq | 06:31 | |
| marios | quiquell: i should have just created new ones | 06:31 |
| quiquell | marios: nope is not password I am generated without it | 06:31 |
| quiquell | And have same problems | 06:31 |
| sshnaidm | quiquell, idk, it would be a big problem in the world if openssh change its way to generate keys | 06:32 |
| quiquell | sshnaidm: the problem is paramiko I think | 06:32 |
| sshnaidm | quiquell, which version do you use now? | 06:32 |
| quiquell | Or the way they use within zuul | 06:32 |
| sshnaidm | quiquell, mm.. that's possible, which version of paramiko? | 06:32 |
| quiquell | Tell you in a few | 06:33 |
| quiquell | marios: generated the key for now inside the containers and use those | 06:33 |
| *** udesale has quit IRC | 06:33 | |
| sshnaidm | quiquell, is there way to use "text" or "binary" version of file? I saw problems with gerrit about it | 06:33 |
| quiquell | While we figure it out | 06:33 |
| quiquell | Sshnai | 06:33 |
| quiquell | sshnaidm: don't know | 06:33 |
| sshnaidm | maybe permissions. | 06:33 |
| sshnaidm | marios, and you generated the new key, right? | 06:34 |
| quiquell | sshnaidm: no permissions normal ssh within the container works is just python stuff that don't work | 06:34 |
| quiquell | But the new line review is legit was broken | 06:34 |
| sshnaidm | quiquell, can you add debug message there to print a key it tries to use? | 06:35 |
| sshnaidm | quiquell, I think it's somewhere in zuul scheduler.. | 06:35 |
| quiquell | Will do | 06:35 |
| sshnaidm | maybe marios has 2 keys in gerrit now and it can't work with multiple keys | 06:36 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 06:39 |
| quiquell | Also I don't know if the user@hostname at the pub key in Gerrit affects it | 06:41 |
| quiquell | Will test some ideas | 06:42 |
| sshnaidm | quiquell, shouldn't affect | 06:47 |
| quiquell | At least we have a temporal solution | 06:51 |
| quiquell | docket-compose run scheduler /bin/sh | 06:51 |
| quiquell | ssh-keygen | 06:51 |
| quiquell | And copy keys | 06:52 |
| quiquell | But we should discover the issue | 06:52 |
| quiquell | First thing is to unblock repro CI | 06:53 |
| quiquell | https://review.openstack.org/#/c/629839 | 07:04 |
| quiquell | sshnaidm, marios: the libvirt review revote, thanks | 07:05 |
| quiquell | sshnaidm: about sudo I think it's not a blocker | 07:05 |
| marios | quiquell: ack going to reviews in a sec | 07:05 |
| quiquell | We need that to merge the two main repro patches | 07:05 |
| quiquell | Livirt and launch job | 07:06 |
| quiquell | marios: do you have reviews to check? | 07:06 |
| sshnaidm | quiquell, +2 | 07:06 |
| quiquell | sshnaidm: what was your nodepool review? | 07:07 |
| quiquell | We have to start to merge things before sprint ends | 07:07 |
| quiquell | Or we will be in sprint limbo | 07:08 |
| marios | just this (updated for your comments thanks) https://review.openstack.org/631024 & check this though i have to update (and needs the jobs merged first https://review.rdoproject.org/r/#/q/topic:standalone-scenario-promotion | 07:09 |
| marios | quiquell: thanks ^ | 07:09 |
| marios | quiquell: the diff between v29 and 30 is this? https://review.openstack.org/#/c/629839/29..30/roles/libvirt/teardown/nodes/tasks/main.yml | 07:10 |
| marios | why the update | 07:10 |
| sshnaidm | quiquell, https://review.openstack.org/#/c/630649/ | 07:13 |
| quiquell | marios: commented https://review.openstack.org/631024 | 07:15 |
| *** jfrancoa has joined #oooq | 07:15 | |
| quiquell | marios: do we want to put stuff about phase1 there? | 07:15 |
| marios | quiquell: thanks | 07:15 |
| marios | quiquell: not sure maybe can come later, i wanted it to be mainly about gate/check/promotion | 07:16 |
| marios | quiquell: i mean phase 1 is explained a bit in the promotions stages doc i refer to | 07:16 |
| quiquell | Ack let's merge that is very good info we all missed at the beginning | 07:16 |
| marios | quiquell: it is action item from retrospective. like how i messed up with voting check scenario standalones, that weren't in the gate ! | 07:17 |
| marios | quiquell: so that is where the doc is coming from. i just had to explain some bare minimum other stuff to describe that bit. hence it became like a primer on the jobs | 07:17 |
| quiquell | Yep, devil is in the details | 07:17 |
| quiquell | Mario's re: main | 07:17 |
| quiquell | marios: linting | 07:18 |
| marios | wut | 07:18 |
| quiquell | https://review.openstack.org/#/c/629839/29..30/roles/libvirt/teardown/nodes/tasks/main.yml | 07:18 |
| marios | oh k thx | 07:19 |
| quiquell | If you see it's ok just workflow it | 07:20 |
| chandankumar | marios: quiquell \o/ | 07:22 |
| quiquell | chandankumar: hello there | 07:22 |
| chandankumar | marios: quiquell https://review.openstack.org/#/c/628421/ https://review.openstack.org/#/c/628415/ and https://review.openstack.org/#/c/627500/ in sequence are good to go for os_tempest | 07:23 |
| marios | quiquell: done https://review.openstack.org/#/c/631024/ | 07:27 |
| marios | ack chandankumar | 07:28 |
| *** saneax has joined #oooq | 07:29 | |
| *** udesale has joined #oooq | 07:30 | |
| quiquell | +2 | 07:31 |
| quiquell | sshnaidm: lapton OpenSSH_7.8p1, OpenSSL 1.1.0i-fips 14 Aug 2018, container OpenSSH_7.7p1, LibreSSL 2.7.4 | 07:36 |
| quiquell | paramiko version is the same in both | 07:36 |
| *** saneax has quit IRC | 07:36 | |
| *** saneax has joined #oooq | 07:38 | |
| *** ykarel is now known as ykarel|lunch | 07:39 | |
| *** kopecmartin|off is now known as kopecmartin | 07:47 | |
| sshnaidm | chandankumar, where os_tempest is actually running? do you have logs? | 07:48 |
| chandankumar | sshnaidm: marios https://review.openstack.org/#/c/627500/ | 07:50 |
| chandankumar | here it is running | 07:50 |
| chandankumar | sshnaidm: http://logs.openstack.org/00/627500/68/check/tripleo-ci-centos-7-standalone-os-tempest/a0faeaf/ | 07:50 |
| marios | man does anyone know anyone in openstack-telemetry | 07:50 |
| marios | i keep spamming them | 07:50 |
| marios | but they ignore me | 07:50 |
| marios | wai | 07:50 |
| chandankumar | silhet is the guy | 07:51 |
| *** jpena|off is now known as jpena | 08:14 | |
| *** gkadam has joined #oooq | 08:22 | |
| chandankumar | marios: replied anything we need to change there | 08:22 |
| *** dsneddon has quit IRC | 08:24 | |
| quiquell | sshnaidm: could be related to this https://github.com/net-ssh/net-ssh/issues/633 | 08:25 |
| quiquell | marios: Did your key have RSA or OPENSSH delimiters | 08:25 |
| marios | chandankumar: ack will check | 08:25 |
| marios | quiquell: the ones i created yesterday were all defaults (id_rsa etc) | 08:26 |
| quiquell | marios: Do they have BEGIN OPENSSH PRIVATE KEY header or BEGIN RSA PRIVATE KEY header | 08:26 |
| *** panda|off is now known as panda | 08:28 | |
| marios | quiquell: yes | 08:29 |
| marios | -----BEGIN OPENSSH PRIVATE KEY----- | 08:29 |
| quiquell | marios: this is the solution https://github.com/net-ssh/net-ssh/issues/638#issuecomment-441189002 | 08:29 |
| panda | "I'm turning japanese, I'm really turning japanese, I really think so" the weird musics on your way to the office | 08:29 |
| quiquell | marios: new key needed :-( | 08:29 |
| marios | quiquell: sweet something to try today then thanks | 08:30 |
| quiquell | marios: let me really check this | 08:30 |
| marios | quiquell: but anyway. once we work out the right keys w'e're gonna need new images anyway | 08:30 |
| marios | panda: are you wearing a kimono? | 08:31 |
| quiquell | marios: problem is if people needing to regenering gerrit keys is not a good thing | 08:31 |
| marios | quiquell: yeah but this is more about getting setup. maybe we can find some workaround or proper fix merges for that issues | 08:31 |
| quiquell | marios: problem is paramiko thinks it's not an RSA key ed25519 | 08:31 |
| quiquell | marios: because it parses the OPENSSH header, chaning header is not enough | 08:32 |
| panda | marios: no, with this nice winter chill I thought it was better to wear a yukata. | 08:33 |
| marios | nice | 08:34 |
| *** dsneddon has joined #oooq | 08:34 | |
| marios | panda-san | 08:34 |
| quiquell | marios: this is the paramiko issue https://github.com/paramiko/paramiko/issues/1015 | 08:34 |
| panda | quiquell: I have a question about definition of done for this sprint, are you targeting users at all ? Because I'm trying to get the reproducer to work as a user, and I don't want to wase anyone's time | 08:35 |
| marios | quiquell: damn looks like an old problem | 08:35 |
| panda | marios: a little respect please: "panda-kun". | 08:35 |
| panda | marios: also it seems for net-ssh not openssh | 08:35 |
| quiquell | panda: Just trying to make keys works mate | 08:36 |
| panda | mh, nevermind | 08:36 |
| quiquell | marios: well 21 days is not too old | 08:36 |
| marios | https://github.com/paramiko/paramiko/issues/1015 was opened in 17 | 08:37 |
| marios | https://github.com/net-ssh/net-ssh/issues/638#issuecomment-441189002 or this one you mean | 08:37 |
| *** dsneddon has quit IRC | 08:39 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 08:39 |
| *** ykarel|lunch is now known as ykarel | 08:40 | |
| *** ratailor_ has joined #oooq | 08:42 | |
| sshnaidm | chandankumar, does os_tempest job have some debug from tempest? | 08:42 |
| sshnaidm | chandankumar, like we have with validate_tempest role | 08:42 |
| chandankumar | sshnaidm: https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L17 | 08:43 |
| chandankumar | sshnaidm: we need to just enable flags | 08:43 |
| chandankumar | sshnaidm: but ara_report is useful it has all the info there what wents wrong | 08:43 |
| sshnaidm | chandankumar, so maybe we should do it? | 08:43 |
| chandankumar | sshnaidm: will update | 08:44 |
| chandankumar | sshnaidm: one more help I need | 08:44 |
| chandankumar | sshnaidm: https://review.openstack.org/#/c/628415/36/roles/collect-logs/tasks/publish.yml@74 | 08:44 |
| chandankumar | sshnaidm: copying stestr result is not working | 08:45 |
| *** ratailor has quit IRC | 08:45 | |
| sshnaidm | chandankumar, seems like use_os_tempest is not defined when running collect-logs playbook | 08:47 |
| chandankumar | sshnaidm: but https://review.openstack.org/#/c/628415/36/roles/collect-logs/tasks/publish.yml@61 stackviz one is working | 08:48 |
| sshnaidm | chandankumar, no, none of them: http://logs.openstack.org/00/627500/68/check/tripleo-ci-centos-7-standalone-os-tempest/a0faeaf/logs/quickstart_collect_logs.log | 08:49 |
| *** ccamacho has joined #oooq | 08:49 | |
| quiquell | marios, panda, sshnaidm: We need to regenerate keys with "ssh-keygen -m PEM -t rsa" in case they have the -----BEGIN OPENSSH PRIVATE KEY----- | 08:50 |
| chandankumar | sshnaidm: Do I need a task to make it cacheable also? | 08:50 |
| sshnaidm | chandankumar, oh, wait, it did work | 08:50 |
| *** tosky has joined #oooq | 08:50 | |
| marios | ssbarnea|rover: https://review.openstack.org/#/c/595795/10 ready for merge? did you have any chance for testing (i.e. were there any provision errors :) | 08:50 |
| ssbarnea|rover | marios: do we have a special irc keyword that we use to notify our entire time? for example openstack-infra has infra-root - anyone from the team is notified when this is mentioned. very handy. | 08:51 |
| sshnaidm | chandankumar, look at log file I posted, there are errors: p: cannot stat ‘/home/zuul/workspace/logs/undercloud/home/stack/tempest/tempest.xml’: No such file or directory | 08:51 |
| marios | ssbarnea|rover: not that i know of but that would be useful | 08:51 |
| sshnaidm | chandankumar, search by name of task | 08:51 |
| ssbarnea|rover | marios: re prov one, yes i tested, at the time I raised it there were such errors. is very easy to isolate the change an test it outside. i think that the msg explain well what was the bug. | 08:52 |
| ssbarnea|rover | marios: panda chandankumar sshnaidm weshay_PTO rlandy arxcruz|ruck : any keyword to propose? it should be short but not so short to get false positive. once we agree I will make a CR to document it. I propose oooq-core | 08:54 |
| marios | ssbarnea|rover: ci-squad | 08:55 |
| marios | ssbarnea|rover: tripleo-ci | 08:55 |
| ssbarnea|rover | marios: cannot use repo names, due to false positives | 08:55 |
| marios | ssbarnea|rover: hm right | 08:55 |
| marios | oooq-core +1 then | 08:55 |
| ssbarnea|rover | the best would be to document keyword in channel topic. i will wait a little to get enough feedback on name. | 08:56 |
| quiquell | os-tripleo-ci | 08:56 |
| ssbarnea|rover | we could also use the same pattern as infra team and use "oooq-root" -- "-root suffix on channel name" | 08:57 |
| *** apetrich has joined #oooq | 08:57 | |
| ssbarnea|rover | arxcruz|ruck: can you add +op to our team members in irc? it would be nice to see all listed at the top. somehow you are the only op now. | 08:58 |
| ssbarnea|rover | ...not that ever plan to start kicking people out :D | 08:58 |
| chandankumar | sshnaidm: I think I have used the wrong path | 08:59 |
| *** arxcruz|ruck sets mode: +o ssbarnea|rover | 08:59 | |
| panda | ssbarnea|rover: sorry, context ? | 08:59 |
| ssbarnea|rover | panda: intead of manually mentioning all team members to notify them on irc, we could use a keyword that we add on our irc clients. | 09:00 |
| ssbarnea|rover | for example, if you want to quickly notify anyone from #openstack-infra just use "infra-root" in your message, they will read the message for sure. | 09:01 |
| ssbarnea|rover | is like a group tag, another missing feature of irc. or we could use "@oooq" even shorter. | 09:02 |
| *** dsneddon has joined #oooq | 09:05 | |
| *** jaosorior has quit IRC | 09:05 | |
| quiquell | sshnaidm: changed secrets with the fix for the key https://review.rdoproject.org/r/18471 | 09:07 |
| quiquell | sshnaidm: we need to merge that | 09:07 |
| *** jaosorior has joined #oooq | 09:08 | |
| sshnaidm | quiquell, done | 09:08 |
| panda | sshnaidm: I still don't understand | 09:09 |
| panda | ssbarnea|rover: ^ | 09:09 |
| panda | one of you should change his name... just sayin' | 09:10 |
| panda | not just the nick, because it's unfair | 09:10 |
| quiquell | marios, panda, sshnaidm: Added info about keys issue at taiga user story https://tree.taiga.io/project/tripleo-ci-board/us/610?milestone=215191 | 09:11 |
| sshnaidm | panda, I thought to change to "pandy" | 09:11 |
| ssbarnea|rover | panda: re name, it may happen after i get my uk passport, i may decide to go for an easier name, en-friendly. | 09:12 |
| ssbarnea|rover | i am considering, "ssh" as seems very popular in it :D | 09:12 |
| sshnaidm | ssbarnea|rover, if you need help with name finding.. | 09:12 |
| sshnaidm | ssbarnea|rover, just call! | 09:12 |
| ssbarnea|rover | something with scottish accents like McBrokeit | 09:13 |
| quiquell | sshnaidm: testing new key here https://review.rdoproject.org/r/#/c/18450 | 09:13 |
| sshnaidm | ssbarnea|rover, but seems like to change nick is little easier than passport | 09:13 |
| ssbarnea|rover | sshnaidm: i would nor mind changing it but i am member on LOTs of ircchannels and people usually find me using this nickname which matches github username. that's the tricky part. | 09:14 |
| sshnaidm | quiquell, I need to test it for myself, I wouldn't like to break my setup :) | 09:14 |
| quiquell | sshnaidm: CI will tell us | 09:15 |
| sshnaidm | quiquell, I need to test with my key | 09:15 |
| sshnaidm | quiquell, seems like you use different key format | 09:16 |
| sshnaidm | quiquell, for me it worked great w/o newline | 09:16 |
| quiquell | sshnaidm: Yep the newline thing was also a issue that happends sometimes, but better to have exactly the same key injected | 09:20 |
| quiquell | sshnaidm: the real problem is the other issue with new versions of openssh | 09:20 |
| sshnaidm | quiquell, I have a newer version, seems like everything is ok | 09:21 |
| quiquell | ssh do you have the OPENSSH delimiters at your priv key ? | 09:22 |
| quiquell | opensshnaidm | 09:23 |
| quiquell | sshnaidm, panda, panda: CI working again https://review.rdoproject.org/r/#/c/18450/ | 09:23 |
| quiquell | :-) | 09:23 |
| marios | quiquell: i have your change from yesterday should i keep it (missing new line at injection) | 09:23 |
| quiquell | key issue is legit | 09:23 |
| marios | quiquell: gonna try now with the new keys | 09:23 |
| *** dtantsur|afk is now known as dtantsur | 09:23 | |
| quiquell | marios: yep totally in fact we are going to merge it | 09:24 |
| quiquell | sshnaidm, panda: can you merge https://review.rdoproject.org/r/#/c/18450/ ? | 09:24 |
| marios | quiquell ack | 09:24 |
| sshnaidm | quiquell, I want to test it before | 09:25 |
| quiquell | sshnaidm: ack | 09:25 |
| panda | quiquell: hey, where's my +2 Verified ? | 09:30 |
| quiquell | panda: Removed them aster yesterdays screw | 09:30 |
| panda | quiquell: good boy :) | 09:30 |
| quiquell | panda: attaboy they say | 09:30 |
| quiquell | panda: check your ssh private keys if they have a "OPENSSH" delimiter you need new ones for gerrit | 09:31 |
| quiquell | panda: there is a paramiko bug | 09:31 |
| *** derekh has joined #oooq | 09:31 | |
| sshnaidm | quiquell, ok, seems working for me | 09:34 |
| sshnaidm | panda, you can merge | 09:35 |
| sshnaidm | panda, oh, wait... | 09:35 |
| marios | quiquell: i _think_ its working :/ :) ... like no more key error but code": 204, message": "Tenant tripleo-ci-reproducer isn't ready and trace in scheduler like http://pastebin.test.redhat.com/700482 (docker ps looks good though? ) | 09:37 |
| quiquell | marios: this error is normal, it take time for tenant to be ready | 09:38 |
| marios | - ERROR - Exception on ssh event stream | 09:38 |
| quiquell | marios: you are all good | 09:38 |
| quiquell | sshnaidm: did you found something ? | 09:38 |
| sshnaidm | quiquell, that's fine | 09:39 |
| quiquell | sshnaidm: ok, let's merge | 09:39 |
| sshnaidm | I thin it's already | 09:40 |
| sshnaidm | s/thin/think | 09:40 |
| *** ratailor_ has quit IRC | 09:41 | |
| quiquell | sshnaidm: going back with OVB, did you advance on that ? | 09:44 |
| quiquell | sshnaidm: what was the issue ? | 09:44 |
| quiquell | sshnaidm: also public images, where we are with that ? | 09:44 |
| sshnaidm | quiquell, working on that now | 09:44 |
| sshnaidm | quiquell, fedoras are fixed | 09:44 |
| sshnaidm | quiquell, just share them from nodepool tenant to yours | 09:45 |
| quiquell | sshnaidm: they have key injected or they are cloud init aware ? | 09:45 |
| quiquell | zuul user I mean | 09:45 |
| quiquell | a wait the nodepool review | 09:45 |
| *** ratailor has joined #oooq | 09:45 | |
| sshnaidm | quiquell, there are both | 09:45 |
| sshnaidm | upstream-cloudinit-centos-7 | 09:46 |
| sshnaidm | upstream-cloudinit-fedora-28 | 09:46 |
| sshnaidm | with ssh keys: | 09:46 |
| sshnaidm | upstream-infra-centos-7 | 09:46 |
| sshnaidm | upstream-infra-fedora-28 | 09:46 |
| quiquell | sshnaidm: -cloudinit- depends on the nodepool review that's right ? | 09:46 |
| sshnaidm | quiquell, if you use cloud init image, you need rdoci/nodepool-launcher:patched - it includes my patch | 09:46 |
| quiquell | sshnaidm: so if we merge the nodepool review and make -cloudinit- public we are good for now | 09:46 |
| quiquell | sshnaidm: ahh.. the container :-) | 09:47 |
| sshnaidm | quiquell, either, yea | 09:47 |
| quiquell | sshnaidm: we can do that without upstream | 09:47 |
| sshnaidm | quiquell, of course | 09:47 |
| quiquell | sshnaidm: dangerous though | 09:47 |
| quiquell | sshnaidm: it has already a +2 | 09:47 |
| quiquell | sshnaidm: maybe we are not far | 09:47 |
| sshnaidm | quiquell, why dangerous? | 09:47 |
| sshnaidm | quiquell, we can do the same for holding nodes - just patch zuul and that's it | 09:47 |
| quiquell | sshnaidm: I mean changing images without automatic process is not infra as code | 09:47 |
| sshnaidm | one line change | 09:47 |
| sshnaidm | quiquell, well, yeah, but this is a different task to care about | 09:48 |
| quiquell | sshnaidm: we can forget why was working | 09:48 |
| quiquell | sshnaidm: agree | 09:48 |
| quiquell | sshnaidm: I am going to ask kforce to make -cloudinit- images public | 09:48 |
| sshnaidm | quiquell, yeah, maybe it's better - to update them once a half year and that's it | 09:49 |
| sshnaidm | that's a pity we can't update the image and use it with the same ID | 09:50 |
| quiquell | sshnaidm: I already found issues at fedora28 image with libvirt, after updateing it was working fine (kernel issue) | 09:50 |
| quiquell | sshnaidm: it was a CI(CI) test though | 09:50 |
| sshnaidm | quiquell, we can put update into cloud-init so it will execute before the job | 09:51 |
| sshnaidm | quiquell, but not sure about kernel updates, it should boot it.. | 09:52 |
| quiquell | sshnaidm: if have to update and reboot | 09:52 |
| quiquell | sshnaidm: but it takes a lot of time to update depending how old is it, moving this updateing offline is good | 09:52 |
| quiquell | sshnaidm: we can do the same with vanilla livirt images | 09:52 |
| sshnaidm | quiquell, yeah, but then you need to ask kforde to make them public every time | 09:54 |
| quiquell | sshnaidm: Maybe they can give nodepool user permissions to make public images ? | 09:55 |
| *** apetrich has quit IRC | 09:57 | |
| *** apetrich has joined #oooq | 09:58 | |
| quiquell | sshnaidm: Do you have the images IDs are the ones at the taiga task ? | 10:01 |
| sshnaidm | quiquell, I'll update the task | 10:02 |
| quiquell | sshnaidm: cool thanks | 10:02 |
| *** apetrich has quit IRC | 10:04 | |
| sshnaidm | quiquell, last comment: https://tree.taiga.io/project/tripleo-ci-board/task/570 | 10:08 |
| *** bogdando has joined #oooq | 10:08 | |
| *** jpena is now known as jpena|brb | 10:08 | |
| quiquell | sshnaidm: missing fedoras d40c59c9-8d6b-4f59-983f-8eff071590ac and 88006cd1-d089-4bd3-b70a-8cbc7eb32f63 | 10:10 |
| sshnaidm | quiquell, missing where? | 10:11 |
| quiquell | openstack --os-cloud rdo-cloud image set --accept 88006cd1-d089-4bd3-b70a-8cbc7eb32f63 | 10:11 |
| quiquell | Could not find resource 88006cd1-d089-4bd3-b70a-8cbc7eb32f63 | 10:11 |
| sshnaidm | quiquell, yeah, firstly you need to share it to your tenant | 10:14 |
| quiquell | sshnaidm: Ahh though you have already do that | 10:14 |
| quiquell | ack | 10:14 |
| sshnaidm | quiquell, what's your tenant id? | 10:14 |
| quiquell | sshnaidm: no problem has the nodepool creadentials | 10:14 |
| sshnaidm | quiquell, maybe we need a list of tenant ids of the team | 10:14 |
| sshnaidm | to make it automatically | 10:15 |
| quiquell | sshnaidm: well the username is good enough | 10:16 |
| quiquell | sshnaidm: so we have them | 10:16 |
| quiquell | nah is not | 10:16 |
| sshnaidm | quiquell, then list of usernames | 10:16 |
| sshnaidm | quiquell, I'm still not sure what's name of marios tenant :) | 10:16 |
| quiquell | "user marios" ? | 10:17 |
| jaosorior | anybody seeing this issue when running quickstart ? http://paste.openstack.org/show/743160/ | 10:17 |
| *** dsneddon has quit IRC | 10:19 | |
| sshnaidm | jaosorior, nope, seems like casual | 10:19 |
| marios | jaosorior: not seen that one maybe new ssbarnea|rover or arxcruz|ruck have you seen it ? http://paste.openstack.org/show/743160/ | 10:20 |
| marios | sshnaidm: for upstream gerrit you mean? | 10:20 |
| jaosorior | seeing that locally | 10:20 |
| marios | sshnaidm: yeah marios | 10:20 |
| sshnaidm | marios, no, tenant name in rdo-cloud | 10:20 |
| marios | sshnaidm: ah sec mandreou i think | 10:20 |
| sshnaidm | quiquell, ^ | 10:20 |
| ssbarnea|rover | marios: lookin now. | 10:20 |
| marios | ssbarnea|rover: yeah mandreou | 10:21 |
| quiquell | sshnaidm: nah we need tenant ids, tenant names does not work with "add project" | 10:21 |
| marios | er sorry ssbarnea|rover wrong nick | 10:21 |
| marios | sshnaidm: mandreou | 10:21 |
| sshnaidm | quiquell, ok, so list of IDs | 10:21 |
| arxcruz|ruck | jaosorior: nope, haven't seen that... but... | 10:21 |
| *** apetrich has joined #oooq | 10:21 | |
| arxcruz|ruck | ssbarnea|rover: told me that pip was updated yesterdao to 0.19 | 10:22 |
| arxcruz|ruck | yesterday* | 10:22 |
| quiquell | sshnaidm: do you have the commands around that you did to customize images ? | 10:22 |
| sshnaidm | quiquell, yep | 10:22 |
| arxcruz|ruck | pip 19.0 was released yesterday | 10:22 |
| quiquell | sshnaidm: going to put a playbook in place while you work at OVB | 10:22 |
| marios | quiquell: so how do i add a change to this thing. like do i have to git clone something for the local gerrit and push change review to it? | 10:22 |
| quiquell | marios: http://localhost:8080 | 10:23 |
| marios | quiquell: yeah seen that and zuul | 10:23 |
| quiquell | you have there two test projects | 10:23 |
| marios | quiquell: so how do i run a job there is what i want. like i need to submit a change to that gerrit | 10:23 |
| quiquell | just clone the projects and add a zuul.yaml review with the job you want to run | 10:23 |
| quiquell | we are automatizing that for reproducer | 10:23 |
| marios | quiquell: sweet this is coolness | 10:23 |
| quiquell | marios: yep submit a change to test1 or test2 | 10:24 |
| quiquell | marios: Yep, first time you see it working it's like blowing your mind | 10:24 |
| marios | quiquell: but will make my laptop explode | 10:24 |
| marios | even a standalone i think | 10:24 |
| quiquell | marios: is launching at your openstack tenant | 10:26 |
| quiquell | marios: not your laptop | 10:26 |
| quiquell | marios: If you want to make it blowup you use libvirt :-) | 10:26 |
| quiquell | sshnaidm: kforde making public the images, pass me the step you did to prepare a playbook | 10:27 |
| sshnaidm | quiquell, playbook for what? | 10:27 |
| quiquell | sshnaidm: get latest nodepool image and prepare it | 10:28 |
| sshnaidm | quiquell, ok, will send you | 10:29 |
| marios | quiquell: yeah i was thinking about libvirt | 10:29 |
| marios | quiquell: so thats cool didn't think about rdo. so we can launch from local onto rdo cloud | 10:29 |
| marios | w | 10:29 |
| marios | w | 10:29 |
| marios | o | 10:29 |
| marios | quiquell: my task is about libvirt https://tree.taiga.io/project/tripleo-ci-board/task/615?kanban-status=1447274 | 10:30 |
| quiquell | marios: we have to fix autohold, if you don't want to get the node destroy after it finish you have to run one command at your scheduler | 10:30 |
| marios | quiquell: ack | 10:30 |
| quiquell | marios: kind of, your task is about executing CI with dry_run | 10:31 |
| marios | quiquell: yeah but nodes still have to be created | 10:31 |
| marios | even if the tasks arent executed | 10:31 |
| marios | anyway maybe i'll run on a beaker box instead | 10:31 |
| quiquell | marios: you will have to put a review in top of https://review.rdoproject.org/r/#/c/18279/ | 10:31 |
| quiquell | marios: to launch dry_run, right now just run a "hello reproducer" playbook not toci | 10:31 |
| marios | quiquell: ack yeah thats the one i have in notes from our call | 10:31 |
| quiquell | marios: also you can execute the playbook towards the beaker from your laptop | 10:32 |
| quiquell | marios: works on remote too | 10:32 |
| quiquell | (or it was working) so you have your clones at your laptop | 10:32 |
| marios | nice | 10:32 |
| marios | ssbarnea|rover: harrassment has paid off \o/ | 10:33 |
| marios | https://review.openstack.org/#/c/628179/ https://review.openstack.org/#/c/628251 ssbarnea|rover | 10:33 |
| quiquell | sshnaidm: kforde has make public the cloudinit images :-) | 10:35 |
| quiquell | sshnaidm: also he will check if he can give us permissions to update those images from openstack-nodepool | 10:35 |
| sshnaidm | quiquell, that would be ideal | 10:36 |
| quiquell | sshnaidm: he is working on it :-) | 10:37 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 10:40 |
| *** holser_ has joined #oooq | 10:43 | |
| *** dsneddon has joined #oooq | 10:48 | |
| chandankumar | sshnaidm: stestr copy still not working http://logs.openstack.org/00/627500/69/check/tripleo-ci-centos-7-standalone-os-tempest/ba22a5d/logs/undercloud/var/log/tempest/stestr_results.html.gz | 10:55 |
| *** dsneddon has quit IRC | 10:55 | |
| *** udesale has quit IRC | 10:57 | |
| quiquell | marios: Something has change at tripleo-ci-centos-7-standalone job and we cannot run it from test1/test2 repro projects :-/ | 10:58 |
| chandankumar | sshnaidm: stestr_results.html file file generally getting dumped in /var/log/tempest/ for copying it to root log dir I think I am using the wrong path | 10:58 |
| chandankumar | ? | 10:58 |
| quiquell | marios: damn I think we don't have tripleo-ci-base at repro | 10:59 |
| quiquell | marios: something is no good at config | 10:59 |
| chandankumar | sshnaidm: or it is necessary to use gz? | 10:59 |
| sshnaidm | chandankumar, not sure I understand | 11:00 |
| quiquell | Damn we are missing this job at repro http://zuul.openstack.org/job/tripleo-ci-base | 11:02 |
| chandankumar | sshnaidm: I think i got the issue | 11:03 |
| *** dsneddon has joined #oooq | 11:04 | |
| *** jpena|brb is now known as jpena | 11:07 | |
| *** dsneddon has quit IRC | 11:09 | |
| *** dsneddon has joined #oooq | 11:20 | |
| quiquell | Found it | 11:25 |
| quiquell | sshnaidm: You have break reproducer !!! Unknown project git.openstack.org/openstack/openstack-virtual-baremetal | 11:25 |
| quiquell | :-) | 11:25 |
| quiquell | marios: that's why we need the dry run | 11:25 |
| sshnaidm | quiquell, o_O | 11:25 |
| sshnaidm | quiquell, how is it unknown? very well known :) | 11:26 |
| quiquell | sshnaidm: reproducer is the slow kid in the class | 11:26 |
| quiquell | classroom | 11:26 |
| chandankumar | quiquell: does os_tempest changes also going to break reproducer? | 11:26 |
| chandankumar | as other TOCI and TQE changes are in review | 11:27 |
| quiquell | chandankumar: is a new openstack project ? | 11:27 |
| quiquell | chandankumar: is so we have to add it | 11:27 |
| chandankumar | quiquell: it is validate-tempest replacement https://github.com/openstack/openstack-ansible-os_tempest | 11:28 |
| quiquell | chandankumar: before you merge the changes at tripleo jobs we have to change repro too | 11:28 |
| quiquell | chandankumar: same as https://review.rdoproject.org/r/#/c/18472/ | 11:28 |
| quiquell | sshnaidm: merge merge merge https://review.rdoproject.org/r/#/c/18472/ | 11:29 |
| chandankumar | quiquell: sure | 11:29 |
| *** dsneddon has quit IRC | 11:31 | |
| panda | quiquell: can you be mine ? | 11:31 |
| panda | quiquell: I will make you mine | 11:31 |
| quiquell | panda: free ? | 11:31 |
| panda | quiquell: so yesterday you sugegsted to launch playbook/pre.yaml | 11:32 |
| quiquell | panda: Did I break your laptop ? | 11:32 |
| panda | quiquell: it completed correctly, now running run.yaml, it's missing zuul variable | 11:32 |
| panda | quiquell: which would be inherited by the patch once rlandy's work is completed | 11:33 |
| panda | quiquell: but what about now ? | 11:33 |
| quiquell | panda: can I see your playbook it should not miss any | 11:33 |
| panda | quiquell: how do you usually run this thing now that we don't have the zuul so easily attached | 11:33 |
| quiquell | panda: what version or you using master ? | 11:33 |
| panda | quiquell: I'm using ansible-role-tripleo-ci-reproducer/playbooks/tripleo-ci-reproducer/run.yaml | 11:33 |
| panda | quiquell: master from yesterday | 11:34 |
| quiquell | panda: have a super silly playbook start.yaml with my gerrit user and stuff like that | 11:34 |
| panda | quiquell: oh, so you're cheating | 11:34 |
| quiquell | panda: we have to merge this https://review.rdoproject.org/r/#/c/18472/ | 11:34 |
| quiquell | panda: wee ned the new OVB project there | 11:34 |
| quiquell | panda: to have tripleo-ci-base jobs in place | 11:34 |
| panda | cat you share wyour dummy start.yaml ? | 11:34 |
| quiquell | panda: chiting ? | 11:34 |
| panda | quiquell: or maybe add it to /playbooks/examples/quickstart.yaml. So we can quickstart our quickstart | 11:35 |
| panda | yo dawg | 11:35 |
| quiquell | panda: https://paste.fedoraproject.org/paste/hJ44jo~6VdhiWRxhr1pzgg | 11:35 |
| quiquell | panda: I have differnt clones of the rol under different directorys | 11:36 |
| quiquell | panda: that why there is a master prefix in the include_role | 11:36 |
| quiquell | i have for example libvirt/launch_job so I only have to change the playbook to the correct role version | 11:36 |
| quiquell | panda: Can you workflow this ? https://review.rdoproject.org/r/#/c/18472/ | 11:37 |
| quiquell | panda: so we can run tripleo jobs | 11:37 |
| panda | quiquell: No, I want my +2V first ! | 11:38 |
| panda | I want to break things | 11:38 |
| quiquell | panda: cannot have it all | 11:38 |
| panda | quiquell: in your start playbook I still don't see zuul variables passed | 11:43 |
| quiquell | panda: they are in my playbook so I don' thave to pass them | 11:44 |
| quiquell | all is there | 11:44 |
| panda | quiquell: also in playbooks/tripleo-ci-reproducer/pre.yaml I had to change ansible_user to ansible_user_id in groups | 11:44 |
| sshnaidm | marios, panda let's merge it https://review.openstack.org/#/c/632495/ | 11:45 |
| quiquell | panda: Do a review with it, if CI passes we are good | 11:45 |
| quiquell | sshnaidm: +2 | 11:49 |
| quiquell | panda: also https://review.rdoproject.org/r/18473 | 11:50 |
| marios | sshnaidm: looking | 12:00 |
| marios | sshnaidm: so is a typo (missing workspace) | 12:00 |
| *** dsneddon has joined #oooq | 12:03 | |
| *** skramaja has quit IRC | 12:06 | |
| *** ratailor has quit IRC | 12:08 | |
| honza | panda: could you help me debug this job failure? it's the new selenium stuff for tripleo-ui https://review.openstack.org/#/c/612756/ It seems to always error out with "write failed: standard output: Broken pipe" | 12:10 |
| *** dsneddon has quit IRC | 12:11 | |
| panda | honza: that error is a generic warning after the log collection | 12:12 |
| panda | honza: your actual error is here http://logs.openstack.org/56/612756/5/check/tripleo-ci-centos-7-undercloud-selenium/b48f1b2/job-output.txt.gz#_2019-01-23_11_11_48_725083 | 12:12 |
| honza | panda: *facepalm* | 12:13 |
| panda | honza: seems to be a problem building the package based on this change. | 12:13 |
| honza | panda: thanks, much appreciated | 12:13 |
| *** dsneddon has joined #oooq | 12:23 | |
| panda | honza: you're good from here or you need additional help ? | 12:24 |
| honza | panda: i think i'm fine for now. this helped me discover that my distgit patch never merged so i'm looking into that now. thanks again | 12:24 |
| panda | honza: cool | 12:24 |
| *** dsneddon has quit IRC | 12:28 | |
| panda | [panda@localhost ~]$ openstack-3 --os-cloud rdo-cloud image list | 12:29 |
| panda | Exception raised: cannot import name 'image_signer' | 12:29 |
| panda | hmpf | 12:29 |
| panda | ykarel: delorean packages are more in sync, but not free from bugs :) | 12:30 |
| panda | I need something stable, I'm already working with something too unstable as the reproducer. | 12:30 |
| *** panda is now known as panda|launch | 12:32 | |
| ykarel | panda should not happen, what's version of ur openstackclient package, rpm -q python3-openstackclient | 12:32 |
| ykarel | i think it's side effect of what u installed earlier | 12:33 |
| *** jpena is now known as jpena|lunch | 12:33 | |
| ykarel | dependencies not updated | 12:33 |
| ykarel | also check rpm -q python3-openstacksdk | 12:34 |
| ykarel | i think ^^ is installed from fedora(not delorean repo) | 12:34 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, (1 more message) | 12:40 |
| chandankumar | sshnaidm: now the changes are working on os_Tempest | 12:42 |
| chandankumar | sshnaidm: http://logs.openstack.org/00/627500/70/check/tripleo-ci-centos-7-standalone-os-tempest/3230d14/logs/ | 12:42 |
| chandankumar | sshnaidm: http://logs.openstack.org/00/627500/70/check/tripleo-ci-centos-7-standalone-os-tempest/3230d14/logs/stestr_results.html stestr is in root logs | 12:42 |
| sshnaidm | chandankumar, and stackviz works too as I see | 12:43 |
| chandankumar | sshnaidm: yup | 12:43 |
| sshnaidm | chandankumar, cool | 12:43 |
| *** dsneddon has joined #oooq | 12:44 | |
| sshnaidm | chandankumar, you have a zuul error in patch | 12:44 |
| chandankumar | sshnaidm: where? | 12:46 |
| chandankumar | sshnaidm: you mean rdo third party jobs? | 12:47 |
| sshnaidm | chandankumar, look at the patch itself in gerrit - 627500 | 12:47 |
| *** weshay_PTO is now known as weshay | 12:48 | |
| *** dsneddon has quit IRC | 12:48 | |
| chandankumar | sshnaidm: I think that was on patch 27 | 12:49 |
| sshnaidm | chandankumar, ack | 12:49 |
| chandankumar | sshnaidm: current patch set is 70 | 12:49 |
| weshay | arxcruz|ruck, ssbarnea|rover like a blood red mooon.. . the status board a few times a year goes "ALL GREEN" http://rhos-release.virt.bos.redhat.com:3030/rhosp :) | 12:50 |
| sshnaidm | chandankumar, and why RDO 3party gives a zuul error? | 12:50 |
| sshnaidm | quiquell, look at that: http://paste.openstack.org/show/743170/ | 12:50 |
| weshay | at least on our side | 12:50 |
| arxcruz|ruck | weshay: i saw, was talking with panda|launch | 12:50 |
| arxcruz|ruck | he told me to take a selfie with it | 12:50 |
| sshnaidm | quiquell, Collecting job variants for tripleo-ci-base - No matching parents for job tripleo-ci-base | 12:50 |
| sshnaidm | quiquell, how is that.. we have it in tripleo-ci | 12:50 |
| quiquell | sshnaidm: refresh, this is the ovb project not bein at main.yaml | 12:51 |
| quiquell | sshnaidm: merged into the role | 12:51 |
| chandankumar | sshnaidm: Unknown project git.openstack.org/openstack/openstack-ansible-os_tempest | 12:51 |
| quiquell | sshnaidm: all the openstack projects that are required by our jobs nee dto be at main.yaml | 12:51 |
| quiquell | sshnaidm: Already updated that | 12:51 |
| panda|launch | 3 | 12:52 |
| panda|launch | 2 | 12:52 |
| panda|launch | 1 | 12:52 |
| sshnaidm | quiquell, yeah, will update, but not sure it's related. It happens when I copy everything from config/ rdo repo to zuul-config | 12:52 |
| panda|launch | ignition | 12:53 |
| *** panda|launch is now known as panda | 12:53 | |
| marios | is there a reason why we don't do this already https://review.openstack.org/#/q/topic:standalone_scenario_tqe (standalone scenario jobs for the standalone role) | 12:53 |
| panda | I was launched | 12:53 |
| marios | space control to major tom | 12:53 |
| quiquell | sshnaidm: If you do a docker-compose web -f and then try to access the tripleo-ci-base it will show you the issues job config have | 12:54 |
| marios | panda: see https://review.openstack.org/#/q/topic:standalone_scenario_tqe once you hit safe orbit thanks | 12:54 |
| panda | ahah | 12:54 |
| quiquell | sshnaidm: to have a clear new lines you can s/\/n\/b/\r/g | 12:54 |
| sshnaidm | quiquell, logs -f you mean? | 12:54 |
| quiquell | sshnaidm: yep | 12:55 |
| sshnaidm | quiquell, ok | 12:55 |
| quiquell | sshnaidm: they are asking about the userdat at #zuul | 12:56 |
| arxcruz|ruck | weshay: will you present it or shall I? | 12:57 |
| ykarel | weshay, panda can u revisit https://review.rdoproject.org/r/#/c/18151 | 12:57 |
| weshay | arxcruz|ruck, by all means... if you want to | 12:57 |
| arxcruz|ruck | up to you, i'm already in the meeting | 12:57 |
| weshay | go ahead | 12:58 |
| arxcruz|ruck | k | 12:58 |
| weshay | I'm trolling mojo for visa content | 12:58 |
| marios | weshay: merged https://docs.openstack.org/tripleo-docs/latest/contributor/ci_primer.html thanks quiquell sshnaidm | 13:02 |
| marios | ssbarnea|rover: nice thanks https://review.openstack.org/#/c/632695/1/doc/source/contributor/ci_primer.rst (did you setup the keyword already?_ | 13:08 |
| *** ykarel is now known as ykarel|away | 13:10 | |
| marios | @oooq test | 13:10 |
| *** trown|outtypewww is now known as trown | 13:14 | |
| *** dsneddon has joined #oooq | 13:17 | |
| weshay | arxcruz|ruck, ssbarnea|rover https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/9d62fa9/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-01-23_07_19_39 | 13:24 |
| arxcruz|ruck | weshay: there's a bug opened for it, emilien is talking about it on #tripleo | 13:25 |
| weshay | ah cool | 13:25 |
| quiquell | marios: what's that @oooq test ? | 13:26 |
| quiquell | marios: the notify stuff ? | 13:26 |
| quiquell | Damn someone can help me with this ? http://logs.rdoproject.org/79/18279/33/check/tripleo-ci-reproducer-centos-7-host/f7957af/job-output.txt.gz | 13:26 |
| quiquell | TL;DR git: 'interpret-trailers' is not a git command. See 'git --help'. | 13:26 |
| marios | quiquell: yeah ssbarnea|rover was going to set it up and he wrote it in https://review.openstack.org/#/c/632695/1/doc/source/contributor/ci_primer.rst | 13:26 |
| marios | quiquell: selinux? | 13:27 |
| marios | quiquell: do we have to setenforce 0 :/ | 13:27 |
| quiquell | marios: it's a centos7 | 13:27 |
| *** rlandy has joined #oooq | 13:27 | |
| quiquell | rlandy: o/ | 13:27 |
| marios | quiquell: just cos i see 2019-01-23 13:15:00.991962 | primary | File created and ownership, perms or SE linux context changed | 13:28 |
| marios | 2019-01-23 13:15:01.477713 | primary | fatal: [localhost]: FAILED! => { | 13:28 |
| *** dsneddon has quit IRC | 13:28 | |
| rlandy | quiquell: hey - got as far as deploying my custom zuul.yaml file | 13:29 |
| rlandy | had to edit the script | 13:29 |
| quiquell | rlandy: OVB project was missing at zuul configuration so jobs cannot start | 13:29 |
| quiquell | rlandy: aso ssh key issue is "fixed" and CI is working now | 13:29 |
| rlandy | quiquell, can't include the install of the repo in the playbook itslef | 13:30 |
| quiquell | rlandy: but there is a bug at paramiko an zuul suffer from it | 13:30 |
| rlandy | quiquell; nice ... so ... | 13:30 |
| quiquell | rlandy: how so ? | 13:30 |
| quiquell | rlandy: so for centain type of key they have to create new one | 13:30 |
| rlandy | quiquell; about half the time 'Wait for zuul tenant' fails | 13:30 |
| rlandy | nice - that ci is working again | 13:31 |
| quiquell | rlandy: TL;DR if your ki has "OPENSSH" delimiters you need a new one | 13:31 |
| quiquell | rlandy: yep you have to increase it | 13:31 |
| rlandy | quiquell: it's at 60 | 13:31 |
| quiquell | rlandy: it depends of the env at RDO it just news 15 retries | 13:31 |
| rlandy | quiquell: ki? | 13:31 |
| quiquell | s/ki/key/ | 13:31 |
| quiquell | quiquell: we have to make the retries configurable | 13:32 |
| quiquell | Talking with myself | 13:32 |
| quiquell | rlandy: make it 100 | 13:32 |
| rlandy | I'll try again this morning | 13:32 |
| rlandy | yesterday it worked about 3times and failed the rest | 13:32 |
| rlandy | I wasn't working with libvirt | 13:32 |
| rlandy | quiquell: we will have to rethink the clone though | 13:33 |
| rlandy | can't have it in the same playbook | 13:33 |
| ssbarnea|rover | marios: there is nothing to setup for the keyword, each team member is supposed to add it to his irc client (once we agree on a keyword) | 13:33 |
| quiquell | rlandy: I was expecting it | 13:33 |
| weshay | arxcruz|ruck, join my blue.. in 5min please | 13:33 |
| quiquell | rlandy: it have to be pre-loaded | 13:33 |
| arxcruz|ruck | weshay: ack | 13:33 |
| quiquell | rlandy: btw launch job not working at centos7 git: 'interpret-trailers' is not a git command. See 'git --help'. | 13:33 |
| *** jpena|lunch is now known as jpena | 13:34 | |
| rlandy | quiquell: that started yesterday | 13:34 |
| marios | ssbarnea|rover: oh i see | 13:34 |
| marios | ssbarnea|rover: k thanks | 13:34 |
| rlandy | quiquell, https://paste.fedoraproject.org/paste/ZLWscgqAvxzQot2RjeaiYQ is the playbook that works for me | 13:34 |
| rlandy | I need the two separate host pieces | 13:35 |
| rlandy | http://10.10.120.171:8080/c/test1/+/1061/1/zuul.yaml | 13:35 |
| rlandy | quiquell: martin posted out about that error | 13:36 |
| rlandy | quiquell: you having a better day though? | 13:36 |
| sshnaidm | quiquell, posted to https://tree.taiga.io/project/tripleo-ci-board/task/570 about images preparing | 13:37 |
| *** ykarel|away has quit IRC | 13:38 | |
| quiquell | rlandy: yep totally all the issues are "fixed" but we have to be prepare for the key issue and help zuul guys document it | 13:38 |
| arxcruz|ruck | weshay: waiting on your bj | 13:39 |
| quiquell | sshnaidm: Thanks!, yep taks is the place | 13:39 |
| quiquell | rlandy: hecking playbook | 13:39 |
| *** agopi has quit IRC | 13:39 | |
| quiquell | rlandy: "localhost" is not needed at ansible-playbook command it's already specify at "hosts: localhost" | 13:40 |
| sshnaidm | @oooq bla | 13:41 |
| sshnaidm | ssbarnea|rover, doesn't work? ^ | 13:41 |
| panda | @oooq-core | 13:41 |
| panda | @oooq-cores ? | 13:41 |
| rlandy | quiquell: I need it on my system | 13:41 |
| ssbarnea|rover | anything works as long we agree on it, i happen to already have @oooq in my "highlight" filter. | 13:41 |
| * panda shrugs | 13:41 | |
| rlandy | because of the way my keys are set up | 13:42 |
| rlandy | ssh to 127.0.0.2 | 13:42 |
| weshay | arxcruz|ruck, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master/a51ada9/logs/undercloud/home/zuul/tempest.log.txt.gz | 13:42 |
| quiquell | rlandy: playbook is awesome very clear | 13:42 |
| quiquell | rlandy: do we have two add-host ? | 13:42 |
| quiquell | rlandy: s/have/need/ | 13:42 |
| rlandy | quiquell: will have to move the clone to the instructions | 13:42 |
| weshay | ssbarnea|rover, hey.. join my blue | 13:42 |
| weshay | for a minute | 13:43 |
| rlandy | if there is no repo,it complains | 13:43 |
| quiquell | rlandy: so we cannot clone tq there and get the inject the roles at ansible env | 13:43 |
| rlandy | quiquell: we will cone tq in the libvirt task | 13:43 |
| rlandy | we can't clone ansible-role-tripleo-ci-reproducer | 13:43 |
| rlandy | it looks for the path because of the include playbook | 13:44 |
| rlandy | not really a big deal | 13:44 |
| * rlandy tries to run again | 13:44 | |
| rlandy | quiquell: I will not leave the two add_hosts | 13:44 |
| rlandy | it's just for my test box setup | 13:44 |
| quiquell | rlandy maybe we can clone it at another play | 13:45 |
| quiquell | rlandy: same playbook new play | 13:45 |
| chandankumar | weshay: arxcruz|ruck https://review.openstack.org/#/c/628415/ | 13:45 |
| rlandy | quiquell: I'll work some thingout | 13:45 |
| rlandy | we can have two playbooks | 13:45 |
| quiquell | rlandy: sure will try it too | 13:45 |
| rlandy | quiquell: but otherwise I have a working system | 13:46 |
| rlandy | quiquell: one question though ... | 13:46 |
| rlandy | quiquell: what do we consider setup? | 13:46 |
| quiquell | rlandy: update your reviews to be able to run a tripleo job | 13:46 |
| marios | panda: o/ replied & fixed the order for https://review.rdoproject.org/r/#/c/18093/13 && see related https://review.openstack.org/632723 | 13:46 |
| quiquell | rlandy: what do you mean ? | 13:46 |
| rlandy | quiquell: ie: where do we get people to create a private network in their tenant? | 13:47 |
| rlandy | add keys to the images? | 13:47 |
| rlandy | quiquell: aldo - what are the latest images you are working with? | 13:47 |
| rlandy | also | 13:47 |
| rlandy | I still have my old ones | 13:47 |
| quiquell | rlandy: about images read comments here https://tree.taiga.io/project/tripleo-ci-board/task/570 | 13:48 |
| rlandy | quiquell: and network? | 13:48 |
| quiquell | rlandy: I can share with your tenant the ones with team pub keys | 13:48 |
| rlandy | quiquell: mine work fine as of now | 13:48 |
| quiquell | rlandy: can we create the network at pre.yaml if it does not exists with ansible os_network modules ? | 13:48 |
| rlandy | quiquell: ack - I'll add that | 13:48 |
| sshnaidm | rlandy, marios panda arxcruz|ruck ssbarnea|rover weshay chandankumar kopecmartin if you have tenant on rdo cloud please send me your tenant ID (OS_TENANT_ID) | 13:48 |
| panda | marios: wasn't a hard requirement, sometimes I just comment to make people know. | 13:48 |
| rlandy | quiquell: k - will update my reviews after this test | 13:49 |
| quiquell | rlandy: well don't know if at pre.yaml or at start.yaml we have a place that check for the network | 13:49 |
| rlandy | marios: are you all set with merging your reviews? | 13:49 |
| arxcruz|ruck | chandankumar: which job is running the os_tempest on this review ? | 13:49 |
| panda | marios: ah read yyour comment | 13:49 |
| rlandy | quiquell: I'll try it out and see what works best | 13:49 |
| marios | panda: well it is a correct observation. yes the order does matter. perhaps we intentionally want the low mem last. we'll find out with https://review.openstack.org/#/c/632723/ and if so we'll flip them .but lets make it consistent | 13:49 |
| panda | marios: thanks | 13:49 |
| quiquell | rlandy: sure sure | 13:49 |
| rlandy | I think the provider task is best | 13:49 |
| quiquell | rlandy: humm that's right the provider tasks | 13:50 |
| marios | rlandy: updated this one please if you have 2 mins https://review.rdoproject.org/r/#/c/18093/13 | 13:50 |
| * rlandy looks | 13:50 | |
| chandankumar | arxcruz|ruck: http://logs.openstack.org/00/627500/70/check/tripleo-ci-centos-7-standalone-os-tempest/3230d14/ | 13:50 |
| quiquell | rlandy: last one, did you say that martin posted about the git error ? | 13:51 |
| marios | rlandy: then we can land the layout once the jobs merge https://review.rdoproject.org/r/18454 | 13:51 |
| rlandy | quiquell: yes - let me look for the email - sec | 13:51 |
| panda | sshnaidm: here, email, etherpad, shared CIFS folder ? | 13:52 |
| sshnaidm | panda, pigeon post of course | 13:53 |
| rlandy | quiquell: email forwarded | 13:53 |
| panda | sshnaidm: damn, I'm short on pidgeon. I'll check the supermarket. | 13:53 |
| sshnaidm | panda, or to private msg if pigeons are eaten | 13:53 |
| arxcruz|ruck | @oooq blah blah blah | 13:56 |
| *** dsneddon has joined #oooq | 13:56 | |
| sshnaidm | arxcruz|ruck, nah, doesn't work | 13:56 |
| sshnaidm | I think ssbarnea|rover has fun from our tries :D | 13:56 |
| *** vinaykns has joined #oooq | 13:58 | |
| *** holser_ has quit IRC | 13:59 | |
| *** holser_ has joined #oooq | 13:59 | |
| arxcruz|ruck | weshay: http://logs.openstack.org/00/627500/70/check/tripleo-ci-centos-7-standalone-os-tempest/3230d14/ | 13:59 |
| quiquell | rlandy: thanks that's the issue | 14:00 |
| *** dsneddon has quit IRC | 14:00 | |
| arxcruz|ruck | weshay: http://logs.openstack.org/00/627500/70/check/tripleo-ci-centos-7-standalone-os-tempest/3230d14/job-output.txt.gz#_2019-01-23_12_02_08_313667 | 14:02 |
| *** panda is now known as panda|brb | 14:03 | |
| weshay | panda|brb, 1-1 | 14:03 |
| sshnaidm | quiquell, seems like there is a progress with ovb \o/ | 14:06 |
| quiquell | sshnaidm: \o/, as always bad day|good day | 14:06 |
| rlandy | marios: one question ... why the captial "F" false for the tempest options? | 14:07 |
| vinaykns | hello channel I have a question | 14:07 |
| *** panda|brb is now known as panda | 14:08 | |
| vinaykns | I'm trying to deploy oooq...but somehow its getting failed due to http://pastebin.test.redhat.com/700737 | 14:08 |
| vinaykns | any help would be appreciated.! | 14:09 |
| marios | rlandy: copy/paste or nit, checking | 14:09 |
| marios | rlandy: likely is the same in the upstream definition i just copied it | 14:09 |
| *** agopi has joined #oooq | 14:09 | |
| weshay | panda, lost ya | 14:09 |
| *** ykarel|away has joined #oooq | 14:10 | |
| marios | rlandy: https://github.com/openstack-infra/tripleo-ci/blob/715794adceeaccd2c94e137b0fb62694e9408ef6/zuul.d/standalone-jobs.yaml#L425 | 14:10 |
| rlandy | ok - git it | 14:10 |
| rlandy | got | 14:10 |
| rlandy | marios: https://review.rdoproject.org/r/#/c/18093/ w+'ed | 14:11 |
| marios | rlandy: thanks | 14:11 |
| sshnaidm | vinaykns, seems like libvirt misconfiguration | 14:11 |
| rlandy | looking at layout | 14:11 |
| marios | rlandy: thanks will have to wait for jobs to merge and recheck the layout | 14:12 |
| sshnaidm | vinaykns, test some virsh commands as a user on libvirt host | 14:12 |
| rlandy | marios: yep - will just give it a basic review | 14:12 |
| rlandy | won't merge | 14:12 |
| marios | rlandy: thank you | 14:12 |
| rlandy | not without alex's approval | 14:12 |
| vinaykns | sshnaidm: I could see error : virNetSocketReadWire:1806 : End of file while reading data: Input/output error as libvirts status | 14:12 |
| sshnaidm | vinaykns, authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found) | 14:13 |
| sshnaidm | vinaykns, the script can't connect to libvirt daemon | 14:13 |
| vinaykns | sshnaidm: I think it's because the script is looking for some sasl authentication parameters. | 14:25 |
| vinaykns | sshnaidm: in this case it is unable to find it | 14:26 |
| *** dsneddon has joined #oooq | 14:30 | |
| *** quiquell is now known as quiquell|lunch | 14:33 | |
| *** ykarel|away is now known as ykarel | 14:34 | |
| *** dsneddon has quit IRC | 14:35 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, (1 more message) | 14:40 |
| rlandy | marios: https://review.rdoproject.org/r/#/c/18454 lgtm. I rechecked it - zuul +1'ed | 14:40 |
| marios | rlandy: cool thanks. so we have to merge it to see the jobs run then ? | 14:41 |
| rlandy | marios: no - we can set up a testproject run | 14:41 |
| chandankumar | sshnaidm: quiquell|lunch https://review.rdoproject.org/r/#/c/18480/ | 14:42 |
| chandankumar | for os_tempest | 14:42 |
| marios | rlandy: otherwise won't they run with the rest of the periodics? | 14:42 |
| rlandy | marios: you can add the jobs to check in zuul.yaml and use this review as a depends-on: https://review.openstack.org/#/c/625271/ | 14:43 |
| weshay | panda, this is done right? https://tree.taiga.io/project/tripleo-ci-board/task/484?kanban-status=1447276 | 14:43 |
| rlandy | marios: yep - merging it will run with periodics | 14:43 |
| rlandy | but if you just want to see a job run testproject can do that | 14:43 |
| rlandy | not sure of I am answering the right question there | 14:43 |
| ssbarnea|rover | weshay: rlandy what was the link for testing the container building on f28? | 14:45 |
| weshay | ssbarnea|rover, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic | 14:45 |
| marios | rlandy: yes thanks | 14:48 |
| marios | rlandy: so we have already seen them run on each of the job reviews but we can also do another run here before merge i'll do that so we can be sure | 14:48 |
| rlandy | I think I must have wrong key | 14:53 |
| rlandy | getting latest | 14:53 |
| rlandy | still wont connect | 14:53 |
| rlandy | panda: do you need anything else merged atm for f28? | 14:53 |
| weshay | rlandy, sshnaidm panda let's move forward and merge e rest of the periodics? | 14:54 |
| weshay | <rlandy> marios: you can add the jobs to check i | 14:54 |
| weshay | oops | 14:54 |
| weshay | what the heck | 14:54 |
| weshay | rlandy, sshnaidm panda sorry.. let's move forward on https://review.rdoproject.org/r/#/c/17740 | 14:55 |
| rlandy | weshay, that need rebase - sec resubmitting | 14:59 |
| sshnaidm | quiquell|lunch, ok, do nodepoll patch is merged: https://review.openstack.org/#/c/630649/ | 15:02 |
| sshnaidm | quiquell|lunch, need to rebuild the nodepool image | 15:02 |
| sshnaidm | docker image I mean | 15:02 |
| rlandy | https://review.rdoproject.org/r/#/c/17740 rebased | 15:03 |
| *** quiquell|lunch is now known as quiquell | 15:03 | |
| quiquell | sshnaidm: \o/ !!! | 15:04 |
| *** dsneddon has joined #oooq | 15:04 | |
| quiquell | sshnaidm: about docker image, maybe we can create semver for us | 15:04 |
| rlandy | sshnaidm: pls let me know which is the latest image I should use | 15:04 |
| quiquell | sshnaidm: instead of replacing stable | 15:04 |
| quiquell | sshnaidm: so we test at review with change on docker tag | 15:04 |
| sshnaidm | quiquell, semver? | 15:05 |
| quiquell | sshnaidm: semantic verssion, or add the commit id | 15:05 |
| quiquell | sshnaidm: from nodepool | 15:05 |
| sshnaidm | quiquell, I see, yeah, but still need a "latest" or "stable" to use in compose | 15:06 |
| sshnaidm | quiquell, it's actually the task about management of these images | 15:06 |
| quiquell | sshnaidm: and when review passes we update stable or something like that, at the end is a promotion of docker images | 15:06 |
| quiquell | sshnaidm: yep | 15:06 |
| sshnaidm | quiquell, well, for that I think we need a more full CI - including job runss | 15:07 |
| quiquell | sshnaidm: we don't know if updating nodepool will need updating the other zuul containers | 15:07 |
| quiquell | sshnaidm: let's just tag everyting with commit-id | 15:07 |
| quiquell | sshnaidm: and use it a docker-compose | 15:07 |
| quiquell | instead of stable | 15:07 |
| quiquell | well we have to discuss it | 15:08 |
| sshnaidm | quiquell, yeah | 15:08 |
| sshnaidm | quiquell, not all maybe, I'd leave gerrit as is, as latest version was broken | 15:08 |
| sshnaidm | quiquell, well, I think it's worth to update all our containers now - zuul and nodepool, and to test if all is ok | 15:09 |
| quiquell | sshnaidm: but we have to be able to go back if they are not ok, so tag them | 15:09 |
| quiquell | sshnaidm: tag the actual ones another tag apart from stable so we can rollback | 15:10 |
| sshnaidm | quiquell, to tag previous ones you mean? | 15:10 |
| quiquell | sshnaidm: btw, docker-compose logs splited https://review.rdoproject.org/r/#/c/18473/ | 15:10 |
| quiquell | sshnaidm: actual docker images before update them | 15:10 |
| quiquell | sshnaidm: so you can go back if something is no good | 15:10 |
| quiquell | sshnaidm: It's better to have pining than stable at docker-compsoe | 15:10 |
| sshnaidm | quiquell, we have them on docker.io | 15:10 |
| quiquell | sshnaidm: yep | 15:11 |
| sshnaidm | so I can always pull from there | 15:11 |
| quiquell | sshnaidm: but can you go back to previous one without tag ? | 15:11 |
| quiquell | sshnaidm: easier if there is a tag | 15:11 |
| sshnaidm | quiquell, yeah, anyway will tag them before pushing | 15:11 |
| *** dsneddon has quit IRC | 15:12 | |
| sshnaidm | quiquell, nice | 15:14 |
| sshnaidm | quiquell, btw did you see: nodepool.driver.static.provider.StaticNodeError: 127.0.0.1:22: ConnectionTimeoutException | 15:14 |
| sshnaidm | quiquell, shouldn't it be an external IP? | 15:15 |
| quiquell | weshay: workflow the log spliting please | 15:15 |
| quiquell | sshnaidm: I mean | 15:15 |
| quiquell | weshay: well you can workflow it too | 15:15 |
| quiquell | sshnaidm: nope | 15:15 |
| weshay | quiquell, which review? | 15:15 |
| quiquell | sshnaidm: libvirt ? | 15:15 |
| quiquell | weshay: https://review.rdoproject.org/r/#/c/18473/ | 15:15 |
| weshay | ah.. I like that | 15:16 |
| weshay | ++ | 15:16 |
| sshnaidm | quiquell, I mean, 127.0.0.1 is address inside the container | 15:16 |
| sshnaidm | quiquell, it's not hosts 127.0.0.1, right? | 15:16 |
| sshnaidm | quiquell, that's why it can't connect | 15:17 |
| quiquell | sshnaidm: for nodepool_provider host it's that | 15:17 |
| quiquell | sshnaidm: for libvirt it's no good if it's like that | 15:17 |
| quiquell | sshnaidm: what review ? | 15:17 |
| rlandy | weshay: panda: want me to merge this? https://review.rdoproject.org/r/#/c/17740/? | 15:17 |
| sshnaidm | quiquell, https://logs.rdoproject.org/73/18473/7/check/tripleo-ci-reproducer-fedora-28/c122041/tripleo-ci-reproducer/launcher.log | 15:17 |
| rlandy | if so, pls +1 | 15:18 |
| sshnaidm | quiquell, just log from your logs patch | 15:18 |
| sshnaidm | quiquell, https://logs.rdoproject.org/73/18473/7/check/tripleo-ci-reproducer-fedora-28/c122041/tripleo-ci-reproducer/etc_nodepool/nodepool.yaml | 15:18 |
| sshnaidm | quiquell, there is 127.0.0.1 ^^ | 15:19 |
| sshnaidm | quiquell, if you don't use "net=host" with docker it's localhost of container itself, not the host | 15:20 |
| *** dsneddon has joined #oooq | 15:22 | |
| quiquell | so 127.0.0.1 is running at the docker image not at the host ? | 15:22 |
| *** udesale has joined #oooq | 15:22 | |
| quiquell | sshnaidm: was passing though :-) | 15:23 |
| quiquell | sshnaidm: that's good news maybe we want that | 15:23 |
| quiquell | sshnaidm: but with docker node :-) | 15:23 |
| quiquell | sshnaidm: so we don't affect host | 15:23 |
| sshnaidm | quiquell, yep, launcher doesn't connect to it actually | 15:23 |
| quiquell | sshnaidm: when we run somet hing | 15:23 |
| sshnaidm | quiquell, why not? it's CI, what can we break there :) | 15:23 |
| quiquell | sshnaidm: nottting | 15:24 |
| quiquell | sshnaidm: will take a look a launch_job review, we actualy launch a job there, so it's running inside the docker image ? | 15:24 |
| sshnaidm | quiquell, idk, need to see | 15:24 |
| *** saneax has quit IRC | 15:25 | |
| quiquell | sshnaidm: well I was trying to run a job at the docker image itself X-D http://logs.rdoproject.org/79/18279/29/check/tripleo-ci-reproducer-centos-7-host/81f6487/tripleo-ci-reproducer/etc_nodepool/nodepool.yaml | 15:26 |
| quiquell | sshnaidm: not so bad :-) | 15:26 |
| *** dsneddon has quit IRC | 15:26 | |
| quiquell | sshnaidm: can you comment in the review ? | 15:26 |
| sshnaidm | quiquell, did it start? :) | 15:26 |
| quiquell | sshnaidm: yep | 15:26 |
| sshnaidm | quiquell, what is the review? | 15:26 |
| quiquell | sshnaidm: and go very far | 15:26 |
| sshnaidm | quiquell, well, why not, it's the same centos | 15:27 |
| quiquell | sshnaidm: well is not a centos is a alpine | 15:27 |
| quiquell | sshnaidm: https://review.rdoproject.org/r/#/c/18279/ | 15:27 |
| sshnaidm | quiquell, hmm.. | 15:27 |
| quiquell | sshnaidm: but I was thinking on something like that to do quick test with molecule | 15:27 |
| quiquell | sshnaidm: put a docker node with toci dryrun | 15:27 |
| quiquell | rlandy, sshnaidm: Also to be able to stream job console logs from command line https://review.rdoproject.org/r/#/c/18475/ | 15:29 |
| quiquell | it start the fingergw zuul service | 15:29 |
| quiquell | ok leave now read you tomorrow | 15:30 |
| *** quiquell is now known as quiquell|off | 15:30 | |
| *** dsneddon has joined #oooq | 15:34 | |
| rlandy | quiquell|off: thanks for rebasing those | 15:37 |
| weshay | if you haven't seen http://rhos-release.virt.bos.redhat.com:3030/rhosp take a look :) | 15:40 |
| *** dsneddon has quit IRC | 15:40 | |
| *** dsneddon has joined #oooq | 15:42 | |
| rlandy | weshay: ^^ it's a beautiful thing | 15:44 |
| rlandy | arxcruz|ruck++ | 15:44 |
| hubbot1 | rlandy: arxcruz|ruck's karma is now 5 | 15:44 |
| rlandy | ssbarnea|rover++ | 15:44 |
| hubbot1 | rlandy: ssbarnea|rover's karma is now 3 | 15:44 |
| arxcruz|ruck | not my fault! | 15:44 |
| rlandy | sshnaidm: can you get the zuul tenant to start? | 15:45 |
| *** ykarel is now known as ykarel|away | 15:45 | |
| sshnaidm | rlandy, usually yes | 15:45 |
| sshnaidm | rlandy, doesn't it work for you? | 15:46 |
| rlandy | actually latest just did start | 15:46 |
| rlandy | better than yesterday | 15:46 |
| ykarel|away | weshay, can u revisit again when u get chance:- https://review.rdoproject.org/r/#/c/18151/ | 15:47 |
| rlandy | my job is queued ... tripleo-ci-centos-7-multinode-1ctlr-featureset010-dlrn-hash-tag | 15:47 |
| *** dsneddon has quit IRC | 15:48 | |
| sshnaidm | rlandy, cool | 15:48 |
| rlandy | will have to check why it's not started | 15:49 |
| rlandy | oh it just did | 15:49 |
| rlandy | ha - job reproducer | 15:49 |
| *** dsneddon has joined #oooq | 15:52 | |
| rlandy | OMG - running reproducer | 15:52 |
| panda | rlandy: quick, catch him! | 15:53 |
| rlandy | from generated zuul.yaml file | 15:53 |
| panda | don't let him go! | 15:53 |
| rlandy | panda: nah - let him run | 15:53 |
| rlandy | needs the exercise after having been stuck all week | 15:53 |
| marios | i think its almost cardioclock here too | 15:58 |
| weshay | rlandy, nice paste the log you are using | 16:02 |
| sshnaidm | weshay, is it a hubbot1 that recheck this patch? https://review.openstack.org/#/c/564291/ | 16:02 |
| weshay | sshnaidm, aye | 16:03 |
| rlandy | rebuilding review | 16:03 |
| sshnaidm | weshay, panda where do you want reproducing ovb to be tracked - in ovb US or reproducer US? | 16:05 |
| weshay | reproducer | 16:07 |
| *** udesale has quit IRC | 16:10 | |
| rlandy | ha - dlrn hashes match | 16:15 |
| rlandy | yay | 16:15 |
| *** bogdando has quit IRC | 16:15 | |
| weshay | is quique still here? | 16:27 |
| weshay | nope | 16:27 |
| *** dsneddon has quit IRC | 16:29 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, (1 more message) | 16:40 |
| sshnaidm | rlandy, ovb is reproducing! \o/ | 16:44 |
| sshnaidm | rlandy, take a look https://review.rdoproject.org/r/#/c/18422 | 16:44 |
| rlandy | sshnaidm: yay!! | 16:44 |
| rlandy | we're getting there | 16:44 |
| sshnaidm | *phew* | 16:44 |
| sshnaidm | on this happy note | 16:46 |
| *** sshnaidm is now known as sshnaidm|afk | 16:46 | |
| chandankumar | weshay: do we want to run os_tempest job against all standalone job template? | 16:57 |
| weshay | chandankumar, lets let it run in a job until the end of the sprint, next sprint we'll add a task to move to all jobs | 16:59 |
| chandankumar | weshay: sure :-) | 16:59 |
| weshay | chandankumar, what we are saying I think is that the mvp is nearly done.. complete mvp.. then move to all jobs | 16:59 |
| chandankumar | weshay: currently 4 os_tempest patches blocked as one of the scenario tests are failing | 17:00 |
| chandankumar | weshay: I am stilling to reproduce that error | 17:00 |
| chandankumar | using OSA | 17:00 |
| weshay | ah.. k | 17:00 |
| weshay | thanks | 17:00 |
| chandankumar | weshay: we will talk in detail tomorrow during 1:1 apevec might miss it | 17:01 |
| weshay | ya.. I saw.. thanks | 17:01 |
| * chandankumar hides now | 17:01 | |
| ssbarnea|rover | @oooq: made ugly discovery about shell module: using env vars works only if you export them. shell: "FOO=bar; echo $FOO" does not work. we have bugs like this in many places. | 17:04 |
| weshay | ugh | 17:04 |
| ssbarnea|rover | newer linter highlighted it, i tested it and it is true. for me is something new.... | 17:04 |
| ssbarnea|rover | i will back with fixes but keep in mind. | 17:05 |
| ssbarnea|rover | i still have no idea why it does not work because I was expecting it to be a shell script. | 17:05 |
| ssbarnea|rover | btw, i am going out for few minutes, i need to mount some RAM in my homelab. | 17:06 |
| panda | ssbarnea|rover: what's the executable on the shell module ? | 17:20 |
| weshay | check it out :)) ssbarnea|rover marios fyi http://lists.openstack.org/pipermail/openstack-discuss/2019-January/002017.html | 17:22 |
| weshay | all ^ | 17:22 |
| weshay | well done everyone!! | 17:22 |
| *** gkadam has quit IRC | 17:29 | |
| honza | could i get some eyes on this patch, please? fixing a typo, literally one character :) https://review.openstack.org/#/c/632764/ | 17:32 |
| ssbarnea|rover | panda: i think that ansible defaults to /bin/sh -- it may be related to this. | 17:33 |
| weshay | done | 17:33 |
| panda | ssbarnea|rover: and you're seeing the bugs locally ? | 17:35 |
| panda | ssbarnea|rover: what does /bin/sh point to in your local machine | 17:35 |
| panda | ? | 17:35 |
| *** holser_ has quit IRC | 17:39 | |
| ssbarnea|rover | panda: something is really weird, now it works. I swear it didn't (centos7 machine), not I need to see what triggers it. | 17:39 |
| *** dsneddon has joined #oooq | 17:40 | |
| panda | ssbarnea|rover: my absence triggers it. | 17:40 |
| panda | ssbarnea|rover: misbehaves when I'm not around. | 17:40 |
| honza | weshay: panda: thanks! | 17:40 |
| *** dsneddon has quit IRC | 17:45 | |
| *** trown is now known as trown|lunch | 17:46 | |
| weshay | rlandy, 1-1 in 10min | 17:50 |
| rlandy | weshay: ack | 17:50 |
| *** dtantsur is now known as dtantsur|afk | 17:55 | |
| *** derekh has quit IRC | 18:00 | |
| *** kopecmartin is now known as kopecmartin|off | 18:01 | |
| *** dsneddon has joined #oooq | 18:01 | |
| *** ykarel|away has quit IRC | 18:03 | |
| *** dsneddon has quit IRC | 18:06 | |
| *** saneax has joined #oooq | 18:11 | |
| *** saneax has quit IRC | 18:11 | |
| *** saneax has joined #oooq | 18:18 | |
| *** saneax has quit IRC | 18:18 | |
| *** ssbarnea|bkp2 has joined #oooq | 18:28 | |
| *** ssbarnea|rover has quit IRC | 18:29 | |
| *** vinaykns has quit IRC | 18:30 | |
| *** vinaykns has joined #oooq | 18:33 | |
| *** dsneddon has joined #oooq | 18:38 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, (1 more message) | 18:40 |
| *** jpena is now known as jpena|off | 18:44 | |
| *** trown|lunch is now known as trown | 18:47 | |
| *** dsneddon has quit IRC | 19:04 | |
| *** dsneddon has joined #oooq | 19:11 | |
| rlandy | ssbarnea|bkp2: hello - could use your help with https://review.openstack.org/#/c/631067/ | 19:41 |
| rlandy | adding some linter checking to that minimal bash script | 19:43 |
| weshay | rlandy, where does the compose log? | 19:48 |
| rlandy | weshay, /home/rlandy/tripleo-ci-reproducer | 19:49 |
| weshay | hrm.. | 19:49 |
| rlandy | docker-compose logs --tail=10 -f | 19:49 |
| rlandy | you will see those after you launch a job | 19:49 |
| rlandy | ^^ there is a logs dir | 19:50 |
| rlandy | weshay: need to tmate? | 19:52 |
| weshay | hrm | 19:53 |
| weshay | maybe | 19:53 |
| weshay | whayutin•~/tripleo-ci-reproducer ᐅ docker-compose logs --tail=10 -f thinkdoe ⌚ 12:52:58 | 19:53 |
| weshay | WARNING: The no_proxy variable is not set. Defaulting to a blank string. | 19:53 |
| rlandy | just a warning? | 19:53 |
| weshay | no.. I get no logs | 19:53 |
| weshay | want me to tmate you in? | 19:54 |
| rlandy | [rlandy@localhost tripleo-ci-reproducer]$ docker-compose logs --tail=10 -f | 19:54 |
| rlandy | WARNING: The no_proxy variable is not set. Defaulting to a blank string. | 19:54 |
| rlandy | that is fine | 19:54 |
| rlandy | get that as well | 19:54 |
| weshay | ya.. but | 19:54 |
| rlandy | Attaching to tripleo-ci-reproducer_merger2_1, tripleo-ci-reproducer_merger0_1, tripleo-ci-reproducer_merger1_1, tripleo-ci-reproducer_merger7_1, tripleo-ci-reproducer_merger3_1, tripleo-ci-reproducer_merger5_1, tripleo-ci-reproducer_executor_1, tripleo-ci-reproducer_web_1, tripleo-ci-reproducer_merger4_1, tripleo-ci-reproducer_merger6_1, tripleo-ci-reproducer_scheduler_1, tripleo-ci-reproducer_gerritconfig_1, tripleo-ci-reproduce | 19:54 |
| rlandy | r_launcher_1, tripleo-ci-reproducer_gerrit_1, tripleo-ci-reproducer_zk_1, tripleo-ci-reproducer_mysql_1, tripleo-ci-reproducer_logs_1 | 19:54 |
| rlandy | merger0_1 | 2019-01-23 19:22:46,921 - gear.Worker.b'Zuul Merger' - DEBUG - Sending GRAB_JOB_UNIQ | 19:54 |
| rlandy | ^^should see that afterwards | 19:54 |
| weshay | hayutin•~/tripleo-ci-reproducer ᐅ docker-compose logs --tail=10 -f thinkdoe ⌚ 12:53:57 | 19:54 |
| weshay | WARNING: The no_proxy variable is not set. Defaulting to a blank string. | 19:54 |
| weshay | Attaching to | 19:54 |
| weshay | whayutin•~/tripleo-ci-reproducer ᐅ | 19:54 |
| weshay | it just dies there | 19:55 |
| rlandy | anything more in the logs dir? | 19:55 |
| rlandy | what job did you try run? | 19:55 |
| rlandy | try something simple first | 19:55 |
| weshay | hayutin•~/tripleo-ci-reproducer ᐅ docker-compose logs --tail=10 -f thinkdoe ⌚ 12:53:57 | 19:55 |
| weshay | WARNING: The no_proxy variable is not set. Defaulting to a blank string. | 19:55 |
| weshay | Attaching to | 19:55 |
| weshay | whayutin•~/tripleo-ci-reproducer ᐅ | 19:55 |
| weshay | oops | 19:56 |
| weshay | whayutin•~/tripleo-ci-reproducer/logs ᐅ ll thinkdoe ⌚ 12:55:46 | 19:56 |
| weshay | total 0 | 19:56 |
| weshay | whayutin•~/tripleo-ci-reproducer/logs ᐅ | 19:56 |
| rlandy | nothing | 19:56 |
| rlandy | ok - pls paste zuul.yaml file | 19:56 |
| weshay | "Status code was 500 and not [200]: HTTP Error 500: Internal Server Error", "redirected": false, "server": "CherryPy/18.1.0", "status": 500, "url": "http://localhost:9000/api/tenant/tripleo-ci-reproducer/status"} | 19:56 |
| rlandy | oh probably dd not start | 19:56 |
| weshay | ya.. | 19:57 |
| rlandy | is your playbook still running? | 19:57 |
| weshay | but I need to see it | 19:57 |
| weshay | ugh. let's tmate for a sec | 19:57 |
| rlandy | weshay: you need the launcher review | 19:57 |
| weshay | hrm.. k | 19:57 |
| rlandy | weshay: blue with me for 5 | 19:57 |
| *** jfrancoa has quit IRC | 20:05 | |
| *** dsneddon has quit IRC | 20:06 | |
| *** dsneddon has joined #oooq | 20:31 | |
| *** dsneddon has quit IRC | 20:35 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, (1 more message) | 20:40 |
| *** vinaykns has quit IRC | 20:51 | |
| *** dsneddon has joined #oooq | 21:04 | |
| tosky | among the logs of the standalone jobs (tripleo-ci-centos-7-scenario003-standalone), are there the files which contains the generated configuration for each service? | 21:09 |
| *** rlandy is now known as rlandy|brb | 21:10 | |
| ssbarnea|bkp2 | is there a way to make docker cli to use a k8n cluster instead of docker? | 21:27 |
| *** jschlueter has joined #oooq | 21:49 | |
| *** agopi has quit IRC | 21:49 | |
| *** rlandy|brb is now known as rlandy | 21:51 | |
| weshay | ssbarnea|bkp2, really doesn't matter which db we use for lp, as long as it's not a time series | 21:58 |
| ssbarnea|bkp2 | weshay: pickle? ;) | 22:01 |
| weshay | oh sush | 22:01 |
| weshay | [weshayutin@thinkdoe tripleo-ci-reproducer]$ docker-compose logs --tail=10 -f | 22:04 |
| weshay | WARNING: The no_proxy variable is not set. Defaulting to a blank string. | 22:04 |
| weshay | Attaching to | 22:04 |
| weshay | [weshayutin@thinkdoe tripleo-ci-reproducer]$ | 22:04 |
| weshay | rlandy, ^ | 22:04 |
| rlandy | weshay: hmmm ... before you got nothing, there was an error? | 22:05 |
| *** trown is now known as trown|outtypewww | 22:05 | |
| rlandy | anyways maybe it's not the same | 22:05 |
| weshay | there was never an error w/ the d-c logs | 22:05 |
| rlandy | weshay: what version of docker are you using? | 22:06 |
| weshay | docker-1.13.1-61.git9cb56fd.fc28.x86_64 | 22:06 |
| weshay | rlandy, works fine when ansible is not starting the compose | 22:08 |
| weshay | rlandy, maybe it is python2 /3 | 22:08 |
| weshay | working atm | 22:08 |
| rlandy | https://forums.docker.com/t/docker-compose-hangs-at-attaching-to/2517/11 | 22:08 |
| weshay | w/ docker-compose up | 22:08 |
| rlandy | was just looking at suggestions there | 22:08 |
| rlandy | something we can edit in the docker-compose.yaml file | 22:09 |
| rlandy | in tripleo-ci-reproducer | 22:09 |
| weshay | it's ansible + docker | 22:10 |
| weshay | rlandy, https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/7025ca4af001d55df60e13aa5d9c7c7315a0a55e/tasks/start.yaml#L7-L13 | 22:14 |
| *** jbadiapa has quit IRC | 22:15 | |
| weshay | rlandy, https://docs.ansible.com/ansible/latest/modules/docker_service_module.html | 22:17 |
| rlandy | we should put more debug https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/7025ca4af001d55df60e13aa5d9c7c7315a0a55e/tasks/start.yaml#L14 | 22:17 |
| weshay | trying w/ debug to start | 22:17 |
| rlandy | and maybe playbook with -vv | 22:17 |
| rlandy | display something more than nothing | 22:17 |
| weshay | "msg": "Status code was -1 and not [200]: Connection failure: [Errno 104] Connection reset by peer", | 22:19 |
| weshay | "redirected": false, | 22:19 |
| weshay | "retries": 61, | 22:19 |
| weshay | "status": -1, | 22:19 |
| weshay | "url": "http://localhost:9000/api/tenant/tripleo-ci-reproducer/status" | 22:19 |
| weshay | http://pastebin.test.redhat.com/701044 | 22:19 |
| rlandy | looking | 22:22 |
| *** agopi has joined #oooq | 22:22 | |
| weshay | nothing there | 22:23 |
| weshay | trying w/ | 22:23 |
| weshay | - name: Start up zuul and friends via cli | 22:23 |
| weshay | command: chdir={{install_path}} docker-compose up | 22:23 |
| rlandy | output from step before | 22:23 |
| rlandy | yeah | 22:23 |
| rlandy | that is the interesting one | 22:23 |
| weshay | that works | 22:23 |
| weshay | f.. ansible | 22:23 |
| rlandy | it doesn't fail | 22:23 |
| rlandy | doesn't mean it works | 22:23 |
| weshay | #- name: Start up zuul and friends | 22:24 |
| weshay | # docker_service: | 22:24 |
| weshay | # project_src: "{{ install_path }}" | 22:24 |
| weshay | # state: present | 22:24 |
| weshay | # pull: "{{ pull | default(false) | bool }}" | 22:24 |
| weshay | # debug: yes | 22:24 |
| weshay | # | 22:24 |
| weshay | - name: Start up zuul and friends via cli | 22:24 |
| weshay | command: chdir={{install_path}} docker-compose up | 22:24 |
| weshay | no.. I have the logs now | 22:24 |
| rlandy | docker-compose logs | 22:25 |
| weshay | ya.. now it works | 22:25 |
| weshay | fine | 22:25 |
| rlandy | back to original user> | 22:25 |
| weshay | using weshayutin still | 22:26 |
| weshay | it never worked there | 22:26 |
| rlandy | so it's working now? | 22:27 |
| weshay | yes | 22:27 |
| rlandy | is zuul up? | 22:27 |
| rlandy | gerrit? | 22:27 |
| weshay | 50 seconds in | 22:27 |
| rlandy | still polling | 22:28 |
| weshay | needs to be something like | 22:33 |
| weshay | - name: Start up zuul and friends via cli | 22:33 |
| weshay | command: chdir={{install_path}} docker-compose up | 22:33 |
| weshay | async: 45 | 22:33 |
| weshay | poll: 0 | 22:33 |
| weshay | the docker command is automatically async | 22:33 |
| weshay | I think I still have an issue w/ my keys | 22:33 |
| weshay | but.. I have logs now at least | 22:33 |
| weshay | TASK [ansible-role-tripleo-ci-reproducer : Start up zuul and friends via cli] *** | 22:33 |
| weshay | changed: [localhost] | 22:33 |
| weshay | TASK [ansible-role-tripleo-ci-reproducer : Wait for zuul tenant] *************** | 22:33 |
| weshay | FAILED - RETRYING: Wait for zuul tenant (60 retries left). | 22:33 |
| weshay | ya.. this will be fun to try and explain to quiquell|off and sshnaidm|afk | 22:34 |
| weshay | rlandy, real error is here: | 22:34 |
| weshay | http://pastebin.test.redhat.com/701049 | 22:34 |
| weshay | 019-01-23 22:34:26,800 - zuul.TenantParser - INFO - Generating RSA keypair for project rdo-jobs | 22:35 |
| rlandy | python3.7 | 22:35 |
| rlandy | wee we know you are running python3 at least | 22:35 |
| weshay | ya.. I need to try this on a clean box | 22:36 |
| rlandy | paramiko.ssh_exception.SSHException: Invalid key | 22:36 |
| rlandy | project rdo-jobs | 22:36 |
| rlandy | are you permission on rdo-jobs projects | 22:36 |
| rlandy | can you submit there | 22:36 |
| weshay | OH | 22:36 |
| weshay | maybe not | 22:37 |
| rlandy | we can check that | 22:37 |
| weshay | https://review.rdoproject.org/r/#/admin/projects/rdo-jobs,access | 22:38 |
| rlandy | weshayutin4 commits 278 ++ 1 -- | 22:38 |
| rlandy | you have 4 commits as weshayutin | 22:38 |
| weshay | rlandy, ya.. but I don't have read access | 22:38 |
| weshay | it's config-core | 22:38 |
| rlandy | where do you see that | 22:39 |
| weshay | I don't know | 22:39 |
| weshay | meh | 22:39 |
| rlandy | maybe I can change it for you? | 22:39 |
| weshay | was hoping | 22:39 |
| weshay | quique is not in it either | 22:39 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007 (1 more message) | 22:40 |
| weshay | command: "sh -c 'ansible-playbook inject-ssh-keys.yaml wait-gerrit.yaml add-gerrit-host-keys.yaml; zuul-scheduler -d'" | 22:41 |
| weshay | I'll have to dig into that.. | 22:42 |
| weshay | stepping away :) | 22:42 |
| weshay | thanks for the hand rlandy | 22:42 |
| rlandy | that was just fixed - supposedly | 22:42 |
| rlandy | do you have this latest change? | 22:44 |
| rlandy | https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/commit/08b2cc9887b462ed3832f644568f9e6651a4364c | 22:44 |
| rlandy | rdo_gerrit_key | 22:45 |
| rlandy | maybe it's your rdo gerrit user and key | 22:53 |
| rlandy | https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/9812d6e3d73b21a10b06689ff5513f73e3e3b989/templates/zuul.conf.j2#L23 | 22:53 |
| weshay | hrm | 23:03 |
| weshay | ya.. content: "{{ lookup('env', item) | b64decode }}\n" | 23:04 |
| rlandy | there's something going on there | 23:28 |
| *** rlandy is now known as rlandy|bbl | 23:29 | |
| *** tosky has quit IRC | 23:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!