| *** Vorrtex has quit IRC | 00:16 | |
| *** weshay has joined #oooq | 00:24 | |
| *** aakarsh has joined #oooq | 02:04 | |
| *** ykarel|away has joined #oooq | 02:12 | |
| *** raukadah is now known as chandankumar | 03:23 | |
| *** dsneddon has quit IRC | 03:31 | |
| *** dsneddon has joined #oooq | 03:46 | |
| *** dsneddon has quit IRC | 03:53 | |
| *** gkadam has joined #oooq | 03:55 | |
| *** udesale has joined #oooq | 03:57 | |
| *** gkadam has quit IRC | 04:08 | |
| *** rfolco has quit IRC | 04:17 | |
| *** surpatil has joined #oooq | 04:26 | |
| *** ykarel|away has quit IRC | 04:27 | |
| *** dsneddon has joined #oooq | 04:30 | |
| *** weshay has quit IRC | 04:40 | |
| *** ykarel|away has joined #oooq | 04:48 | |
| *** ykarel|away is now known as ykarel | 04:51 | |
| *** jaosorior has joined #oooq | 05:05 | |
| *** ratailor has joined #oooq | 05:07 | |
| *** jbadiapa has joined #oooq | 05:13 | |
| ykarel | panda, are there still some issues with promoter server? i see no promotion yet | 05:57 |
|---|---|---|
| *** jfrancoa has joined #oooq | 05:59 | |
| *** surpatil has quit IRC | 06:12 | |
| *** surpatil has joined #oooq | 06:13 | |
| *** jbadiapa has quit IRC | 06:22 | |
| *** jtomasek has joined #oooq | 06:30 | |
| *** dsneddon has quit IRC | 06:55 | |
| *** dsneddon has joined #oooq | 06:57 | |
| *** dsneddon has quit IRC | 07:03 | |
| *** zbr is now known as zbr|ruck | 07:08 | |
| zbr|ruck | ykarel: yeah. | 07:08 |
| *** bogdando has joined #oooq | 07:14 | |
| ykarel | zbr|ruck, and any fix for it? | 07:15 |
| *** tesseract has joined #oooq | 07:16 | |
| *** tosky has joined #oooq | 07:16 | |
| *** apetrich has joined #oooq | 07:17 | |
| *** dtantsur|afk is now known as dtantsur | 07:23 | |
| *** kopecmartin|off is now known as kopecmartin | 07:24 | |
| *** jbadiapa has joined #oooq | 07:27 | |
| *** dsneddon has joined #oooq | 07:27 | |
| *** recheck has quit IRC | 07:27 | |
| *** recheck has joined #oooq | 07:28 | |
| *** soniya29 has joined #oooq | 07:29 | |
| zbr|ruck | i got a msg from wes last night, this is what I am looking at. | 07:31 |
| ykarel | okk | 07:32 |
| panda | I really don't understand | 07:34 |
| arxcruz|ruck | ykarel: https://review.opendev.org/#/c/685709/ worked :) | 07:36 |
| *** dsneddon has quit IRC | 07:36 | |
| panda | zbr|ruck: found anything yet ? | 07:38 |
| panda | zbr|ruck: I really can't find a reason for the different behaviour in the promoter for the docker login | 07:40 |
| zbr|ruck | panda: f***, why we are not testing out code? http://38.145.34.55/centos7_master.log-20190930 | 07:41 |
| zbr|ruck | "Pull the images from rdoproject registry" | 07:41 |
| *** amoralej|off is now known as amoralej | 07:41 | |
| zbr|ruck | and i am not surprised, it has a sudo on it but nobody ever installed requirements for root user. | 07:42 |
| panda | zbr|ruck: it always worked, and what the requirements have to do with getting the wrong token ? | 07:43 |
| *** jpena|off is now known as jpena | 07:43 | |
| panda | jpena: ther he is ! :) | 07:44 |
| zbr|ruck | what wes reported is not what I see on the logs. | 07:45 |
| * jpena ducks | 07:45 | |
| zbr|ruck | docker module was not installed, obviously that any ansible docker module would fail | 07:45 |
| panda | zbr|ruck: I'd say even more, dlrnapi_promoter import configparser (all lowercase) | 07:45 |
| panda | zbr|ruck: wchih is the name in python3 | 07:45 |
| panda | jpena: quack ? | 07:45 |
| panda | zbr|ruck: it doesn't fail | 07:45 |
| panda | zbr|ruck: it works , but give strange results. I tried also manually , as root, inside and outside the same venv the promotion uses, in ansible and with cli | 07:46 |
| panda | manually the login command works | 07:47 |
| zbr|ruck | the logs is full or Failed to import docker or docker-py (Docker SDK for Python) - No module named requests.exceptions. Try `pip install docker` or `pip install docker-py` (Python 2.6) | 07:47 |
| zbr|ruck | it should have NONE | 07:47 |
| zbr|ruck | this is not caused by bad-token/login whatever. | 07:47 |
| panda | zbr|ruck: those were the older logs | 07:47 |
| panda | zbr|ruck: when ansible was installed inside the virtualenv | 07:47 |
| panda | zbr|ruck: we removed it, to use ansible from package | 07:48 |
| panda | zbr|ruck: look at the most recent logs | 07:48 |
| panda | zbr|ruck: those are the result of the manual attempt | 07:48 |
| *** ykarel is now known as ykarel|lunch | 07:48 | |
| zbr|ruck | panda: last msg is "Promoting the container images for" at ~3am and nothing at all after. | 07:49 |
| panda | zbr|ruck: yes, we disabled the automatic script, and launched manually for the last 10 runs | 07:49 |
| jpena | panda: so what is the current situation with the registry? | 07:50 |
| panda | zbr|ruck: but the last 10 runs don't have dependency errors, the push fails with authentication error. | 07:50 |
| panda | jpena: we are not able to rule out any possibilities | 07:51 |
| panda | jpena: manually logins work, but in the promotion process they get the bogus token | 07:51 |
| panda | jpena: but even using older code and completely reinstalled ven, it fails | 07:51 |
| panda | venv* | 07:51 |
| zbr|ruck | panda: can you send me a list of valid/safe arguments/vars to pass to to container-push.yml so I can test? | 07:51 |
| jpena | panda: are there any logs from the promoter I can check? | 07:53 |
| panda | jpena: http://38.145.34.55/centos7_master.log-20190930 not sure if they'll be useful, most of them fail with "authentication error" but unless you acces the promoter server, it's not clear what's happening | 07:54 |
| panda | jpena: I emailed you the .docker/config.job that contains the token result from the docker login there. | 07:55 |
| zbr|ruck | panda: can you please tell me to which line to look? | 07:56 |
| panda | zbr|ruck: for what ? | 07:57 |
| jpena | panda: I understand this is done using an ansible playbook, can you run it with -vvv and see if it provides any meaningful output? | 07:58 |
| zbr|ruck | found auth one. but here is the deal "Pull the images from rdoproject registry" works, | 07:58 |
| panda | zbr|ruck: pull happens before the login | 07:59 |
| panda | zbr|ruck: and the pulls are not authenticated yet | 07:59 |
| jpena | panda: just to check. A couple weeks ago we had to update one of the tripleo tokens in the registry, because it was leaked by a job. Was that changed in the promoter server? | 08:00 |
| panda | jpena: yes. We had promotions since then | 08:00 |
| jpena | panda: oh, ack | 08:00 |
| jpena | the registry authentication has not changed since then, we're planning to do it but we decided to wait a bit | 08:01 |
| zbr|ruck | i had the impression that even for pulls we need auth,... so how it comes that pulls works? | 08:01 |
| panda | jpena: yep, that's what I was afraid to hear from you. I would have liked better a "oh yeah, let me revert this change I made yesterday" | 08:02 |
| jpena | sorry :-/ | 08:02 |
| *** apetrich has quit IRC | 08:02 | |
| panda | jpena: but it's probably a problem on our side | 08:02 |
| panda | zbr|ruck: pulls don't need auth | 08:02 |
| panda | zbr|ruck: yet | 08:02 |
| *** jbadiapa has quit IRC | 08:03 | |
| *** dsneddon has joined #oooq | 08:04 | |
| arxcruz|ruck | panda: zbr|ruck getting late to the party, did you guys check if you have some dot directory with some dirty on the promoter ? | 08:04 |
| *** jbadiapa has joined #oooq | 08:04 | |
| panda | arxcruz|ruck: like what ? | 08:04 |
| arxcruz|ruck | .config/docker for example | 08:04 |
| arxcruz|ruck | i don't know maybe it has some cache that are messing with the promoter | 08:05 |
| panda | arxcruz|ruck: we have a .docker/config.json, it has the cache with auth tokens, but it's replaced at every login | 08:05 |
| panda | jpena: as an extreme measure, what would it take to disable authentication off just the time needed for us to promote, and then turn it back on ? | 08:06 |
| arxcruz|ruck | are you sure? | 08:06 |
| jpena | panda: for pushes? | 08:06 |
| *** apetrich has joined #oooq | 08:06 | |
| panda | jpena: yes | 08:07 |
| jpena | panda: I don't think that is even possible | 08:07 |
| panda | jpena: if we can't solve this in a reasonable time | 08:07 |
| panda | jpena: wonderful, ok | 08:07 |
| chandankumar | zbr|ruck: Hello | 08:09 |
| *** dsneddon has quit IRC | 08:09 | |
| chandankumar | zbr|ruck: in RHEL-8 job we use ansible_python_interpreter in job definition, after ansible-2.8 update is it still needed? | 08:10 |
| zbr|ruck | chandankumar: not needed with 2.8+, safe to remove. | 08:13 |
| *** surpatil has quit IRC | 08:13 | |
| zbr|ruck | panda: i wonder if subsequent docker logins do cancel previous ones | 08:14 |
| zbr|ruck | but we should be able to reproduce the problem by calling this playbook, right? | 08:14 |
| zbr|ruck | sad that the script does not dump all these vars | 08:14 |
| jpena | panda: I see there were some promotions yesterday for queens and rhel8, is that correct? | 08:15 |
| panda | zbr|ruck: dumping increases the risk of leaking | 08:15 |
| panda | zbr|ruck: did we have promotion for queens yesterday ? | 08:15 |
| *** jaosorior has quit IRC | 08:16 | |
| zbr|ruck | because promoter runs on specific machine, i would question why we even deal with credentials in scripts. it should just be configured on the server, permanently. | 08:16 |
| zbr|ruck | like I have locally. | 08:17 |
| panda | zbr|ruck: we don't deal with credentials in the scripts | 08:17 |
| panda | zbr|ruck: the credentials are only in the server | 08:17 |
| panda | zbr|ruck: the scripts just use them | 08:18 |
| chandankumar | zbr|ruck: ok, thanks! | 08:19 |
| panda | jpena: I see queens is at 20 days | 08:19 |
| jpena | panda: yes. I was checking the logs and saw that there was one from yesterday without an auth error... But it probably didn't do a thing | 08:20 |
| zbr|ruck | panda: really? username: "{{ dockerhub_username }}" does look like processing to me. | 08:20 |
| panda | zbr|ruck: how else would you select what username to use ? | 08:20 |
| zbr|ruck | panda: one username is not enough? | 08:21 |
| zbr|ruck | https://etherpad.openstack.org/p/ssbarnea ? | 08:22 |
| zbr|ruck | and guess, what doing a ~/.docker/config.json explains why it does not owkr | 08:23 |
| *** yolanda has joined #oooq | 08:23 | |
| zbr|ruck | I bet "auth": "Og==", --- is not a valid login | 08:24 |
| panda | zbr|ruck: did you read the email I sent yesterday ? | 08:24 |
| zbr|ruck | nope, only wes one. | 08:25 |
| zbr|ruck | missed to scroll more | 08:25 |
| panda | zbr|ruck: that token is recreated every time we launch the promotion | 08:25 |
| jpena | panda: where is the code we're using for the promoter? I can check it and see if I find anything | 08:25 |
| arxcruz|ruck | panda: zbr|ruck tmux ? we are all in the promoter :) | 08:27 |
| panda | jpena: I would not want to inflict you this amount of pain, the code is problematic, and we broke it while trying to make it better. There are lots of things that shouldn't work at all, but they did until yesterday. | 08:27 |
| panda | arxcruz|ruck: there's already a tmux sesison open from me and wes yesterday | 08:28 |
| zbr|ruck | quick workaround: lets remove the docker_login from script and perform a login manually | 08:28 |
| panda | zbr|ruck: let's try | 08:29 |
| zbr|ruck | we can even put a cron to keep the token refreshed. | 08:29 |
| zbr|ruck | this should give us time to debug it | 08:29 |
| panda | zbr|ruck: you're going to do it ? | 08:29 |
| zbr|ruck | yep,.... on it | 08:29 |
| panda | this could give time to just merge the new code we have for this thing also | 08:29 |
| *** derekh has joined #oooq | 08:32 | |
| zbr|ruck | panda: https://review.rdoproject.org/r/#/c/22743/ | 08:33 |
| panda | zbr|ruck: You'll have to do it directly in the server | 08:34 |
| *** pierrepr1netti is now known as pierreprinetti | 08:34 | |
| zbr|ruck | i know, this is what i am preparing to do now | 08:34 |
| panda | zbr|ruck: we disabled automatic update in the server | 08:34 |
| panda | zbr|ruck: ok | 08:34 |
| zbr|ruck | aha. | 08:34 |
| panda | ? | 08:35 |
| *** pierreprinetti has quit IRC | 08:35 | |
| *** chem has joined #oooq | 08:35 | |
| *** surpatil has joined #oooq | 08:37 | |
| *** dsneddon has joined #oooq | 08:37 | |
| zbr|ruck | panda: ok cherry picked new code and performed manual login, also created a bkp cfg. | 08:46 |
| panda | zbr|ruck: ok, we can start a manual promotion in the tmux ? | 08:46 |
| panda | zbr|ruck: already in promoter_venv with user centos | 08:47 |
| panda | zbr|ruck: tmux atta | 08:47 |
| zbr|ruck | panda: obviously that centos user cannot login because it cannot connect to docker socket. | 08:50 |
| panda | zbr|ruck: login as root | 08:50 |
| zbr|ruck | but root is logged in | 08:50 |
| panda | zbr|ruck: and root login is all we need | 08:50 |
| *** ykarel|lunch is now known as ykarel | 08:54 | |
| panda | zbr|ruck: arxcruz|ruck where is the promoter running did someone start it ? | 08:55 |
| panda | zbr|ruck: arxcruz|ruck talk here ... bash is somewhat limited as a chat tool | 08:56 |
| arxcruz|ruck | panda: zbr|ruck google hangouts ? | 08:56 |
| panda | arxcruz|ruck: google meet ? I have not idea how to start a new meet though | 08:56 |
| arxcruz|ruck | let me check how | 08:56 |
| panda | arxcruz|ruck: zbr|ruck I'll start the promotion | 08:56 |
| arxcruz|ruck | ack | 08:57 |
| panda | zbr|ruck: promotion started | 08:57 |
| zbr|ruck | this kind of script is supposed to run on tower/awx. it would have much easier to track it there. | 08:58 |
| zbr|ruck | load avg 3 | 09:02 |
| panda | what ? | 09:03 |
| arxcruz|ruck | what [2] | 09:03 |
| arxcruz|ruck | is it promoting? i'm seeing stucked in promoting... | 09:04 |
| panda | zbr|ruck: we could speak for ages about how this should be. THis is what we have. Until we have tests, there's not much we can change. After we have tests, we'll have to make small incremental changes | 09:04 |
| arxcruz|ruck | not sure if should have logs | 09:04 |
| panda | arxcruz|ruck: it's running the ansible part, we won't get any output until it finishes | 09:04 |
| panda | (oh the joy) | 09:04 |
| zbr|ruck | i proposed setting an awx instance but the boss said is not a prio now. tbh, we have bigger problems. | 09:06 |
| ykarel | arxcruz|ruck, looking | 09:07 |
| ykarel | but i still have doubts | 09:07 |
| arxcruz|ruck | ykarel: come on, have faith | 09:08 |
| panda | zbr|ruck: what would that change ? | 09:09 |
| zbr|ruck | panda: i will show you when i have time (if ever), for start clear execution logs. | 09:10 |
| ykarel | arxcruz|ruck, yes i have faith on that change and you as well | 09:10 |
| ykarel | my doubt is more specific to goal | 09:10 |
| ykarel | so earlier we were using latest centos image | 09:11 |
| ykarel | but now with that patch we will stick to image | 09:11 |
| panda | zbr|ruck: putting this in awx will not automatically clear the logs | 09:11 |
| ykarel | also with centos we get advantage of mirror, but not with rdo, | 09:11 |
| ykarel | but that's secondary, main issue is with the first one | 09:12 |
| panda | zbr|ruck: there's a lot of work to do anyway | 09:12 |
| panda | zbr|ruck: frankly I'm not even sure we would need to choose a single language, instead of trying to integrate python, ansible and bash all together happily | 09:13 |
| panda | I think the promotion is stuck | 09:13 |
| panda | we have 43 images downloaded still :( | 09:14 |
| panda | fix a problem another pops out .. | 09:14 |
| arxcruz|ruck | ykarel: we are still stick to a specific image | 09:17 |
| arxcruz|ruck | the 1901 if i recall correctly | 09:18 |
| arxcruz|ruck | ykarel: but it's also only on ovb promotion jobs | 09:18 |
| panda | yep, promoter stuck at pulling images | 09:18 |
| *** ratailor_ has joined #oooq | 09:24 | |
| *** ratailor has quit IRC | 09:26 | |
| *** rfolco has joined #oooq | 09:29 | |
| *** jbadiapa has quit IRC | 09:29 | |
| *** jbadiapa has joined #oooq | 09:30 | |
| panda | zbr|ruck: arxcruz|ruck the promotion is stuck and I have not idea what is happening. | 09:34 |
| arxcruz|ruck | panda: is it possible to run the ansible code from the promotion manually ? | 09:35 |
| arxcruz|ruck | can jpena show the logs of the rdo registry to see what's happening on the other side ? | 09:35 |
| jpena | arxcruz|ruck: let me get them | 09:35 |
| panda | arxcruz|ruck: I've never tried, but it should be possible | 09:36 |
| panda | arxcruz|ruck: at that point we have to revert to completely manual promotion | 09:36 |
| panda | and at this point it seems the best option. | 09:40 |
| jpena | the last log line from the registry related to the promoter server I see is https://softwarefactory-project.io/paste/show/1605/ | 09:41 |
| jpena | from 10 minutes ago | 09:41 |
| panda | jpena: it's a push ? | 09:42 |
| panda | jpena: seems to be a push | 09:42 |
| jpena | http.request.method=PUT | 09:43 |
| panda | jpena: we push current-tripleo tag to rdo registry befoer pushing to docker io | 09:43 |
| panda | so even when the auth succeed, something wrong happens while pushing | 09:43 |
| arxcruz|ruck | panda: is it possible to switch the order ? | 09:44 |
| arxcruz|ruck | so we can identify if it's an issue with rdo registry or docker | 09:45 |
| arxcruz|ruck | or it tries both and fail ? | 09:45 |
| *** akahat has joined #oooq | 09:45 | |
| panda | arxcruz|ruck: right now we are stuck at rdo registry, the pushes there fail, we can try to remove the push, I don't remember why we were pushing tags ther in the first place | 09:45 |
| arxcruz|ruck | panda: let's try this | 09:46 |
| panda | zbr|ruck: agreed ? | 09:46 |
| panda | zbr|ruck: let's remove also the push tag to rdo registry | 09:47 |
| zbr|ruck | it is needed afaik | 09:47 |
| panda | zbr|ruck: pushing tags to rdo-registry ? I don't remember why | 09:48 |
| jpena | panda, looking at the registry console, I see it is properly tagging the images as current-tripleo | 09:48 |
| jpena | https://console.registry.rdoproject.org/registry#/images/tripleomaster/centos-binary-nova-conductor:current-tripleo , for example | 09:48 |
| *** ratailor__ has joined #oooq | 09:48 | |
| panda | jpena: thanks, I'm afraid at this point it may be incompatibility issue between docker-py and docker-ce | 09:50 |
| arxcruz|ruck | jpena: panda is it possible from docker push directly to rdo registry ? | 09:50 |
| arxcruz|ruck | like a proxy or something like that ? | 09:50 |
| panda | mmhh | 09:50 |
| zbr|ruck | no | 09:50 |
| panda | arxcruz|ruck: I WISH! | 09:50 |
| arxcruz|ruck | lol | 09:50 |
| panda | I'm restarting docker daemon | 09:50 |
| *** ratailor_ has quit IRC | 09:50 | |
| zbr|ruck | panda: better not | 09:51 |
| arxcruz|ruck | panda: zbr|ruck so, shall we test only on docker.io ? | 09:51 |
| panda | zbr|ruck: why not ? I'm out of ideas, it's possible the manual promotion will not work at this point | 09:51 |
| zbr|ruck | panda: you are more likely to mess it with partial pushes. i am now looking that the code. i am going to rewrite ansible execution. | 09:52 |
| panda | sure, rewrite the core logic in the production server in the heat of a bugfix with pressing deadline, and wihout regression tests. What could go wrong | 09:53 |
| arxcruz|ruck | panda: at least it's not friday :D | 09:54 |
| * arxcruz|ruck trying to be positive | 09:54 | |
| zbr|ruck | panda: today we are paying the price of coding on our knees.... not sure who assumed that capturing an entire playbook execution with check_output is ok. | 09:55 |
| zbr|ruck | there are two playbooks running in parallel | 09:59 |
| panda | ? | 09:59 |
| panda | ah wonderful | 10:01 |
| panda | so we didn't coordinate correctly | 10:01 |
| panda | zbr|ruck: the previous promotion did not finish | 10:01 |
| panda | zbr|ruck: ti was stuck in the middle | 10:01 |
| zbr|ruck | lets kill ansible-playbook processes first, one by one | 10:02 |
| zbr|ruck | is there any reason why check_output was not used with PIPE? | 10:03 |
| panda | with fire | 10:03 |
| arxcruz|ruck | panda: https://github.com/ansible/ansible/issues/32868#issuecomment-344055870 | 10:06 |
| arxcruz|ruck | panda: zbr|ruck https://meet.google.com/gwb-ikpz-jvg | 10:07 |
| arxcruz|ruck | panda: ^ | 10:10 |
| zbr|ruck | panda: please join. | 10:11 |
| zbr|ruck | https://etherpad.openstack.org/p/ssbarnea | 10:13 |
| *** jaosorior has joined #oooq | 10:24 | |
| bogdando | folks, do we have one or two ovb jobs with pacemaker (with centos 7)? | 10:26 |
| bogdando | I forgot which one... | 10:27 |
| *** udesale has quit IRC | 10:27 | |
| *** udesale has joined #oooq | 10:28 | |
| rfolco | panda, quick question: how do you run delegated tests locally ? | 10:29 |
| *** soniya29 has quit IRC | 10:37 | |
| panda | rfolco: molecule converge -s $scenario | 10:38 |
| rfolco | panda, it returns an error... | 10:38 |
| rfolco | module_stderr: |- | 10:38 |
| rfolco | /bin/sh: /home/zuul/test-python/bin/python: Permission denied | 10:38 |
| rfolco | panda, shouldn't delegated-pre.yml be the prepare stage ? | 10:39 |
| rfolco | like prepare.yml | 10:40 |
| panda | rfolco: ah yea, you need to export MOLECULE_INTERPRETER=$path/to/your/python | 10:40 |
| panda | rfolco: this is the trade off to be able to run the same scenario in both locally and zuul | 10:45 |
| rfolco | panda, how do you run the delegated-pre.yml locally ? | 10:46 |
| rfolco | ansible-playbook ? | 10:46 |
| panda | rfolco: yes | 10:48 |
| *** jbadiapa has quit IRC | 10:53 | |
| *** akahat has quit IRC | 11:00 | |
| *** ykarel is now known as ykarel|meeting | 11:02 | |
| *** udesale has quit IRC | 11:05 | |
| *** udesale has joined #oooq | 11:06 | |
| *** chem` has joined #oooq | 11:06 | |
| *** chem has quit IRC | 11:08 | |
| rfolco | panda, what you have in your localhost test ? centos ? | 11:10 |
| rfolco | panda, I fix one issue, I get another.... so I suspect I am doing it wrong | 11:11 |
| panda | zbr|ruck: arxcruz|ruck the containers push passed | 11:11 |
| panda | zbr|ruck: arxcruz|ruck we have containers promoted in docker.io | 11:11 |
| *** soniya29 has joined #oooq | 11:11 | |
| panda | two steps missing | 11:11 |
| panda | rfolco: what's the error ? | 11:12 |
| rfolco | panda, cannot find python2-dnf.... running local in fedora | 11:12 |
| *** udesale has quit IRC | 11:17 | |
| panda | rfolco: install python2-dnf | 11:20 |
| panda | rfolco: I haven't tested in fedora | 11:20 |
| panda | rfolco: only centos | 11:20 |
| *** ykarel|meeting is now known as ykarel | 11:20 | |
| rfolco | panda, this is my question... you have a centos local box ? | 11:20 |
| panda | rfolco: a vm | 11:20 |
| rfolco | panda, ok | 11:21 |
| arxcruz|ruck | panda: so, the problem is on rdo registry right | 11:23 |
| panda | rfolco: but the molecule-delegated-pre was created mainly for zuul, locally I had already all the packages needed | 11:23 |
| panda | arxcruz|ruck: what makes you say that ? | 11:23 |
| rfolco | panda, my idea was to rename it to prepare.yml | 11:23 |
| rfolco | panda, so molecule would run it in zuul and also locally in prepare stage | 11:24 |
| panda | arxcruz|ruck: from what I see, the problem is that something breaks whan the python script calls the ansible playbook | 11:24 |
| panda | and it's something that even the integration test can't detect, since we are not logging in to rdoregistry | 11:25 |
| panda | the only thing that we can't test wiht the integration test, is the one that is breaking everything | 11:25 |
| panda | I think I'll start a new career as a farmer in iceland. | 11:26 |
| panda | arxcruz|ruck: zbr|ruck launching promote-image.sh script manually now to promote overcloud images | 11:33 |
| *** jpena is now known as jpena|lunch | 11:38 | |
| *** weshay has joined #oooq | 11:41 | |
| weshay | panda, arxcruz|ruck zbr|ruck ping me when you guys are avail | 11:42 |
| weshay | brb | 11:45 |
| zbr|ruck | i am back | 11:46 |
| arxcruz|ruck | weshay: i'm here | 11:47 |
| zbr|ruck | i joined promoter meeting | 11:48 |
| weshay | k.. /me also on the promoter mtg | 11:50 |
| weshay | https://meet.google.com/ito-fxdo-trb?authuser=1 | 11:50 |
| weshay | arxcruz|ruck, zbr|ruck have you heard from panda today? | 11:50 |
| Tengu | weshay: [13:26] < panda> I think I'll start a new career as a farmer in iceland. | 11:51 |
| zbr|ruck | sure, we were in meeting earlier reg promoter | 11:51 |
| weshay | Tengu, heh. | 11:51 |
| Tengu | guess he went there sooner than expected :) | 11:51 |
| zbr|ruck | good idea, i liked iceland too. | 11:51 |
| zbr|ruck | not so sure about farming, but lava-bread was good. | 11:51 |
| weshay | I like it iceland so much, I think I'll buy it | 11:51 |
| *** dsneddon has quit IRC | 12:17 | |
| *** dsneddon has joined #oooq | 12:20 | |
| *** jfrancoa has quit IRC | 12:22 | |
| *** dsneddon has quit IRC | 12:25 | |
| *** holser has joined #oooq | 12:25 | |
| chandankumar | weshay: zbr|ruck panda jpena|lunch https://review.rdoproject.org/r/#/c/22550/ please have a look, thanks! | 12:27 |
| *** jfrancoa has joined #oooq | 12:37 | |
| *** jpena|lunch is now known as jpena | 12:38 | |
| arxcruz|ruck | zbr|ruck: weshay panda https://docs.ansible.com/ansible/latest/modules/docker_container_module.html | 12:44 |
| *** amoralej is now known as amoralej|lunch | 12:49 | |
| *** Goneri has joined #oooq | 12:56 | |
| *** jaosorior has quit IRC | 12:58 | |
| *** dsneddon has joined #oooq | 13:00 | |
| *** marios has joined #oooq | 13:00 | |
| mjturek | rfolco is the community call in a half hour?? | 13:01 |
| rfolco | mjturek, yes sir :) | 13:01 |
| mjturek | \o/ | 13:01 |
| rfolco | mjturek, feel free to add your topic to the agenda https://etherpad.openstack.org/p/tripleo-ci-squad-meeting @L59 | 13:02 |
| *** jaosorior has joined #oooq | 13:02 | |
| *** jbadiapa has joined #oooq | 13:06 | |
| panda | rfolco: were you able to run delegated ? | 13:08 |
| rfolco | panda, no, not even in a centos vm, its been a nightmare | 13:09 |
| panda | rfolco: .... | 13:09 |
| rfolco | panda, also, delegated job in zuul is reporting false positive, ignoring syntax errors http://logs.rdoproject.org/37/22737/7/check/molecule-tripleo-common/8e1bc25/ | 13:10 |
| panda | let's sync after this, after community call after tripleo meeting and DFG all hands and monthly jim webcast, ok ? | 13:10 |
| rfolco | panda, ok next year then | 13:10 |
| rfolco | great timing | 13:10 |
| panda | rfolco: wat ? | 13:11 |
| rfolco | panda, meanwhile working on the dark waiting for next ci job to report result | 13:11 |
| panda | rfolco: that job is passing ? | 13:11 |
| rfolco | panda, there is a syntax error there... fixed in the next patchset | 13:12 |
| panda | rfolco: you should not use destroy in delegated mode | 13:12 |
| rfolco | panda, I don't. Who said that? | 13:13 |
| panda | rfolco: that's why it's passing, it fails converge, then runs destroy | 13:13 |
| panda | rfolco: ah it's not destroy | 13:13 |
| rfolco | panda, fails converge should fail | 13:13 |
| panda | rfolco: it's another playbook | 13:13 |
| rfolco | yeah yeah | 13:13 |
| rfolco | lets sync asap | 13:13 |
| rfolco | please | 13:13 |
| panda | rfolco: I have a slot. between midnight and 1 o clock, the 25th of december 2021 | 13:14 |
| rfolco | panda, :( | 13:15 |
| *** jbadiapa has quit IRC | 13:16 | |
| *** ratailor__ has quit IRC | 13:19 | |
| arxcruz|ruck | brb in 10-15 minutes | 13:20 |
| panda | rfolco: 5 minuts sync, now | 13:20 |
| panda | rfolco: in comunity call link | 13:21 |
| rfolco | panda, ok going | 13:21 |
| panda | rfolco: so we are already prepared | 13:21 |
| panda | rfolco: you there ? someone needs to let me in | 13:23 |
| panda | let me iiiin | 13:23 |
| panda | let meeee iiiiiiiin | 13:23 |
| *** soniya29 has quit IRC | 13:23 | |
| *** Vorrtex has joined #oooq | 13:23 | |
| *** aakarsh has quit IRC | 13:30 | |
| *** Vorrtex has quit IRC | 13:30 | |
| *** amoralej|lunch is now known as amoralej | 13:33 | |
| rfolco | weshay, you joining? | 13:34 |
| zbr|ruck | validating docker login: https://etherpad.openstack.org/p/ssbarnea | 13:35 |
| rfolco | mjturek, baha https://meet.google.com/bqx-xwht-wky | 13:35 |
| rfolco | fyi we are using google meet for community call | 13:35 |
| mjturek | ahh thank you | 13:35 |
| baha | Thank you! We've been sitting in the bluejeans room | 13:35 |
| rfolco | :) | 13:35 |
| panda | rfolco: the part that adds user to docker group is in molecule-delegated-pre , did it run fully on your system ? is not called automatically | 13:41 |
| *** jaosorior has quit IRC | 13:51 | |
| *** SurajPatil has joined #oooq | 13:59 | |
| *** aakarsh has joined #oooq | 14:01 | |
| *** surpatil has quit IRC | 14:02 | |
| *** surpatil has joined #oooq | 14:08 | |
| *** SurajPatil has quit IRC | 14:11 | |
| weshay | marios, you are avail for the scenario004 sync in a bit? | 14:12 |
| *** chem` has quit IRC | 14:12 | |
| marios | weshay: yeah that's what i'm here for | 14:13 |
| weshay | panda, we have a boog | 14:18 |
| weshay | panda, result.. = "dW51c2VkOg==" which translates to "unused" | 14:19 |
| weshay | panda, maybe we mixed up the creds | 14:20 |
| panda | weshay: nope translates to unused: | 14:20 |
| *** SurajPatil has joined #oooq | 14:20 | |
| amoralej | arxcruz|ruck, weshay check last comments in https://bugs.launchpad.net/tripleo/+bug/1845166 | 14:22 |
| openstack | Launchpad bug 1845166 in tripleo "[queens] [Periodic][check] OVB jobs failing in OC deploy when trying to ssh to nodes" [Critical,Triaged] | 14:22 |
| weshay | arxcruz|ruck, when ur back ping me | 14:22 |
| *** surpatil has quit IRC | 14:22 | |
| amoralej | i think we have root cause | 14:22 |
| amoralej | but i'm not sure about proper fix | 14:23 |
| amoralej | in fact it was not related to 7.7 | 14:23 |
| * weshay reads | 14:23 | |
| weshay | ugh | 14:24 |
| weshay | amoralej, can we turn auth off? | 14:24 |
| weshay | for rhel containers | 14:24 |
| amoralej | it was introduced for https://tree.taiga.io/project/tripleo-ci-board/task/1182 | 14:25 |
| amoralej | weshay, you mean rhel/osp? | 14:25 |
| weshay | amoralej, there are no osp containers in the rdo-registry.. but there are rdo on rhel8 containers | 14:26 |
| weshay | that's why all the docker-login and auth work was done | 14:26 |
| amoralej | ah, got it now | 14:26 |
| amoralej | i thing it'd be better to fix the login | 14:26 |
| *** surpatil has joined #oooq | 14:26 | |
| weshay | we can remove docker login.. if we can remove the auth | 14:27 |
| amoralej | that disabling auth | 14:27 |
| amoralej | auth is required in all containers or only rhel containers? | 14:27 |
| weshay | amoralej, where did your deploy fail w/ w/ this root cause? | 14:27 |
| amoralej | https://review.rdoproject.org/r/#/c/22551/ | 14:27 |
| amoralej | that's the reproducer | 14:28 |
| arxcruz|ruck | weshay: ping | 14:29 |
| *** SurajPatil has quit IRC | 14:29 | |
| panda | weshay: fixed, the credentials were ok, problem with indentation on the docker_login task, should work now | 14:30 |
| weshay | panda, may I rerun? | 14:31 |
| weshay | amoralej, and it only affects queens? | 14:31 |
| panda | weshay: yes | 14:32 |
| weshay | panda, ack.. running | 14:33 |
| weshay | arxcruz|ruck, let's chat https://meet.google.com/dou-geaa-njq?authuser=1 | 14:33 |
| amoralej | weshay, only queens | 14:34 |
| amoralej | my guess, after rocky, something is restoring policy to ACCEPT later | 14:34 |
| weshay | amoralej, ykarel oddly my undercloud keeps shutting down during a deployment | 14:34 |
| amoralej | in rocky masqerading connnfiguration was reworked in https://github.com/openstack/puppet-tripleo/blob/master/manifests/masquerade_networks.pp | 14:34 |
| amoralej | my guess is that someching changed | 14:34 |
| weshay | making it difficult for me to bring credible results back to you | 14:34 |
| rfolco | panda, do I need to run this manually ? | 14:35 |
| rfolco | # Export the path to the mounted docker socket so all docker client commands will use it | 14:35 |
| weshay | amoralej, k.. give me to end of the day to try and reproduce that | 14:36 |
| panda | rfolco: no, that's for the docker on docker while using docker driver | 14:39 |
| rfolco | panda, added zuul to docker group, still getting permission denied.... I hate delegated mode | 14:40 |
| rfolco | HATE | 14:40 |
| panda | zuul ? | 14:40 |
| panda | you're running as zuul ? | 14:40 |
| rfolco | - name: Add user to docker group | 14:41 |
| rfolco | become: true | 14:41 |
| rfolco | user: | 14:41 |
| rfolco | name: zuul | 14:41 |
| rfolco | groups: '{{ docker_group.stdout }}' | 14:41 |
| rfolco | append: true | 14:41 |
| panda | rfolco: because that's supposed to run in upstream, is your user zuul ? | 14:41 |
| *** Vorrtex has joined #oooq | 14:42 | |
| rfolco | panda, ok, w/ ansible interpreter and docker group fixed, now I am back to business, thanks | 14:46 |
| panda | rfolco: show me the money! | 14:48 |
| rfolco | panda, mercenary | 14:48 |
| *** aakarsh has quit IRC | 14:49 | |
| ykarel | weshay, ack | 14:51 |
| ykarel | let us know if u face some issue | 14:51 |
| weshay | k.. thanks | 14:51 |
| weshay | ykarel, amoralej although.. I suppose I won't hit the issue if docker login is not executed | 14:52 |
| amoralej | yes | 14:53 |
| ykarel | weshay, yes that's the reason i too didn't faced the issue | 14:53 |
| amoralej | you need to execute it | 14:53 |
| amoralej | yeah, that ^ | 14:53 |
| weshay | which it wouldn't be in the way I'm trying to execute | 14:53 |
| weshay | perhaps we can exclude docker login for queens | 14:54 |
| amoralej | weshay, that may be workaround | 14:54 |
| weshay | panda, you still alive? | 14:54 |
| ykarel | weshay, remember there is a plan to remove anonymous login in rdo registry | 14:54 |
| zbr|ruck | panda: do a visual on https://review.rdoproject.org/r/#/c/22743/3/ci-scripts/container-push/container-push.yml and let me know if I should test it manually | 14:55 |
| weshay | ykarel, sounds like no one is ready for that yet :) | 14:55 |
| ykarel | so you need to consider that as well while removing container login for queens | 14:55 |
| amoralej | yes, temporary workaround until auth is enabled every where | 14:55 |
| weshay | afaik.. we're running login every where | 14:55 |
| weshay | in rdo-cloud | 14:55 |
| amoralej | yes, but i think anonymous is still open | 14:56 |
| amoralej | not sure for how long | 14:56 |
| ykarel | amoralej, yes it's still upon until https://review.rdoproject.org/r/#/c/22623/ don't merges | 14:56 |
| ykarel | as per the comment there plan was to merge yesterday | 14:56 |
| weshay | amoralej, ykarel ok.. so both arxcruz|ruck and I are running reproducers for queens.. w/o login | 14:57 |
| weshay | will be interesting to see where it fails | 14:57 |
| amoralej | it will not fail | 14:57 |
| ykarel | it should not fail | 14:57 |
| amoralej | ykarel already ran that | 14:57 |
| weshay | heh | 14:57 |
| weshay | ykarel, on the promoted hash.. or tripleo-ci-testing? | 14:57 |
| panda | weshay: more or less | 14:57 |
| ykarel | weshay, promoted hashj | 14:57 |
| weshay | ykarel, ya.. promoted hash should work outside of rdo-cloud | 14:58 |
| weshay | agree | 14:58 |
| *** jbadiapa has joined #oooq | 14:58 | |
| weshay | panda, is it possible to disable the docker_login for jobs running queens? | 14:58 |
| weshay | w/ a when: | 14:58 |
| panda | commit hashish, distro hashish | 14:58 |
| panda | weshay: yes, container push has the release information | 14:59 |
| ykarel | registry_login_enabled: false | 14:59 |
| panda | ah the general docker login | 14:59 |
| weshay | panda, ok.. switch context here.. no talking about the promotion.. just the jobs executing deployments | 14:59 |
| weshay | panda, ya | 14:59 |
| panda | weshay: if we pass the release information to all the jobs, then it should be possible, and it's not passed we'll default to true | 15:00 |
| weshay | arxcruz|ruck, https://review.opendev.org/#/c/685184/1/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 | 15:01 |
| *** jbadiapa has quit IRC | 15:05 | |
| *** aakarsh has joined #oooq | 15:13 | |
| *** amoralej is now known as amoralej|brb | 15:22 | |
| weshay | arxcruz|ruck, is that running for you? | 15:24 |
| arxcruz|ruck | weshay: ya | 15:24 |
| *** ykarel is now known as ykarel|afk | 15:26 | |
| *** amoralej|brb is now known as amoralej | 15:29 | |
| weshay | panda, container promote worked, image promote worked... dlrn_api failed w/ | 15:31 |
| weshay | http://pastebin.test.redhat.com/802154 | 15:31 |
| * weshay going to do it by hand | 15:32 | |
| *** tesseract has quit IRC | 15:33 | |
| weshay | arxcruz|ruck, zbr|ruck fyi ^ | 15:33 |
| zbr|ruck | weshay: re validating registry login, https://review.rdoproject.org/r/#/c/22743/4/ci-scripts/container-push/container-push.yml is ready. code tested with your small script. | 15:37 |
| zbr|ruck | can I cherry pick it to promoter? | 15:37 |
| zbr|ruck | i know is ugly,... but it does validate. | 15:37 |
| *** aakarsh has quit IRC | 15:39 | |
| weshay | zbr|ruck, give me a few | 15:39 |
| *** jpena is now known as jpena|brb | 15:42 | |
| zbr|ruck | weshay: sure. on the funny side, read release notes from 3.0 on https://github.com/docker/docker-py/blob/4c71641544f8cacfd27fbd5b36c67fc6763675a2/docs/change-log.md#L362 | 15:42 |
| zbr|ruck | utils.ping_registry and utils.ping have been removed. | 15:43 |
| zbr|ruck | mainly, there *was* a way to ping the registry.... | 15:43 |
| weshay | arxcruz|ruck, zbr|ruck just in case.. please watch the upstream check / gate jobs as new promotions can some times blow things up | 15:53 |
| weshay | zbr|ruck, /me looks | 15:53 |
| weshay | fyi.. master is fully promoted | 15:53 |
| *** marios has quit IRC | 15:53 | |
| weshay | zbr|ruck, where is noop coming from? | 15:56 |
| weshay | is it in the rdo reg? | 15:57 |
| weshay | https://console.registry.rdoproject.org/registry#/images/tripleomaster/noop | 15:57 |
| zbr|ruck | weshay: I am building it for test, does not come from anywhere. this task is building this mini image and pushing it. | 15:58 |
| weshay | k zbr|ruck I'm on the tmux running it.. it's not fast | 15:58 |
| weshay | might be hanging | 15:59 |
| zbr|ruck | i know, but is much faster than the rest. it does run only once per promotion, not for every image. | 15:59 |
| zbr|ruck | better to fail fast than later with 100s images | 15:59 |
| weshay | ya.. running for several minutes, still going | 15:59 |
| zbr|ruck | ~10-15s not minutes. | 16:00 |
| zbr|ruck | i could try to make it work with the famous "scratch" image which is much smaller but I need to create a linux executable, it does not even have bash on it. | 16:01 |
| *** jfrancoa has quit IRC | 16:03 | |
| *** surpatil has quit IRC | 16:04 | |
| weshay | zbr|ruck, ya.. not working | 16:10 |
| weshay | from my laptop or the promotion server | 16:10 |
| *** surpatil has joined #oooq | 16:10 | |
| zbr|ruck | weshay: what is the issue? | 16:10 |
| weshay | just hanging | 16:10 |
| zbr|ruck | give me few minutes, I am trying to make it work with a smaller image. | 16:11 |
| zbr|ruck | weshay: run ./test_bad.sh on promoter now | 16:13 |
| *** SurajPatil has joined #oooq | 16:13 | |
| chandankumar | panda: jpena|brb: weshay arxcruz|ruck zbr|ruck can we merge this https://review.rdoproject.org/r/#/c/22623/ ? | 16:14 |
| weshay | zbr|ruck, that worked great | 16:14 |
| weshay | zbr|ruck, let's put up a review | 16:15 |
| zbr|ruck | i am updating the review now... | 16:15 |
| weshay | we're in no danger of promoting anything else soon | 16:15 |
| weshay | chandankumar, it's potentially blocking queens now | 16:15 |
| weshay | chandankumar, so let's hold | 16:15 |
| weshay | arxcruz|ruck, I'm in deploy | 16:16 |
| *** surpatil has quit IRC | 16:16 | |
| chandankumar | weshay: ok | 16:16 |
| weshay | thanks for asking | 16:16 |
| *** jfrancoa has joined #oooq | 16:17 | |
| chandankumar | weshay: tomorrow is public holiday in india. | 16:18 |
| arxcruz|ruck | weshay: Prepare for the containerized deployment | 16:19 |
| *** jpena|brb is now known as jpena | 16:19 | |
| arxcruz|ruck | weshay: https://review.opendev.org/#/c/685709/ please check the comments and advice what to do | 16:21 |
| zbr|ruck | weshay: do we need any local ci-config changes on promoter? can I reset? | 16:22 |
| *** weshay_ has joined #oooq | 16:25 | |
| *** weshay has quit IRC | 16:26 | |
| weshay_ | ssh is amazing | 16:26 |
| *** SurajPatil has quit IRC | 16:27 | |
| arxcruz|ruck | weshay_: https://review.opendev.org/#/c/685709/ please check the comments and advice what to do | 16:31 |
| *** ykarel|afk is now known as ykarel|away | 16:36 | |
| *** dtantsur is now known as dtantsur|afk | 16:40 | |
| weshay_ | arxcruz|ruck, my job failed in heat | 16:42 |
| weshay_ | arxcruz|ruck, you have any luck | 16:42 |
| weshay_ | ? | 16:42 |
| arxcruz|ruck | weshay_: deploying overcloud now | 16:42 |
| weshay_ | k.. arxcruz|ruck what hash are you on? | 16:42 |
| *** matbu has quit IRC | 16:43 | |
| *** ykarel|away has quit IRC | 16:43 | |
| weshay_ | arxcruz|ruck, /me tries a more recent hash | 16:43 |
| arxcruz|ruck | weshay_: checking | 16:43 |
| arxcruz|ruck | weshay_: dlrn_hash_tag=e2208117a31eea79c50330333501b6581f9faafe_15a9b671 | 16:44 |
| weshay_ | arxcruz|ruck, k.. good | 16:44 |
| weshay_ | latest tripleo-ci-testing | 16:44 |
| *** bogdando has quit IRC | 16:46 | |
| *** weshay_ is now known as weshay | 16:46 | |
| weshay | arxcruz|ruck, if it get's late for you.. give me the ip of the node.. and I can watch it | 16:54 |
| weshay | arxcruz|ruck, put my github key on it | 16:54 |
| arxcruz|ruck | weshay: ack | 16:55 |
| arxcruz|ruck | weshay: what's your github username ? | 16:57 |
| weshay | weshayutin | 16:57 |
| *** derekh has quit IRC | 17:01 | |
| *** jfrancoa has quit IRC | 17:05 | |
| *** aakarsh has joined #oooq | 17:07 | |
| *** amoralej is now known as amoralej|off | 17:11 | |
| *** jpena is now known as jpena|off | 17:16 | |
| *** ykarel|away has joined #oooq | 17:22 | |
| *** aakarsh has quit IRC | 17:33 | |
| ykarel|away | arxcruz|ruck, zbr|ruck is stein standalone upgrade failure already known? | 17:35 |
| ykarel|away | i see last success from 18th september:- https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-upgrade-stein | 17:35 |
| ykarel|away | and seems the only blocker for stein | 17:36 |
| ykarel|away | as per the last run:- https://trunk-primary.rdoproject.org/api-centos-stein/api/civotes_detail.html?commit_hash=a361c5edd5734320f781e10358a602f4ba57fc04&distro_hash=647b08e47a72f7533142c09074f227628f08f9fa | 17:36 |
| ykarel|away | fs035 failure is overcloud timeout, that fails randomly, but stein-upgrade one is real one | 17:37 |
| weshay | ykarel|away, we're focused on queens atm.. oom's in rdo-cloud :) | 17:47 |
| ykarel|away | weshay, ack | 17:48 |
| ykarel|away | but good to have a bug and some one from upgrade should be involved if not yet done | 17:49 |
| weshay | ykarel|away, aye.. just don't kill urself brotha | 17:52 |
| ykarel|away | me alright, have holiday tomorrow, so will rest | 17:53 |
| *** Goneri has quit IRC | 17:59 | |
| *** Vorrtex has quit IRC | 18:03 | |
| *** Goneri has joined #oooq | 18:11 | |
| *** aakarsh has joined #oooq | 18:31 | |
| *** Goneri has quit IRC | 18:34 | |
| *** aakarsh has quit IRC | 18:36 | |
| *** matbu has joined #oooq | 18:38 | |
| rfolco | zbr|ruck, around? | 18:42 |
| rfolco | panda, you? | 18:43 |
| arxcruz|ruck | rfolco: it's almost 9pm here... | 18:43 |
| arxcruz|ruck | weshay: +w https://review.rdoproject.org/r/#/c/22743/ or wait ? | 18:44 |
| *** aakarsh has joined #oooq | 18:44 | |
| rfolco | ok let me figure this out myself | 18:44 |
| arxcruz|ruck | okay, gotta watch macguyver | 18:44 |
| arxcruz|ruck | :D | 18:44 |
| * weshay looks | 18:44 | |
| weshay | arxcruz|ruck, zbr|ruck wait for the team to review on that | 18:44 |
| weshay | rfolco, something is wrong w/ the current promoter code w/ dlrnapi | 18:45 |
| * weshay should find the code | 18:45 | |
| *** Goneri has joined #oooq | 18:48 | |
| rfolco | weshay, sftp issue and dlrnapi auth looks like | 18:48 |
| weshay | sftp worked | 18:49 |
| weshay | well there was an error thrown | 18:49 |
| weshay | but it got the job done | 18:49 |
| weshay | ya.. auth issues | 18:49 |
| weshay | I don't see auth here https://github.com/rdo-infra/ci-config/blob/master//ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py#L85-L111 | 18:50 |
| weshay | ah here https://github.com/rdo-infra/ci-config/blob/master//ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py#L348-L352 | 18:51 |
| rfolco | exactly weshay | 18:52 |
| rfolco | DLRNAPI_PASSWORD env variable is missing or empty | 18:52 |
| rfolco | this is in the logs | 18:52 |
| weshay | rfolco, can you pdb api_instance = dlrnapi_client.DefaultApi(api_client=api_client) | 18:53 |
| weshay | ah k | 18:53 |
| weshay | you got it | 18:53 |
| rfolco | checking promoter env | 18:53 |
| rfolco | weshay, exported dlrnapi_password env, watching logs | 18:58 |
| rfolco | does anybody know how to escape {{ template in ansible properly ? imagename: {{namespace}}/{{name_prefix}}{{ item }}{{name_suffix}}:{{tag} >> only {{ item }} is a var | 19:03 |
| panda | rfolco: use {% raw %} {%endraw%} | 19:07 |
| panda | rfolco: how's the promoter server ? | 19:07 |
| rfolco | panda, I reloaded env var, watching in tmux logs | 19:08 |
| *** aakarsh has quit IRC | 19:08 | |
| panda | why is nothing suddenly working | 19:08 |
| rfolco | panda, did not run again yet... is it stuck ? | 19:08 |
| panda | rfolco: I have no idea, I have to fight a hyperactive daughter right now ... | 19:09 |
| rfolco | panda, may the force be with you | 19:09 |
| rfolco | panda, better not | 19:09 |
| rfolco | panda, may the patience be with you | 19:09 |
| rfolco | weshay, did you stop the service ? | 19:21 |
| weshay | rfolco, ya.. we did yesterday | 19:21 |
| weshay | probably safe to restart.. although we may want to wait for tomorrow morning.. nothing is dire need of promotion now | 19:21 |
| rfolco | weshay, ok, just wanted to check if you needed to test w/ the dlrnapi_password env var set | 19:22 |
| weshay | rfolco, so if we could um have a test that does that :) | 19:26 |
| weshay | that would be super cool | 19:26 |
| weshay | arxcruz|ruck, I have two queens deployments at overcloud_deploy atm | 19:26 |
| weshay | one virt one ovb | 19:26 |
| rfolco | oh glory | 19:35 |
| rfolco | image prepare just worked | 19:35 |
| rfolco | is to glorify standing up, church! << arxcruz|ruck | 19:36 |
| rfolco | sorry will control my emotions now | 19:37 |
| weshay | woot | 19:44 |
| weshay | queens is fine :) | 19:44 |
| weshay | TASK [overcloud-deploy : echo deployment_status] ********************************************************* | 19:44 |
| weshay | Tuesday 01 October 2019 19:43:49 +0000 (0:00:00.120) 2:39:25.090 ******* | 19:44 |
| weshay | ok: [undercloud] => { | 19:44 |
| weshay | "overcloud_deploy_result": "passed" | 19:44 |
| weshay | } | 19:44 |
| weshay | arxcruz|ruck, ^ | 19:44 |
| weshay | arxcruz|ruck, zbr|ruck ok.. I have a passing ovb passing deployment fs001 ( tempest fails to setup ) and a libvirt passing deployment + passing tempest on baseurl=https://trunk.rdoproject.org/centos7-queens/e2/20/e2208117a31eea79c50330333501b6581f9faafe_15a9b671 | 20:07 |
| weshay | I'm going to send queens through | 20:08 |
| weshay | arxcruz|ruck, zbr|ruck focus laser beams on rhel8 scen01/02 and fs001 ovb master | 20:08 |
| *** kopecmartin is now known as kopecmartin|off | 20:11 | |
| *** ykarel|away has quit IRC | 20:17 | |
| *** ykarel|away has joined #oooq | 20:20 | |
| *** ykarel|away has quit IRC | 20:33 | |
| *** Goneri has quit IRC | 20:57 | |
| *** openstackstatus has quit IRC | 21:32 | |
| *** openstackstatus has joined #oooq | 21:34 | |
| *** ChanServ sets mode: +v openstackstatus | 21:34 | |
| *** holser has quit IRC | 21:51 | |
| *** Goneri has joined #oooq | 22:28 | |
| *** weshay has quit IRC | 22:29 | |
| *** aakarsh has joined #oooq | 22:35 | |
| *** Goneri has quit IRC | 22:48 | |
| *** weshay has joined #oooq | 22:52 | |
| *** weshay has quit IRC | 22:55 | |
| *** weshay has joined #oooq | 22:58 | |
| *** tosky has quit IRC | 23:02 | |
| *** weshay has quit IRC | 23:12 | |
| *** weshay has joined #oooq | 23:15 | |
| *** ChanServ sets mode: +o weshay | 23:19 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!