| *** lseki has quit IRC | 00:03 | |
| *** yamamoto has quit IRC | 00:04 | |
| openstackgerrit | Merged opendev/ansible-role-puppet master: Clear out old facts https://review.opendev.org/661888 | 00:05 |
|---|---|---|
| *** yamamoto has joined #openstack-infra | 00:08 | |
| *** Weifan has quit IRC | 00:09 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** gyee has quit IRC | 00:14 | |
| *** slaweq has quit IRC | 00:16 | |
| *** yamamoto has quit IRC | 00:26 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: test_v3: replace while loop with iterate_timeout https://review.opendev.org/662112 | 00:29 |
| *** markvoelker has quit IRC | 00:32 | |
| *** yamamoto has joined #openstack-infra | 00:35 | |
| *** ccamacho has joined #openstack-infra | 00:39 | |
| *** yamamoto has quit IRC | 00:42 | |
| *** jmccrory has quit IRC | 00:46 | |
| *** imacdonn has quit IRC | 00:46 | |
| *** jmccrory has joined #openstack-infra | 00:46 | |
| *** imacdonn has joined #openstack-infra | 00:46 | |
| *** michael-beaver has quit IRC | 00:55 | |
| *** ianychoi has quit IRC | 00:56 | |
| *** factor has quit IRC | 01:02 | |
| *** roman_g has quit IRC | 01:06 | |
| *** yamamoto has joined #openstack-infra | 01:19 | |
| *** gagehugo has quit IRC | 01:21 | |
| *** markvoelker has joined #openstack-infra | 01:21 | |
| *** hwoarang has quit IRC | 01:25 | |
| *** hwoarang has joined #openstack-infra | 01:25 | |
| *** yamamoto has quit IRC | 01:29 | |
| *** roman_g has joined #openstack-infra | 01:29 | |
| *** gagehugo has joined #openstack-infra | 01:30 | |
| *** hwoarang has quit IRC | 01:34 | |
| *** hwoarang has joined #openstack-infra | 01:34 | |
| *** whoami-rajat has joined #openstack-infra | 01:35 | |
| *** hwoarang has quit IRC | 01:46 | |
| *** hwoarang has joined #openstack-infra | 01:46 | |
| *** ccamacho has quit IRC | 02:09 | |
| ianw | sigh, the file isn't removing i don't think :/ | 02:11 |
| *** eernst has joined #openstack-infra | 02:14 | |
| *** hwoarang has quit IRC | 02:20 | |
| *** hwoarang has joined #openstack-infra | 02:20 | |
| *** eernst has quit IRC | 02:23 | |
| *** hwoarang has quit IRC | 02:25 | |
| *** hwoarang has joined #openstack-infra | 02:26 | |
| *** sreejithp has joined #openstack-infra | 02:28 | |
| *** ccamacho has joined #openstack-infra | 02:29 | |
| *** bgmccollum has joined #openstack-infra | 02:32 | |
| *** sreejithp has quit IRC | 02:35 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: test_v3: replace while loop with iterate_timeout https://review.opendev.org/662112 | 02:45 |
| *** hwoarang has quit IRC | 02:55 | |
| *** hwoarang has joined #openstack-infra | 02:56 | |
| *** michael-beaver has joined #openstack-infra | 02:56 | |
| *** roman_g has quit IRC | 03:08 | |
| *** dave-mccowan has quit IRC | 03:15 | |
| ianw | this is certainly a debugging adventure | 03:18 |
| *** hwoarang has quit IRC | 03:18 | |
| *** hwoarang has joined #openstack-infra | 03:19 | |
| *** hwoarang has quit IRC | 03:30 | |
| *** hwoarang has joined #openstack-infra | 03:31 | |
| ianw | ok, i ran it from kick.sh and it *did* clear out the file ... | 03:34 |
| ianw | either i'm too impatient; or runs from cron are re-setting it somehow | 03:34 |
| clarkb | I guess the next run will tell you? | 03:35 |
| clarkb | (if file is put back) | 03:35 |
| ianw | yeah | 03:35 |
| ianw | /etc/ansible/roles/puppet on bridge isn't a git repo, but it has the changes | 03:36 |
| *** yamamoto has joined #openstack-infra | 03:38 | |
| clarkb | ya it is installed via ansible galaxy tooling which takes git repo and convert it to justthe working set of data without the git stuff | 03:38 |
| *** rlandy|bbl has quit IRC | 03:42 | |
| *** ykarel|away has joined #openstack-infra | 03:52 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** hwoarang has quit IRC | 04:23 | |
| *** hwoarang has joined #openstack-infra | 04:23 | |
| *** raukadah is now known as chandankumar | 04:31 | |
| *** hwoarang has quit IRC | 04:34 | |
| *** hwoarang has joined #openstack-infra | 04:34 | |
| *** ricolin has joined #openstack-infra | 04:45 | |
| *** ramishra has joined #openstack-infra | 04:48 | |
| ianw | so the file did *not* reappear on ze01, but it was also *not* removed from ze02 :/ | 04:48 |
| *** hwoarang has quit IRC | 04:56 | |
| *** hwoarang has joined #openstack-infra | 04:56 | |
| *** pcaruana has joined #openstack-infra | 05:00 | |
| *** michael-beaver has quit IRC | 05:05 | |
| *** igordc has joined #openstack-infra | 05:15 | |
| *** kjackal has joined #openstack-infra | 05:23 | |
| *** ricolin has quit IRC | 05:25 | |
| *** ricolin has joined #openstack-infra | 05:28 | |
| *** quiquell has joined #openstack-infra | 05:33 | |
| *** factor has joined #openstack-infra | 05:37 | |
| *** ykarel|away is now known as ykarel | 05:37 | |
| *** stakeda has joined #openstack-infra | 05:57 | |
| *** igordc has quit IRC | 06:01 | |
| *** lpetrut has joined #openstack-infra | 06:02 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 06:02 |
| *** armax has quit IRC | 06:02 | |
| *** kjackal has quit IRC | 06:04 | |
| *** hwoarang has quit IRC | 06:05 | |
| *** hwoarang has joined #openstack-infra | 06:05 | |
| *** ccamacho has quit IRC | 06:05 | |
| *** slaweq has joined #openstack-infra | 06:06 | |
| AJaeger | ianw: did you see the readthedocs gerrit-dash-creator question in backscroll? Do you have access and can fix it? | 06:14 |
| ianw | AJaeger: hrm ... change looks good; did we figure out why it's blank? | 06:19 |
| *** efried1 has joined #openstack-infra | 06:21 | |
| *** efried has quit IRC | 06:23 | |
| *** efried1 is now known as efried | 06:23 | |
| AJaeger | ianw: no idea. Do you have access to the account? | 06:24 |
| AJaeger | ianw: is that a fallout from the rename of repos - openstack/g-d-c -> x/... ? | 06:24 |
| AJaeger | or python3 and sphinx? Let me check... | 06:25 |
| ianw | ahhh,could be ... | 06:25 |
| ianw | sorry, can look at logs and stuff in a few hours | 06:26 |
| *** kjackal has joined #openstack-infra | 06:26 | |
| *** ricolin has quit IRC | 06:26 | |
| AJaeger | mmh, http://logs.openstack.org/82/661982/1/check/openstack-tox-docs/13fc4c3/html/ looks fine... | 06:26 |
| AJaeger | ianw: no urgency - just don't want it forgotten. | 06:26 |
| AJaeger | Thanks, ianw | 06:26 |
| *** hwoarang has quit IRC | 06:34 | |
| yoctozepto | hello, infra team! can wiki username be changed? I wanted to be yoctozepto everywhere, yet wiki gave me no choice and made me Radoslaw.piliszek (which I don't like, especially due to wrong case of surname) | 06:34 |
| *** hwoarang has joined #openstack-infra | 06:37 | |
| *** ccamacho has joined #openstack-infra | 06:43 | |
| *** roman_g has joined #openstack-infra | 06:45 | |
| *** jtomasek has joined #openstack-infra | 06:54 | |
| *** ianychoi has joined #openstack-infra | 06:57 | |
| *** ginopc has joined #openstack-infra | 07:00 | |
| *** markvoelker has quit IRC | 07:01 | |
| *** markvoelker has joined #openstack-infra | 07:01 | |
| *** markvoelker has quit IRC | 07:06 | |
| *** ramishra has quit IRC | 07:08 | |
| *** dchen has quit IRC | 07:08 | |
| *** tobias-urdin has quit IRC | 07:09 | |
| *** jpich has joined #openstack-infra | 07:09 | |
| *** iokiwi has quit IRC | 07:10 | |
| *** adriant has quit IRC | 07:10 | |
| *** rcernin has quit IRC | 07:10 | |
| *** adriant has joined #openstack-infra | 07:10 | |
| *** iokiwi has joined #openstack-infra | 07:10 | |
| *** ccamacho has quit IRC | 07:16 | |
| *** ccamacho has joined #openstack-infra | 07:16 | |
| *** ramishra has joined #openstack-infra | 07:19 | |
| *** xek_ has joined #openstack-infra | 07:23 | |
| *** ricolin has joined #openstack-infra | 07:23 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: executor: run cleanup playbook on stop https://review.opendev.org/661881 | 07:27 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: docs: add cleanup-run documentation https://review.opendev.org/662147 | 07:27 |
| *** iurygregory has joined #openstack-infra | 07:28 | |
| *** n-saito has joined #openstack-infra | 07:33 | |
| *** jpena|off is now known as jpena | 07:36 | |
| *** toabctl has quit IRC | 07:50 | |
| *** priteau has joined #openstack-infra | 07:53 | |
| *** roman_g has quit IRC | 07:59 | |
| *** lucasagomes has joined #openstack-infra | 08:04 | |
| *** ccamacho has quit IRC | 08:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add constraints file https://review.opendev.org/662156 | 08:06 |
| *** rossella_s has joined #openstack-infra | 08:13 | |
| *** Lucas_Gray has joined #openstack-infra | 08:29 | |
| *** pkopec has joined #openstack-infra | 08:29 | |
| *** tkajinam has quit IRC | 08:33 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| *** derekh has quit IRC | 08:38 | |
| *** derekh has joined #openstack-infra | 08:39 | |
| *** gfidente has joined #openstack-infra | 08:44 | |
| *** markvoelker has joined #openstack-infra | 09:02 | |
| *** ricolin has quit IRC | 09:09 | |
| *** kopecmartin|off is now known as kopecmartin | 09:17 | |
| *** panda is now known as panda|ruck | 09:17 | |
| *** ociuhandu has joined #openstack-infra | 09:20 | |
| *** Lucas_Gray has quit IRC | 09:32 | |
| *** yamamoto has quit IRC | 09:35 | |
| *** markvoelker has quit IRC | 09:35 | |
| openstackgerrit | Slawek Kaplonski proposed zuul/zuul-jobs master: Add role to fetch journal log from test node https://review.opendev.org/643733 | 09:37 |
| *** Lucas_Gray has joined #openstack-infra | 09:39 | |
| *** electrofelix has joined #openstack-infra | 09:44 | |
| *** e0ne has joined #openstack-infra | 09:52 | |
| *** ykarel is now known as ykarel|lunch | 09:52 | |
| *** factor has quit IRC | 09:53 | |
| *** pkopec has quit IRC | 09:59 | |
| *** stakeda has quit IRC | 10:05 | |
| *** jpich has quit IRC | 10:05 | |
| *** jpich has joined #openstack-infra | 10:06 | |
| *** pkopec has joined #openstack-infra | 10:07 | |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Resolve some issue with tox.ini, setup.cfg https://review.opendev.org/660297 | 10:18 |
| *** yamamoto has joined #openstack-infra | 10:25 | |
| *** yamamoto has quit IRC | 10:25 | |
| *** yamamoto has joined #openstack-infra | 10:25 | |
| *** yamamoto has quit IRC | 10:30 | |
| *** ykarel|lunch is now known as ykarel | 10:30 | |
| *** kjackal has quit IRC | 10:32 | |
| *** factor has joined #openstack-infra | 10:37 | |
| *** nicolasbock has joined #openstack-infra | 10:39 | |
| openstackgerrit | Slawek Kaplonski proposed zuul/zuul-jobs master: Add role to fetch journal log from test node https://review.opendev.org/643733 | 10:42 |
| *** kjackal has joined #openstack-infra | 10:47 | |
| *** yamamoto has joined #openstack-infra | 10:51 | |
| *** jonher has joined #openstack-infra | 10:57 | |
| *** jpena is now known as jpena|lunch | 11:02 | |
| *** Lucas_Gray has quit IRC | 11:05 | |
| *** rcernin has joined #openstack-infra | 11:10 | |
| *** dave-mccowan has joined #openstack-infra | 11:11 | |
| *** _erlon_ has joined #openstack-infra | 11:51 | |
| *** Lucas_Gray has joined #openstack-infra | 11:56 | |
| *** rh-jelabarre has joined #openstack-infra | 11:57 | |
| *** lseki has joined #openstack-infra | 11:58 | |
| *** kaiokmo has quit IRC | 12:00 | |
| *** priteau has quit IRC | 12:01 | |
| *** ykarel is now known as ykarel|afk | 12:01 | |
| *** priteau has joined #openstack-infra | 12:03 | |
| *** n-saito has quit IRC | 12:05 | |
| *** rcernin has quit IRC | 12:10 | |
| *** tosky has joined #openstack-infra | 12:14 | |
| *** Lucas_Gray has quit IRC | 12:18 | |
| *** Lucas_Gray has joined #openstack-infra | 12:18 | |
| openstackgerrit | Merged opendev/git-review master: Build docs using Python 3 https://review.opendev.org/661750 | 12:19 |
| slaweq | hi infra-root, do You know about errors like http://logs.openstack.org/15/661915/2/check/openstack-tox-py27/c73f404/job-output.txt.gz#_2019-05-30_11_05_05_964258 ? | 12:20 |
| slaweq | I noticed it couple of times at least | 12:20 |
| *** eernst has joined #openstack-infra | 12:20 | |
| *** rh-jelabarre has quit IRC | 12:23 | |
| *** sreejithp has joined #openstack-infra | 12:24 | |
| *** jpena|lunch is now known as jpena | 12:25 | |
| *** niceplace has quit IRC | 12:26 | |
| *** niceplace has joined #openstack-infra | 12:29 | |
| *** yamamoto has quit IRC | 12:29 | |
| *** sreejithp has quit IRC | 12:30 | |
| *** eernst has quit IRC | 12:31 | |
| fungi | slaweq: in that example it's trying to use http://DFW.rax.openstack.org/pypi/simple and that's not in dns | 12:31 |
| fungi | i suspect something's causing it to assemble the hostname of the mirror incorrectly | 12:32 |
| *** sshnaidm|off has quit IRC | 12:32 | |
| *** rlandy has joined #openstack-infra | 12:33 | |
| AJaeger | fungi: is https://review.opendev.org/#/c/661187/2/zuul/site-variables.yaml the culprit? | 12:33 |
| fungi | yoctozepto: the first time you logged in, if you checked the "username" checkbox on the ubuntu one personal data request page the wiki would have been set up with the same username as you used on launchpad. since you didn't, it concatenated your full name with a . instead; as for capitalization, mediawiki usernames are case-insensitive but it will always display them with the first character | 12:35 |
| fungi | capitalized and all the rest in lower case; according to https://www.mediawiki.org/wiki/Help:Preferences#Basic_information changing someone's username requires a plugin we don't presently have installed | 12:35 |
| *** pkopec has quit IRC | 12:35 | |
| slaweq | fungi: maybe, but it happens in many jobs I think so maybe it is some more generic issue there? That's why I asked :) | 12:35 |
| openstackgerrit | Merged opendev/git-review master: Fix complimentary typo https://review.opendev.org/661453 | 12:36 |
| *** priteau has quit IRC | 12:36 | |
| fungi | AJaeger: good catch. if it's only jobs running in rax-dfw hitting this error, then quite likely | 12:36 |
| fungi | seems that merged a little over 16 hours ago | 12:37 |
| AJaeger | slaweq: is that only happening in rax-dfw? | 12:37 |
| *** pkopec has joined #openstack-infra | 12:38 | |
| AJaeger | still wonder why DFW is capitalized... | 12:38 |
| fungi | oh, actually i see the error in the script, and it looks like it should affect everything | 12:39 |
| fungi | also i see why it ceased to be capitalized | 12:39 |
| slaweq | AJaeger: according to http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22stdout%3A%20New%20python%20executable%20in%20%2Fusr%2Fzuul-env%2Fbin%2Fpython2%5C%22 it looks that it happens on many different providers | 12:40 |
| fungi | hrm, though also it's ending up with openstack.org instead of opendev.org there | 12:40 |
| fungi | likely due to case-sensitive matching | 12:40 |
| AJaeger | indeed, lower is missing in the change ;( | 12:41 |
| fungi | that jinja is matching against dfw.rax but is constructing DFW.rax | 12:41 |
| *** aaronsheffield has joined #openstack-infra | 12:41 | |
| fungi | which i think explains the lack of opendev | 12:41 |
| AJaeger | should we revert or do you have an idea for a fix? | 12:42 |
| fungi | i'll get the fix pushed up real quick and we can expedite | 12:42 |
| AJaeger | thanks! | 12:42 |
| *** pkopec has quit IRC | 12:43 | |
| AJaeger | fungi: it's also missing "mirror", isn't it? | 12:43 |
| AJaeger | shame on me for not noticing during review ;( | 12:43 |
| *** ramishra_ has joined #openstack-infra | 12:44 | |
| *** ramishra has quit IRC | 12:46 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Fix mirror hostnames after opendev migration test https://review.opendev.org/662201 | 12:48 |
| fungi | AJaeger: ^ if that looks good i'll bypass ci testing on it since it's probably going to fail jobs otherwise | 12:48 |
| AJaeger | fungi: I've +2A, let's try ;) | 12:49 |
| AJaeger | thanks | 12:49 |
| *** sshnaidm has joined #openstack-infra | 12:49 | |
| *** michael-beaver has joined #openstack-infra | 12:50 | |
| mordred | fungi: wow - that's a fun one. +2 from me too - although I'm not actually here | 12:50 |
| * AJaeger needs to step out for a bit now... | 12:51 | |
| openstackgerrit | Merged openstack/project-config master: Fix mirror hostnames after opendev migration test https://review.opendev.org/662201 | 12:52 |
| *** rfarr_ has joined #openstack-infra | 12:53 | |
| *** Lucas_Gray has quit IRC | 12:54 | |
| *** priteau has joined #openstack-infra | 12:59 | |
| *** Lucas_Gray has joined #openstack-infra | 13:00 | |
| *** mriedem has joined #openstack-infra | 13:02 | |
| *** eharney has joined #openstack-infra | 13:05 | |
| *** rfarr__ has joined #openstack-infra | 13:10 | |
| *** pkopec has joined #openstack-infra | 13:11 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Build a slack integration https://review.opendev.org/662208 | 13:11 |
| *** pkopec has quit IRC | 13:11 | |
| *** ykarel|afk is now known as ykarel | 13:11 | |
| *** rfarr_ has quit IRC | 13:13 | |
| *** happyhemant has joined #openstack-infra | 13:16 | |
| fungi | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22no%20address%20associated%20with%20hostname%5C%22 | 13:19 |
| fungi | the past 24 hour view clearly shows it starting up around 03:30z | 13:20 |
| AJaeger | indeed | 13:21 |
| fungi | the lag between 20:05z when 661187 merged until 03:30z when the first of those failures appeared seems to be related to ianw getting the project-config state unstuck on the zuul servers | 13:22 |
| *** priteau has quit IRC | 13:23 | |
| *** pcaruana has quit IRC | 13:23 | |
| AJaeger | why did it work sometimes? I would have expected failures everywhere... | 13:23 |
| AJaeger | do we have some hosts where both X and mirror.X works? | 13:24 |
| fungi | if you turn on the zuul_executor column you'll see why | 13:25 |
| fungi | my guess is we're seeing ~1/12 of our jobs impacted | 13:25 |
| clarkb | fungi: AJaeger not really here yet but ianw was struggling to get that applied due to yhe old project config vheckouy on executors | 13:25 |
| fungi | yep | 13:25 |
| fungi | caught that from his messages in scrollback | 13:25 |
| fungi | so the evidence coupled with what ianw was saying in here suggests that ze02-12 are still stuck on a much older project-config state | 13:26 |
| *** Goneri has joined #openstack-infra | 13:26 | |
| clarkb | ya | 13:27 |
| *** sreejithp has joined #openstack-infra | 13:27 | |
| *** sreejithp_ has joined #openstack-infra | 13:28 | |
| AJaeger | fungi: I see now - missed that in scrollback... | 13:29 |
| *** yamamoto has joined #openstack-infra | 13:29 | |
| fungi | and /etc/project-config/zuul/site-variables.yaml on ze01 updated with the fix at 13:07z so hopefully we see this problem vanish as of shortly after that (any jobs started before may take a few minutes to hit the problem and wrap up and report and get indexed into logstash, but that's probably all happened by now) | 13:29 |
| clarkb | I wonder if we got the path wrong on the ansible side fix | 13:30 |
| clarkb | so it still isnt removing the file because it cant find it | 13:30 |
| AJaeger | but this also means we still have a problem with project-config checkout on ze02-ze12 and should fix that before the project renames - or before we merge anything... | 13:30 |
| clarkb | AJaeger: fungi the manual fix is to clear out the ansible puppet facter json file on thosehosts (zm* too) | 13:31 |
| *** sreejithp has quit IRC | 13:32 | |
| *** sreejithp_ has quit IRC | 13:33 | |
| clarkb | but also not sure it matters for the executors or mergers except for corner cases like site variables | 13:34 |
| AJaeger | config-core, please put https://review.opendev.org/662156 on your review queue - we missed for a couple of post jobs to provide the constraints file | 13:37 |
| AJaeger | fungi, no new failures for the problem since 20 minutes... | 13:39 |
| *** sreejithp has joined #openstack-infra | 13:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use retired ACL everywhere https://review.opendev.org/662223 | 13:46 |
| *** pcaruana has joined #openstack-infra | 13:47 | |
| *** dciabrin has joined #openstack-infra | 13:47 | |
| clarkb | thinking more about this it would be good to fix it today if we can just so there is one less thing to worry about | 13:48 |
| clarkb | I will double check file paths when Im at computer | 13:48 |
| *** dciabrin_ has quit IRC | 13:50 | |
| *** boden has joined #openstack-infra | 13:50 | |
| boden | hi... with the env markers for sphinx, I now can't run tox targets locally due to not finding a marker for sphinx http://paste.openstack.org/show/752303/ seems like the venv doesn't have a python_version marker set, but I can't seem to resolve | 13:52 |
| boden | this does not happen in the gate; only locally | 13:52 |
| boden | any ideas how to resolve? | 13:53 |
| boden | sorry... false alarm please disregard ^^ I found the issue | 13:55 |
| AJaeger | no worries, boden - happy to hear you found the issue | 13:56 |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Follow ci-sandbox rename https://review.opendev.org/662225 | 13:58 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Follow rename of sandbox repos https://review.opendev.org/662226 | 13:58 |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Follow opendev renames https://review.opendev.org/662225 | 14:02 |
| AJaeger | fungi, clarkb, the change above needs to merge after the opendev rename - otherwise we have broken Zuul config ^ | 14:02 |
| * AJaeger adds to etherpad | 14:02 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use retired ACL everywhere https://review.opendev.org/662223 | 14:09 |
| AJaeger | infra-root (question for later): can we retire infra puppet 3 jobs? See https://review.opendev.org/662100 | 14:12 |
| clarkb | AJaeger: yes I think we can | 14:12 |
| *** chandankumar is now known as raukadah | 14:13 | |
| *** rh-jelabarre has joined #openstack-infra | 14:14 | |
| *** yamamoto has quit IRC | 14:16 | |
| AJaeger | clarkb: ok - still gave -1 since it removes too much... | 14:16 |
| *** armax has joined #openstack-infra | 14:23 | |
| *** yamamoto has joined #openstack-infra | 14:25 | |
| *** yamamoto has quit IRC | 14:25 | |
| *** pkopec has joined #openstack-infra | 14:25 | |
| *** ykarel is now known as ykarel|away | 14:25 | |
| *** yamamoto has joined #openstack-infra | 14:26 | |
| fungi | sean-k-mooney: the trick with out of date gerrit dependencies is that it insists that the parent sha(s) of your change must merge to the branch history before that change can merge. if you update a parent change which is still open, then its previous sha is never going to be in the branch history of the depending change | 14:29 |
| *** dpawlik has quit IRC | 14:29 | |
| sean-k-mooney | fungi: does it work after teh parent has merged? bug if not thats fine too | 14:29 |
| fungi | having a change whose parent is an out of date head can be fine, on the other hand, as long as there are no merge conflicts, because its parent sha does exist in the branch history | 14:30 |
| sean-k-mooney | i try to avoid merge commit so i rebase before i push | 14:30 |
| *** yamamoto has quit IRC | 14:30 | |
| fungi | the way our gerrit is set up, the sha is not altered when a change merges (gerrit's cherry-pick method would cause that, but we use the merge method) | 14:31 |
| *** ykarel|away has quit IRC | 14:32 | |
| sean-k-mooney | right ok good to know | 14:32 |
| fungi | and yeah, pushing a merge commit yourself is possible if the acls are set to allow it (we disable that by default but do turn it on for folks who want to merge to and from feature branches) | 14:32 |
| fungi | but merge commits are better avoided because they make it waaay too easy to accidentally pollute the repository with a bunch of stray commits | 14:33 |
| sean-k-mooney | personally i prefer the workflow of local rebases before you push for review to keep the tree clean | 14:33 |
| fungi | causing you to either accidentally push hundreds of changes or to accidentally merge all the state from master into a stable branch | 14:34 |
| sean-k-mooney | fungi: ya we have some code in the repos that were nver actully merged due to issues like that | 14:34 |
| sean-k-mooney | for example there is a patch that addes sriov nic agent supprot to devestack that are in the github repo but not in the master or stable branches | 14:36 |
| openstackgerrit | Colleen Murphy proposed openstack/openstack-zuul-jobs master: Update puppet jobs to default to puppet 4 https://review.opendev.org/662100 | 14:39 |
| fungi | well, gerrit also replicates commits (and associated named refs) for changes which are in review and not yet merged | 14:40 |
| fungi | so that's not entirely unexpected, unless i'm misunderstanding | 14:41 |
| AJaeger | cmurphy: sorry, I apparently need new glasses ;( | 14:41 |
| cmurphy | AJaeger: no worries :) | 14:41 |
| corvus | sean-k-mooney: i try to avoid local rebases unless absolutely necessary because they make it difficult to see differences between patchsets -- if they are necessary, the best way to keep inter-patchset diffs usable is to do the rebase as a patchset, leave a comment indicating it was merely a rebase, then make any necessary changes in separate patchsets. | 14:42 |
| sean-k-mooney | corvus: im aware of that approch too | 14:46 |
| sean-k-mooney | it depend on how update my current parent is vs master | 14:46 |
| corvus | yeah, one size does not fit all :) | 14:47 |
| *** pkopec has quit IRC | 14:48 | |
| paladox | fungi i think that you have to configure the replication plugin to do refs/*:refs/* (at least we have refs/head/* at the wmf). | 14:51 |
| AJaeger | clarkb: could you approve this infra-specs change, please? That will change the publish location for the docs, I'll send a followup... https://review.opendev.org/#/c/659285/2 | 14:52 |
| *** ricolin has joined #openstack-infra | 14:53 | |
| fungi | paladox: yeah, we've done both at different times | 14:53 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update infra-specs URL https://review.opendev.org/662239 | 14:54 |
| AJaeger | clarkb: merge https://review.opendev.org/#/c/659290/1 as well, please | 14:55 |
| paladox | fungi i guess having it push refs/* was painful (if it went to github). | 14:55 |
| fungi | it does mean a lot more commits getting replicated, but not sure why that's necessarily painful | 14:56 |
| fungi | it can be useful to refer to commits for not-yet-merged changes (the "gitweb" integration in gerrit) | 14:57 |
| *** lpetrut has quit IRC | 14:57 | |
| paladox | github had a bug that prevented large repos pushing | 14:57 |
| fungi | ahh | 14:57 |
| paladox | it was painful for us to push mediawiki :P | 14:57 |
| fungi | we've run into repos with certain states gh wouldn't allow us to push. particularly "very large" commits (its max commit size is smaller than gerrit's default) | 14:58 |
| paladox | chad created a script that pushed a small amount of refs for each loop. | 14:58 |
| *** imacdonn has quit IRC | 14:58 | |
| paladox | though this bug should be fixed now :) | 14:58 |
| *** pkopec has joined #openstack-infra | 15:00 | |
| *** zbr_ has joined #openstack-infra | 15:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Follow rename of sandbox repos https://review.opendev.org/662226 | 15:06 |
| *** zbr has quit IRC | 15:09 | |
| clarkb | ok zuul-scheduler also has a stale (but not as stale) project config ref ansible fact file. On closer inspection I think this is a result of the run_all.sh/base.yaml split up as zuul-scheduler is no longer part of the puppet-git playbook | 15:10 |
| clarkb | the facts look correct on review.o.o where they should be up to date | 15:11 |
| clarkb | I think that points to us wanting to fix the role properly so it clears out those files as thsi is the second case of us being unexpectedly stale | 15:13 |
| clarkb | that said I'm not seeing any obvious reasons for why the role update isn't clearing those files out | 15:13 |
| *** quiquell has quit IRC | 15:19 | |
| *** e0ne has quit IRC | 15:23 | |
| *** gyee has joined #openstack-infra | 15:28 | |
| *** pcaruana has quit IRC | 15:33 | |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Retire aeromancer https://review.opendev.org/657054 | 15:37 |
| clarkb | nothing I can find in the bridge ansible logs that might explain the lack of file removals (eg no exceptions or errors running puppet on ze01/2/3/etc) | 15:38 |
| *** roman_g has joined #openstack-infra | 15:38 | |
| clarkb | makes me think we are skipping the unlink call entirely due to the conditionals somehow | 15:38 |
| clarkb | ok I think I may know why this happens | 15:44 |
| *** tosky has quit IRC | 15:44 | |
| clarkb | ansible cache bites again (need to confirm but basically we probably have those fact values cached and then when ansible runs it uses the cache and so it "writes" the file out) | 15:44 |
| clarkb | maybe | 15:44 |
| *** Lucas_Gray has quit IRC | 15:45 | |
| clarkb | ya facter_project_config_ref shows up in the fact cache for ansible nodes | 15:45 |
| clarkb | does it never clear its cache? | 15:45 |
| clarkb | pabelanger: dmsimard ^ any tips on how to manage an ansible fact cache? maybe facts are considered to never change in ansible? | 15:45 |
| pabelanger | fact_caching_timeout should work | 15:46 |
| dmsimard | yeah there's an expiration/timeout setting | 15:46 |
| clarkb | cool I'll look into that, thanks | 15:47 |
| pabelanger | but, I've also resorted to deleting facts manually between ansible-playbook crontab | 15:47 |
| *** pkopec has quit IRC | 15:47 | |
| clarkb | ya I think that is going to be my next step | 15:47 |
| clarkb | infra-root if I delete the cached facts for ze02.openstack.org we should see if that causes it to delete the ansible provided puppet facts | 15:48 |
| clarkb | any objection to me doing that on bridge.o.o against ze02.o.o? | 15:48 |
| clarkb | I would be deleting /var/cache/ansible/facts/ze02.openstack.org | 15:52 |
| AJaeger | ianw, if you look at gerrit-dash-creator publishing, please review https://review.opendev.org/662251 as well | 15:52 |
| fungi | clarkb: i say go for it | 15:54 |
| clarkb | ok doing that now | 15:55 |
| clarkb | done | 15:55 |
| clarkb | now we wait and see what happens on ze02 | 15:55 |
| openstackgerrit | Merged opendev/infra-specs master: Move from oslosphinx to openstackdocstheme https://review.opendev.org/659290 | 15:57 |
| clarkb | AJaeger: ^ thank you for pointing those changes out | 15:57 |
| openstackgerrit | Merged opendev/infra-specs master: Fix reference to StoryBoard project https://review.opendev.org/659285 | 15:57 |
| AJaeger | clarkb: thanks for merging! | 15:58 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Update puppet jobs to default to puppet 4 https://review.opendev.org/662100 | 15:59 |
| clarkb | infra-root can we get https://review.opendev.org/#/c/662058/ in today before we rename projects tomorrow? This adds zuul secrets key moves to the rename playbook | 16:06 |
| *** rfolco has quit IRC | 16:06 | |
| fungi | ooh, good idea | 16:06 |
| clarkb | any other items of business to figure out before tomorrows renames? | 16:07 |
| AJaeger | clarkb: looking at https://review.opendev.org/661845 and https://review.opendev.org/662097 - those are both fine but I would like to use the same "human readable" yaml format. Want to change the first one? | 16:07 |
| AJaeger | We should get both in as well IMHO | 16:07 |
| clarkb | AJaeger: ya I can update the first one. What was that yaml lib that will pretty print things? ruamel? | 16:08 |
| AJaeger | yes ,ruamel | 16:08 |
| *** lucasagomes has quit IRC | 16:08 | |
| clarkb | I'll work on that once I've confirmed thsi fix for the fact thing is the fix | 16:09 |
| clarkb | puppet should run soon on ze02 | 16:09 |
| clarkb | indeed it just ran /me checks project-config there | 16:09 |
| clarkb | ok that didn't fix it, but gives us more clues. The new cache file for ze02 has the same old project config ref | 16:11 |
| *** rfolco has joined #openstack-infra | 16:11 | |
| clarkb | that is entirely unexpected | 16:11 |
| fungi | clarkb: pyyaml can pretty-print things too, you just have to pass it options for how you want the dump done | 16:11 |
| * fungi gets an example | 16:12 | |
| clarkb | in the ansible cache only zuul nodes and review.o.o have that fact | 16:14 |
| clarkb | grepping system-config we should only set the fact on the remote_puppet_git playbook which does not include zuul anymore | 16:15 |
| fungi | clarkb: this is how i get one of my programs to dump yaml which yamllint will approve of: http://paste.openstack.org/show/752310/ | 16:15 |
| clarkb | fungi: cool | 16:15 |
| corvus | clarkb, fungi: -1 on 662058 | 16:15 |
| fungi | corvus: oops, thanks!!! | 16:16 |
| *** electrofelix has quit IRC | 16:16 | |
| clarkb | anyone else have ideas on where those stale ansible vars are coming from? | 16:16 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Project renames include keys on zuul scheduler https://review.opendev.org/662058 | 16:17 |
| fungi | corvus: clarkb: ^ | 16:17 |
| clarkb | interestinly ze01 does have that same ref in its cache but it doesn't rewrite the file on the host | 16:17 |
| clarkb | so where is the state here getting all messed up? | 16:18 |
| openstackgerrit | Merged zuul/nodepool master: Add error handling when cleaning up resources https://review.opendev.org/661866 | 16:18 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 16:20 |
| clarkb | ok I think this may be cyclic | 16:21 |
| clarkb | facter_project_config_ref in the cache is ansible reading facter's facts | 16:21 |
| clarkb | now I don't think it pipes that back into the ansible rune as just project_config_ref. That means the cache values don't explain why we fail to remove the file | 16:22 |
| *** electrofelix has joined #openstack-infra | 16:22 | |
| AJaeger | publishing of infra-specs failed, error is not shown, see http://logs.openstack.org/9c/9c282d47babb1eb8a1e424195268bb69aa00660d/post/publish-openstack-specs/47f2cee/ara-report/result/0fef56d7-9607-4960-963e-43a9b32b9c16/ | 16:22 |
| clarkb | AJaeger: do we need to make an opendev/ spot for it on the docs afs volume? | 16:23 |
| AJaeger | is it a problem that the opendev directory does not exist on the server? http://specs.openstack.org/opendev/ gives 404, so maybe an mkdir is missing? | 16:23 |
| clarkb | ya that is my guess | 16:23 |
| AJaeger | clarkb: that's my guess... | 16:23 |
| AJaeger | ;) | 16:23 |
| clarkb | I think I'm back to some bug in the role change to remove that file now | 16:23 |
| AJaeger | fungi, do you have time to create the opendev dir and re-enque the publish job, please? | 16:24 |
| clarkb | I'm going to clean up my yaml now so that I can come back to the ansible things with fresh eyes a bit later | 16:24 |
| corvus | AJaeger, clarkb: perhaps we should move that to docs.opendev.org instead? | 16:24 |
| corvus | AJaeger, clarkb: opendev/base-jobs publishes there | 16:25 |
| corvus | (probably need to move the repo to the opendev tenant for that) | 16:25 |
| corvus | (or maybe not, i'm not sure) | 16:25 |
| *** roman_g has quit IRC | 16:25 | |
| fungi | AJaeger: i may be able to later today, but i agree docs.opendev.org would be nicer | 16:25 |
| AJaeger | corvus: https://docs.opendev.org/ needs first some more love ;) | 16:25 |
| corvus | AJaeger: how so? | 16:26 |
| AJaeger | corvus: there's no index page on https://docs.opendev.org/ - just open in browser | 16:26 |
| corvus | i see an index | 16:26 |
| corvus | it's not visually attractive, but it's an index | 16:26 |
| AJaeger | an index yes - but no real content, so no index.html | 16:27 |
| corvus | right, i have no idea what an index.html there would contain, but i don't see why that should stop us from publishing infra-specs there | 16:27 |
| AJaeger | I'm fine moving infra-specs to opendev - do we want that tomorrow or wait? | 16:28 |
| corvus | i've been handing out this url quite a bit: https://docs.opendev.org/opendev/base-jobs/latest/ | 16:28 |
| AJaeger | yeah, that works... | 16:28 |
| corvus | i think we can move it any time? i don't think it has to wait until tomorrow or after -- it's just switching the publishing job afaik... | 16:29 |
| AJaeger | corvus: indeed, just create / update the jobs. I can work on that... | 16:30 |
| corvus | \o/ | 16:30 |
| mnaser | is there any docs on how to setup GitHub replication / moving a project out of x/ | 16:32 |
| AJaeger | corvus: should I add the job to opendev/base-jobs? | 16:32 |
| AJaeger | mnaser: there was an email by dmsimard . Is that something to add to the infra-manual? | 16:33 |
| clarkb | AJaeger: remote: https://review.opendev.org/661845 Record historical project renames | 16:33 |
| clarkb | its updated | 16:33 |
| AJaeger | thanks, clarkb | 16:33 |
| clarkb | mnaser: dmsimard wrote an excellent email to the openstack-discuss list on how to set up github replication. Let me dig up that list. Then moving a project out of x/ follows our normal rename procedures | 16:34 |
| corvus | AJaeger: opendev/base-jobs already has that job | 16:34 |
| AJaeger | corvus: to use it, I would need to move infra-specs to opendev tenant, correct? | 16:35 |
| clarkb | mnaser: https://docs.openstack.org/infra/manual/creators.html#project-renames and http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html | 16:35 |
| corvus | AJaeger: i think opendev/base-jobs is in the openstack tenant, so i think we may be able to use the job even in the current tenant | 16:36 |
| corvus | AJaeger: of course, we eventually want to move it to the opendev tenant... | 16:36 |
| *** ccamacho has joined #openstack-infra | 16:36 | |
| AJaeger | opendev/base-jobs is only in opendev and zuul tenants | 16:37 |
| AJaeger | wrong - it's everywhere | 16:38 |
| AJaeger | will work on it later today | 16:38 |
| AJaeger | thanks, corvus | 16:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 16:39 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Publish infra-specs on docs.opendev.org https://review.opendev.org/662263 | 16:47 |
| AJaeger | corvus: ^ | 16:48 |
| *** lpetrut has joined #openstack-infra | 16:48 | |
| clarkb | I think I'm back to: either the path we generate in the role's cleanup section is wrong or the os.unlink is silently failing | 16:51 |
| clarkb | I do notice we'll write to a different path if not running as root (get_facter_path will return a homedir path) but I'm fairly certain we are running as root | 16:52 |
| clarkb | however ianw ran kick.sh (persumably as root) and that cleared out the file so maybe this isn't running as root somehow? | 16:52 |
| clarkb | puppet itself runs as puppet-user? | 16:53 |
| *** jpich has quit IRC | 16:58 | |
| clarkb | I have confirmed that puppet roel thing is being run with facts=None | 16:58 |
| clarkb | so the cache loop isn't complete. We are just getting ansible reading the facts from puppet there | 16:58 |
| clarkb | I suppose we could also be running in monkey patched stdlib via ansible mode and maybe os.unlink doesn't quite do what we want? | 16:59 |
| *** armax has quit IRC | 16:59 | |
| clarkb | (that seems like a stretch however knowing what I know of ansible's source not impossible) | 16:59 |
| *** derekh has quit IRC | 17:01 | |
| *** armax has joined #openstack-infra | 17:05 | |
| *** happyhemant has quit IRC | 17:05 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 17:08 |
| *** kjackal has quit IRC | 17:11 | |
| *** rkukura has quit IRC | 17:14 | |
| *** rkukura has joined #openstack-infra | 17:15 | |
| *** rkukura has quit IRC | 17:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update promote pipeline precedence https://review.opendev.org/662268 | 17:17 |
| *** armax has quit IRC | 17:17 | |
| AJaeger | corvus, fungi, clarkb, we changed the precedence of pipelines (making post high) but promote is still low, let's update ^ | 17:18 |
| *** armax has joined #openstack-infra | 17:18 | |
| openstackgerrit | Merged opendev/system-config master: Project renames include keys on zuul scheduler https://review.opendev.org/662058 | 17:20 |
| *** kopecmartin is now known as kopecmartin|off | 17:28 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate builds with event id https://review.opendev.org/658895 | 17:28 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Log github requests with annotated events https://review.opendev.org/660800 | 17:28 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around build completion and cancellation https://review.opendev.org/660806 | 17:28 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around build states https://review.opendev.org/661489 | 17:28 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around reporting https://review.opendev.org/661490 | 17:28 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around finished builds https://review.opendev.org/661491 | 17:28 |
| *** gfidente is now known as gfidente|afk | 17:32 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 17:34 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Store autohold requests in zookeeper https://review.opendev.org/661114 | 17:34 |
| *** eernst has joined #openstack-infra | 17:35 | |
| *** ociuhandu has quit IRC | 17:37 | |
| *** armax has quit IRC | 17:38 | |
| *** armax has joined #openstack-infra | 17:38 | |
| *** ricolin has quit IRC | 17:42 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 17:44 |
| *** eernst has quit IRC | 17:46 | |
| openstackgerrit | Merged openstack/project-config master: Update promote pipeline precedence https://review.opendev.org/662268 | 17:47 |
| *** electrofelix has quit IRC | 17:48 | |
| *** Lucas_Gray has joined #openstack-infra | 17:48 | |
| *** electrofelix has joined #openstack-infra | 17:50 | |
| *** ginopc has quit IRC | 17:50 | |
| *** iurygregory has quit IRC | 17:51 | |
| *** eernst has joined #openstack-infra | 17:53 | |
| AJaeger | team, as prep for tomorrow's rename, please review https://review.opendev.org/661845 and https://review.opendev.org/662097 to record the renames | 17:54 |
| AJaeger | config-core, to fix infra-specs publishing, we took out the big hammer and moved it to docs.opendev.org, please review https://review.opendev.org/662263 (see discussion earlier) | 17:55 |
| *** Lucas_Gray has quit IRC | 17:55 | |
| *** kjackal has joined #openstack-infra | 17:57 | |
| *** eernst has quit IRC | 17:59 | |
| *** electrofelix has quit IRC | 18:03 | |
| *** jpena is now known as jpena|off | 18:13 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 18:17 |
| *** pcaruana has joined #openstack-infra | 18:24 | |
| corvus | AJaeger: can/should we move those in-tree? | 18:24 |
| corvus | (i've approved 662263 since we can still do so after it has merged) | 18:25 |
| AJaeger | corvus: we could but since we currently own both and it'S only Clarkb that approves infra-specs, I would leave it there for now... thanks for reviews | 18:26 |
| corvus | oh, heh, i forgot it's special | 18:28 |
| clarkb | I remain stumped on the project_config_ref thing | 18:29 |
| clarkb | I think my recommendation at this point (which we can run by ian in a couple hours) is that we manually remove those files where we don't want them then we can take our time debugging why the role doesn't clean up after itself | 18:30 |
| corvus | i'm in favor of that. | 18:30 |
| corvus | even more so if the time we take for the second part is nil | 18:31 |
| clarkb | also we should clear out the ansible cache for those nodes after we do that just to sanity check that any potential cycle between those really does not exist (I'm pretty sure there isn't a loop now but we can verify with hard evidence via cleanups) | 18:32 |
| openstackgerrit | Merged openstack/project-config master: Publish infra-specs on docs.opendev.org https://review.opendev.org/662263 | 18:34 |
| AJaeger | clarkb: to test the change above, here's another trivial infra-specs change - could you put that on your queue for later, please? https://review.opendev.org/#/c/631419/1 | 18:35 |
| clarkb | AJaeger: yes | 18:36 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 18:38 |
| clarkb | AJaeger: done. lets see if it publishes happily now | 18:39 |
| AJaeger | thanks, i'll check | 18:39 |
| openstackgerrit | Merged opendev/infra-specs master: Fix the misspelling of "search" https://review.opendev.org/631419 | 18:48 |
| AJaeger | #success https://docs.opendev.org/opendev/infra-specs/latest/ is published | 18:51 |
| openstackstatus | AJaeger: Added success to Success page (https://wiki.openstack.org/wiki/Successes) | 18:51 |
| clarkb | woot thanks for sorting that out | 18:51 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update infra-specs link https://review.opendev.org/662293 | 18:52 |
| AJaeger | now a few followups, first one ^ | 18:52 |
| fungi | we may want a redirect, though i have no opinion on that | 18:53 |
| openstackgerrit | Andreas Jaeger proposed opendev/infra-specs master: Update site URL https://review.opendev.org/662297 | 19:00 |
| AJaeger | and second one ^ | 19:01 |
| clarkb | fungi: I doubt that we get a ton of traffic there (and mosto f it is from peopel that can probably ask where to find it again if necessary?) | 19:01 |
| fungi | yeah, so i'm cool with not having one | 19:01 |
| fungi | just wanted to remind people that's worth thinking about | 19:02 |
| mnaser | can a 'ownership' transfer be done at anytime right now (i.e. over GitHub) | 19:02 |
| fungi | mnaser: yep | 19:02 |
| mnaser | ok cool | 19:02 |
| fungi | mnaser: also with the (very recent) changes gh has made to the transfer process, we no longer need any action taken on the receiving org in advance | 19:02 |
| clarkb | if someone else can sanity check 662297 I can approve it | 19:03 |
| fungi | they just have to accept the transfer request once it's submitted | 19:03 |
| clarkb | one nice thing about still setting up our account on both sides is it is immediate and we can confirm things worked | 19:03 |
| clarkb | I don't think we can even check if there is a request on the remote side otherwise | 19:03 |
| clarkb | AJaeger: I think you added 9 at https://etherpad.openstack.org/p/project-renames-2019-05-31 ? | 19:04 |
| clarkb | I hadn't planned on "fixing" the individual projects | 19:05 |
| clarkb | though I guess we can probably do some simple fixups like git review updates and so on | 19:05 |
| fungi | clarkb: yeah, it becomes reliant on the receiving org admins to let us know if things worked, but that seems fine to me | 19:05 |
| fungi | mainly because they're likely to care more about whether it worked than we are anyway | 19:06 |
| AJaeger | clarkb: yes, I did. Just wanted to ask. We should merge the change I pushed... | 19:06 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Store autohold requests in zookeeper https://review.opendev.org/661114 | 19:06 |
| *** pkopec has joined #openstack-infra | 19:06 | |
| *** diablo_rojo has joined #openstack-infra | 19:06 | |
| clarkb | AJaeger: https://review.opendev.org/#/c/662225/ that one? | 19:06 |
| mnaser | Could I get openstack/osops-{coda,example-configs,tools-contrib,tools-generic,tools-logging,tools-monitoring} moved to openstack-operators/{coda,example-configs,tools-contrib,tools-generic,tools-logging,tools-monitoring} ? | 19:07 |
| clarkb | AJaeger: and ya we'll need that one on our side | 19:07 |
| mnaser | I have the openstack-operators org in github | 19:07 |
| clarkb | fungi: do you know if the api call changes? | 19:07 |
| AJaeger | clarkb: yes, 662225 | 19:07 |
| *** hwoarang has quit IRC | 19:07 | |
| fungi | clarkb: mordred looked into the api docs but i don't remember if it implied any changes to the script | 19:08 |
| clarkb | fungi: ok I guess we can test it on one and see if it works | 19:08 |
| *** hwoarang has joined #openstack-infra | 19:08 | |
| clarkb | mnaser: note we'll change orgs with the same repo name then you can rename the repo in the new org | 19:08 |
| fungi | seems like maybe it's the same api method but now instead of checking whether the requester is an admin in both orgs it just submits a transfer request for the receiving org | 19:08 |
| fungi | which then goes through once they accept it | 19:08 |
| clarkb | gotcha | 19:08 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update infra-specs URL https://review.opendev.org/662298 | 19:09 |
| AJaeger | and one more update ^ | 19:09 |
| *** rosmaita has left #openstack-infra | 19:09 | |
| fungi | clarkb: mnaser: anyway, this does indeed seem like a good opportunity to find out | 19:09 |
| clarkb | mnaser: give me a few minutes ( have to log in to github and make an api key thing) but I can do one to confirm it works as we expect and then we can do the others if it does | 19:09 |
| mnaser | fungi, clarkb: cool.. I'll prepare a patch for the replication too | 19:10 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: rename x/osops- to openstack-operators/ https://review.opendev.org/662300 | 19:11 |
| mnaser | ^ can I add that to the stack that is pending for tomorrow? | 19:11 |
| AJaeger | mnaser: who owns those repos? Are those part of an official OpenStack entitity? Otherwise we might have a trademark issue... | 19:12 |
| mnaser | they are not an openstack deliverable but there is an openstack operators 'thing' that exist.. | 19:12 |
| mnaser | https://wiki.openstack.org/wiki/Operators_Tools_and_Monitoring | 19:12 |
| mnaser | part of the ops team (these deliverables).. | 19:12 |
| mnaser | but IANAL | 19:13 |
| AJaeger | mnaser: I know you're part of TC, but since TC decided to move those out of openstack namespace, putting them into another openstack namespace requires IMHO a TC approval | 19:13 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 19:13 |
| AJaeger | mnaser: did the TC discuss this? | 19:13 |
| clarkb | I'm not a lawyer but I can see how "openstack" in this case is descriptive and therefore fair game | 19:13 |
| clarkb | they are ops tools for openstack | 19:13 |
| mnaser | and its not openstack/ but openstack-operators (aka from the actual openstack operators group) | 19:14 |
| AJaeger | clarkb: I'm neither a lawyer - still, I think it needs some discussion. | 19:14 |
| * mnaser shrugs | 19:14 | |
| clarkb | mnaser: fwiw I've already generated all of the documents for tomorrows renaming :/ | 19:14 |
| corvus | yeah... i'm not sure we have a platform to "veto" that, but i can see how it would be friendly to have a tc conversation about it | 19:14 |
| mnaser | if it's that much of an issue I'll find another name instead of it living under x for the new N weeks for a rename to happen | 19:14 |
| mnaser | alright, I'll bring it up to the ML anyways | 19:14 |
| mnaser | let's hold the org transfer then.. | 19:15 |
| clarkb | ok | 19:15 |
| mnaser | s/org/repo/ | 19:15 |
| AJaeger | mnaser: what about asking the TC whether they see a issue? I don#t like to sneak this in last minute without notice | 19:15 |
| AJaeger | mnaser: commented on change, it's not correct | 19:17 |
| *** imacdonn has joined #openstack-infra | 19:17 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update infra-specs URL https://review.opendev.org/662298 | 19:20 |
| AJaeger | clarkb: had to fix sphinx 2.0.1 and python3 ^ | 19:20 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 19:24 |
| boden | AJaeger hate to nag, but is there any other core I can ask to help land https://review.opendev.org/#/c/661977/ we are waiting to get the dashboard working so we can gauge health for a release of neutron-lib | 19:25 |
| clarkb | boden: I can look | 19:26 |
| boden | clarkb thanks! | 19:26 |
| clarkb | fungi: thinking about tomorrow if you can disable the ansible cron in your early am that would probably help a bunch (reduces wait time at 1500UTC and we should be able to dive straight in) | 19:28 |
| fungi | clarkb: sure, setting a reminder for that now | 19:28 |
| clarkb | the other thing I was thinking about is with system confused about gerrit being stopped/running whatever will our playbook function? we might need to use an edited playbook and stop/start gerrit ourselves | 19:28 |
| fungi | reminder sety | 19:29 |
| fungi | er, set | 19:29 |
| fungi | yeah, if we try to systemctl restart it, systemd will think it's not running and so will also refuse to start it | 19:29 |
| fungi | or at least that's been my experience recently | 19:29 |
| fungi | on several occasions now | 19:30 |
| clarkb | in this case it will be a systemctl stop which may noop beacuse it thinks it isn't running? | 19:30 |
| fungi | yeah, that should be fine | 19:30 |
| AJaeger | clarkb: want to +2A the other dashboard change by boden as well? https://review.opendev.org/#/c/662024 | 19:30 |
| clarkb | in that case I'll add a note to the etherpad that we should manually do that and use edited playbook. After this restart we should be in a place where systemd is happy with it though due to the longer timeout | 19:30 |
| *** tosky has joined #openstack-infra | 19:38 | |
| *** raissa has joined #openstack-infra | 19:39 | |
| *** rlandy is now known as rlandy|brb | 19:39 | |
| openstackgerrit | Merged openstack/project-config master: fix neutron-lib grafana dashboard https://review.opendev.org/661977 | 19:40 |
| clarkb | I rarely use the unified diff view except for on changes like these | 19:41 |
| clarkb | and still my screen isn't wide enough :) | 19:41 |
| *** yamamoto has joined #openstack-infra | 19:42 | |
| *** bdodd has quit IRC | 19:44 | |
| AJaeger | yeah, those grafana updates are evil ;( | 19:44 |
| *** bdodd has joined #openstack-infra | 19:44 | |
| AJaeger | fungi, could you review https://review.opendev.org/662298, https://review.opendev.org/662293 , and https://review.opendev.org/662297 as infra-specs renames, please? | 19:45 |
| * AJaeger calls it a day | 19:45 | |
| fungi | thanks AJaeger! | 19:45 |
| clarkb | AJaeger: thanks for all the help! see you tomorrow | 19:46 |
| clarkb | boden: I've approved the other change AJaeger poitned out as well. That one took a bit longer to get through | 19:46 |
| clarkb | I guess I should do a project-confing and system-config pass through today | 19:47 |
| boden | clarkb appreciate that... unfortunately I'm still not seeing the data I expect... I'll have to dig | 19:47 |
| clarkb | boden: you have to wait for the deployment pulses to happen | 19:47 |
| clarkb | usually within 45 minutes | 19:47 |
| clarkb | (we've actually improved that number recently so may be as quick as 30 minutes in the worst case now) | 19:48 |
| boden | clarkb ah... didn't realize that | 19:48 |
| *** yamamoto has quit IRC | 19:48 | |
| *** igordc has joined #openstack-infra | 19:49 | |
| fungi | and in the not too distant future, faster still | 19:49 |
| fungi | eventually we hope to have change merges trigger deployment for basically everything | 19:50 |
| *** igordc has quit IRC | 19:50 | |
| clarkb | anyone else want to weigh in on the version of ansible we use with ansible-lint? https://review.opendev.org/#/c/659810/1 | 19:51 |
| clarkb | 2.7 is what we run with on bridge so that seems appropriate | 19:51 |
| clarkb | (its also zuul's default) | 19:51 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 19:52 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Retiring TripleO-UI https://review.opendev.org/661332 | 19:56 |
| clarkb | hopefully that fixes that chagne so we can clear it out of the list | 19:56 |
| *** lpetrut has quit IRC | 19:57 | |
| openstackgerrit | Merged openstack/project-config master: Update the neutron-lib failure rate dashboard https://review.opendev.org/662024 | 19:58 |
| openstackgerrit | Merged openstack-infra/opendev-website master: Retire repo https://review.opendev.org/661911 | 19:59 |
| clarkb | https://review.opendev.org/#/c/662156/1 seems like a relatively important job fix (to avoid builds working magically without constraints) | 20:00 |
| *** igordc has joined #openstack-infra | 20:00 | |
| clarkb | oh config-core can we avoid making changes to openstack/project-config for a bit (taht way I don't have to rebase the stack of project renames) | 20:02 |
| clarkb | er specifically to the gerrit/projects.yaml file | 20:03 |
| *** rlandy|brb is now known as rlandy | 20:08 | |
| *** hwoarang has quit IRC | 20:11 | |
| *** hwoarang has joined #openstack-infra | 20:12 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 20:12 |
| *** factor has quit IRC | 20:14 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Update axios version and yarn.lock https://review.opendev.org/662316 | 20:16 |
| *** Lucas_Gray has joined #openstack-infra | 20:19 | |
| *** hwoarang has quit IRC | 20:24 | |
| *** hwoarang has joined #openstack-infra | 20:24 | |
| *** Lucas_Gray has quit IRC | 20:24 | |
| *** Lucas_Gray has joined #openstack-infra | 20:26 | |
| *** Goneri has quit IRC | 20:27 | |
| *** rkukura has joined #openstack-infra | 20:31 | |
| *** diablo_rojo has quit IRC | 20:31 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 20:32 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove misplaced html tag https://review.opendev.org/656807 | 20:33 |
| clarkb | jhesketh: ^ rebased that for you since seems like a good cleanup | 20:34 |
| *** xek_ has quit IRC | 20:34 | |
| * clarkb is trying to get through some review backlogs | 20:34 | |
| clarkb | corvus: fungi earlier this week I remember being told that we'd updated gerrit config to use gitea links and that tomorrow's restart will pcik that up. Where does https://review.opendev.org/#/c/654034/3 fit in? | 20:37 |
| corvus | clarkb: i think you may have been misinformed and indeed that is the change which would cause that to happen | 20:38 |
| fungi | or did we set it in review.pp only? | 20:38 |
| clarkb | I seem to recall mordred specifically checking things, but my memories are known to be faulty at times :) | 20:39 |
| corvus | i recall that mordred set a WIP on that until we were happy with the result of the ref replication. i have no recollection beyond that. | 20:39 |
| clarkb | gotcha | 20:39 |
| clarkb | well maybe this is a better as a followup then | 20:39 |
| fungi | indeed, it seems https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/manifests/review.pp does not have any equivalent of 654034 applied | 20:40 |
| fungi | for some reason i thought we had approved everything and were just waiting on a gerrit restart, so could be i was circulating misinformation | 20:40 |
| corvus | "what do you want?" "misinformation." "you won't get it" "by hook or by crook, we will | 20:44 |
| clarkb | any other changes that could use review? I'm trying to dig around and stick to ones that are "safe" aka not touching projects.yaml related items | 20:51 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add opendev service-announce list https://review.opendev.org/661843 | 20:53 |
| clarkb | theres a rebase to fix a merge conflict. Not sure if we ended up with a decision on the bike shedding through | 20:53 |
| clarkb | ianw: for when your day starts: I did more digging into the project_config_ref thing and am stumped. At this point I think we should probably manually clear out those files from the zuul nodes and move on (or at least be able to debug things more slowly) | 21:02 |
| clarkb | I'm happy to do that and also happy for others to do that :) just let me know | 21:02 |
| *** rh-jelabarre has quit IRC | 21:06 | |
| openstackgerrit | Merged opendev/system-config master: Add opendev migration repo rename scripts https://review.opendev.org/653138 | 21:07 |
| fungi | also probably worth checking for jobs successfully using mirror.dfw.rax.opendev.org since we fixed the typos in the confug | 21:09 |
| fungi | er, config | 21:09 |
| *** Lucas_Gray has quit IRC | 21:15 | |
| *** pcaruana has quit IRC | 21:18 | |
| *** pkopec has quit IRC | 21:21 | |
| *** Lucas_Gray has joined #openstack-infra | 21:23 | |
| *** whoami-rajat has quit IRC | 21:24 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Update axios version and yarn.lock https://review.opendev.org/662316 | 21:27 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Use nodejs v10 in testing https://review.opendev.org/662339 | 21:27 |
| *** Wryhder has joined #openstack-infra | 21:31 | |
| *** Lucas_Gray has quit IRC | 21:32 | |
| *** Wryhder is now known as Lucas_Gray | 21:32 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Update axios version and yarn.lock https://review.opendev.org/662316 | 21:45 |
| *** sreejithp has quit IRC | 21:54 | |
| *** rkukura has quit IRC | 21:57 | |
| *** hwoarang has quit IRC | 22:05 | |
| *** hwoarang has joined #openstack-infra | 22:05 | |
| openstackgerrit | Merged opendev/system-config master: Remove misplaced html tag https://review.opendev.org/656807 | 22:06 |
| *** boden has quit IRC | 22:06 | |
| *** rfarr__ has quit IRC | 22:07 | |
| *** bobh has joined #openstack-infra | 22:10 | |
| *** Weifan has joined #openstack-infra | 22:13 | |
| clarkb | its starting to get late enough in my day that I'm wondering if we shouldn't go ahead with the manual removal of unwanted ansible puppet facts on zuul nodes (after verifying that dfw jobs on ze01 work) | 22:13 |
| *** Weifan has quit IRC | 22:14 | |
| clarkb | found a bug with the opendev mirror | 22:15 |
| *** bobh has quit IRC | 22:15 | |
| clarkb | we don't have ubuntu/ served off of it | 22:15 |
| clarkb | I'm looking into that now | 22:16 |
| *** efried1 has joined #openstack-infra | 22:19 | |
| *** efried has quit IRC | 22:20 | |
| *** efried1 is now known as efried | 22:20 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Serve ubuntu package content on opendev mirrors https://review.opendev.org/662345 | 22:21 |
| clarkb | fungi: ianw corvus ^ fix for the opendev mirror issue | 22:21 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 22:21 |
| clarkb | hrm we are also missing apt-puppetlabs | 22:22 |
| clarkb | which is I think effectively dead now anyway? | 22:22 |
| clarkb | it was only yum that tripleo was using. Maybe we can leave apt-puppetlabs off? | 22:22 |
| fungi | yeah, maybe, though in theory they temporarily "fixed" it a week and a half ago | 22:23 |
| fungi | so it's possible some jobs are still relying on it for the moment | 22:23 |
| clarkb | thinking out loud here should I manually create the symlink then remove the ansible facter file on ze* zm* and zuul01? or should we maybe stop worrying about this for now? | 22:24 |
| clarkb | trying to decide in my head if that has/should be done before the renaming tomorrow. I expect we will be fine with the renaming tomorrow so maybe just wait for opendev to be fixed then rollout to executors when more convenuent than end of my day (or maybe ianw can do that today and watch it) | 22:24 |
| fungi | i don't think it's urgent | 22:24 |
| clarkb | and for those of you still around now is probably a good time to reread the etehrpad and make sure we aren't missing anything or need to prep something https://etherpad.openstack.org/p/project-renames-2019-05-31 | 22:25 |
| clarkb | https://review.opendev.org/#/c/662058/ did merge | 22:26 |
| clarkb | I think I'll take a break now then pop back in a bit with the idea of working a little later than usual to see if I can catch ianw | 22:30 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Discard GitHub PullReview if incomplete https://review.opendev.org/662347 | 22:35 |
| *** mriedem is now known as mriedem_away | 22:35 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Discard GitHub PullReview if incomplete https://review.opendev.org/662347 | 22:37 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Discard GitHub PullReview if incomplete https://review.opendev.org/662347 | 22:38 |
| paladox | clarkb https://docs.openstack.org/infra/system-config/gerrit.html looks nice! | 22:39 |
| *** aaronsheffield has quit IRC | 22:41 | |
| fungi | it's probably a smidge out of date... we've been maintaining documentation on how we install gerrit since basically as long as we've been documenting things and probably almost as long as we've been running gerrit | 22:42 |
| fungi | we try to update it when we think of something that's changed, but we don't reinstall gerrit for production all that often | 22:42 |
| ianw | clarkb: hey, sorry, starting late but here now! | 22:42 |
| fungi | ianw: no need to apologize! welcome back and happy friday! | 22:43 |
| paladox | fungi we have docs, which are definitely gonna be out of date on our next upgrade :P (but they are not as clean as yours!) | 22:43 |
| paladox | https://wikitech.wikimedia.org/wiki/Gerrit | 22:44 |
| *** hwoarang has quit IRC | 22:44 | |
| *** hwoarang has joined #openstack-infra | 22:45 | |
| *** yamamoto has joined #openstack-infra | 22:45 | |
| *** Weifan has joined #openstack-infra | 22:46 | |
| *** ccamacho has quit IRC | 22:47 | |
| *** Weifan has quit IRC | 22:47 | |
| *** Weifan has joined #openstack-infra | 22:47 | |
| *** ccamacho has joined #openstack-infra | 22:47 | |
| *** Weifan is now known as weifan | 22:48 | |
| *** weifan has quit IRC | 22:49 | |
| clarkb | ianw tldr is need to fix ubuntu mirror on opendev mirror then I think we should just remove that facter file on ze* zm* and zuul01 unless you have more ideas | 22:49 |
| *** weifan has joined #openstack-infra | 22:49 | |
| ianw | oh, well at least the templating bit worked? :) one of out three | 22:50 |
| *** rkukura has joined #openstack-infra | 22:51 | |
| fungi | ianw: after some fixing the templating bit worked ;) | 22:51 |
| ianw | it's so weird, i ran kick.sh ze01 and it did clear out. i wonder if somehow it's a cached version of the library? | 22:51 |
| fungi | there was a ~10 hour timeframe where jobs running from ze01 couldn't find their mirrors, but that's been smooth sailing after some quick edits | 22:52 |
| ianw | fungi: OK, zero for three :) something something breaking eggs making omelettes | 22:52 |
| *** zhurong has quit IRC | 22:52 | |
| fungi | it's turning out to be a very tasty omelette | 22:53 |
| *** bobh has joined #openstack-infra | 22:54 | |
| ianw | ok, let me try kicking ze02 and see if it disappears too or i'm nuts | 22:58 |
| *** bobh has quit IRC | 22:59 | |
| ianw | the unlink is definitely in /etc/ansible/roles/puppet/roles on bridge | 23:00 |
| clarkb | ianw: one idea I had was maybe the path is wrong somehow | 23:01 |
| clarkb | get_facter_dir can return different answers | 23:01 |
| ianw | yeah, it's difficult to get debugging out of ansible libraries | 23:03 |
| ianw | ok, so the file has gone on ze02 after i ran it with kick.sh | 23:04 |
| clarkb | so weird | 23:04 |
| ianw | output in /tmp/kick-ze02.txt | 23:04 |
| clarkb | ianw: did you run that as root? | 23:04 |
| clarkb | 662345 is the fix for the ubuntu/ mirror on the opendev mriror host. It is in the gate now | 23:05 |
| *** tkajinam has joined #openstack-infra | 23:05 | |
| ianw | clarkb: sudo'd yep | 23:05 |
| clarkb | ok so it shouldn't be somehow magically working via the non uid == 0 path | 23:06 |
| ianw | http://paste.openstack.org/show/752336/ ... it seems like it gets invoked in exactly the same way | 23:10 |
| clarkb | hrm maybe the remote side is caching the role lib? | 23:10 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Update axios version and yarn.lock https://review.opendev.org/662316 | 23:13 |
| ianw | clarkb: aahhhh!!! /var/cache/ansible/facts/ze12.openstack.org | 23:13 |
| ianw | guess what's in there ... | 23:13 |
| *** ianychoi has quit IRC | 23:13 | |
| clarkb | facter_project_config_ref | 23:13 |
| clarkb | yup I went through all that earlier today | 23:13 |
| clarkb | I think it is "ok" though | 23:14 |
| clarkb | its reading the facter facts from the remote host and caching them but because it has the facter_ prefix it isn't fed back in | 23:14 |
| clarkb | (I deleted the cached facts for ze02.openstack.org and it read them right back again) | 23:14 |
| *** rlandy has quit IRC | 23:16 | |
| *** rcernin has joined #openstack-infra | 23:17 | |
| ianw | that makes sense ... do we have something that is s/facter_// and installing variables? | 23:18 |
| clarkb | I couldn't find that anywhere | 23:18 |
| clarkb | I grepped in /etc/ansible/hosts as well as system-config and the ansible fact cache | 23:18 |
| clarkb | but ya that was my next idea. Its probably worht someone else double checking on that just to make sure I didn't miss a location | 23:18 |
| ianw | but yet it is clearly invoked with facts=None | 23:20 |
| ianw | if module.params['facts']: ... that doesn't seem like it would ever come from somewhere else | 23:20 |
| clarkb | ya | 23:21 |
| ianw | we can just delete it, but damn i want to know what is actually going on here :/ | 23:22 |
| *** panda|ruck has quit IRC | 23:22 | |
| ianw | i feel like there's clearly something i'm (we're) assuming that is not holding | 23:22 |
| *** panda has joined #openstack-infra | 23:23 | |
| ianw | it *must* be old code | 23:26 |
| *** dchen has joined #openstack-infra | 23:26 | |
| openstackgerrit | Merged opendev/system-config master: Serve ubuntu package content on opendev mirrors https://review.opendev.org/662345 | 23:26 |
| clarkb | one crazy idea I had was maybe ansible monkey patches os.unlink. But I grepped around the ansibel source tree and found no evidence of that | 23:27 |
| *** stakeda has joined #openstack-infra | 23:27 | |
| ianw | clarkb: http://paste.openstack.org/show/752337/ ... these all appear to be old code | 23:29 |
| ianw | i don't know if they're leaked, or reused, or what | 23:29 |
| clarkb | oh interesting so it could be cache of the code | 23:30 |
| clarkb | and when you run it with sudo and not -H it uses your home dir? | 23:30 |
| clarkb | hrm none of those are from 2019? | 23:31 |
| ianw | i think it must, let me try kicking from a full "su -" on ze03 | 23:32 |
| clarkb | your homedir only has one dir in .ansible/tmp and it is from december | 23:32 |
| clarkb | but ya maybe things are leaking or stale and somehow found | 23:33 |
| *** tosky has quit IRC | 23:33 | |
| ianw | argggh, it disappeared on ze03! | 23:34 |
| clarkb | ok so probably not the homedir diff then | 23:34 |
| ianw | that was from kick.sh in a full "su -" so using root's homedir! what could cron be doing differently?! | 23:35 |
| fungi | cron doesn't run in a shell, it invokes executables directly | 23:37 |
| fungi | not sure if that's coming into play somehow here | 23:37 |
| fungi | the environment it runs with is also minimal to nonexistent | 23:37 |
| ianw | that invokes /opt/system-config/run_all.sh which then runs ansible-playbook under timeout | 23:38 |
| fungi | manpage says cron allows you to configure via /etc/default/cron to use environment from /etc/environment | 23:40 |
| fungi | oh, except tasks run *by* cron don't inherit those | 23:41 |
| fungi | see cron(5) | 23:41 |
| fungi | er, crontab(5) | 23:41 |
| fungi | Several environment variables are set up automatically by the cron(8) daemon. SHELL is set to /bin/sh, and LOGNAME and HOME are set from the /etc/passwd line of the crontab's owner. PATH is set to "/usr/bin:/bin". HOME, SHELL, and PATH may be overridden by settings in the crontab; LOGNAME is the user that the job is running from, and may not be changed. | 23:43 |
| fungi | so unless we're setting additional envvars within the crontab file itself, that's it | 23:44 |
| ianw | but then ... how would that relate to ansible then seeming to run old/cached code on nodes, but only under cron? it's the mystery of our times | 23:46 |
| clarkb | where does ansible stick code on the remote node? | 23:47 |
| clarkb | maybe it doesn't write it to disk? | 23:48 |
| *** yamamoto has quit IRC | 23:49 | |
| clarkb | ANSIBLE_KEEP_REMOTE_FILES can be set to 1 to keep the remote files around | 23:50 |
| *** tjgresha has joined #openstack-infra | 23:50 | |
| clarkb | looks like it copies a shim over that unzips and executes the code out of a temp dir in /tmp | 23:53 |
| clarkb | which then gets imp.load_module()'d ? | 23:55 |
| *** tjgresha has quit IRC | 23:55 | |
| *** tjgresha has joined #openstack-infra | 23:55 | |
| *** hwoarang has quit IRC | 23:58 | |
| *** hwoarang has joined #openstack-infra | 23:58 | |
| ianw | i just ... i dunno. i can just kick 0-12 ze if we want to give up on this | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!