| ianw | yeah, i think 35 jobs only run on xenial, and i think 3.5 is the worst affected by the new setuptools | 00:03 |
|---|---|---|
| openstackgerrit | Gage Hugo proposed openstack/project-config master: Add openstack/openstack-helm-deployments https://review.opendev.org/748304 | 00:52 |
| *** diablo_rojo has quit IRC | 00:54 | |
| *** DSpider has quit IRC | 01:07 | |
| *** wolsen has quit IRC | 03:04 | |
| *** bhagyashris is now known as bhagyashris|away | 05:23 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update hashicorp jobs file with correct title https://review.opendev.org/747853 | 05:43 |
| *** fressi has joined #opendev | 05:44 | |
| frickler | infra-root: py35 still seems to show the same error https://zuul.opendev.org/t/zuul/build/8c5a67e79c694844a209e75865c4549f | 06:16 |
| ianw | frickler: was that the one i just rechecked? | 06:16 |
| frickler | ianw: that was the one AJaeger amended just half an hour ago | 06:17 |
| *** lpetrut has joined #opendev | 06:33 | |
| *** andrewbonney has joined #opendev | 07:13 | |
| *** tosky has joined #opendev | 07:21 | |
| ianw | frickler: hrm, https://zuul.opendev.org/t/zuul/build/579dfd55aaea4a64abe6e61cb96cc75d passed though. maybe just an old node? | 07:27 |
| frickler | ianw: that's possible, I've wondered whether there's an easy way to find out which image a node was based on, maybe we could add that to the job output? | 07:35 |
| * AJaeger rechecks 747853 | 07:37 | |
| ianw | frickler: umm, i think it is, it dumps the dib-build.txt header or something somehwere | 07:37 |
| frickler | ianw: ah, right, top of https://8abbfecd00d9996fdb0c-5c4643ce01a9b304087712e3e08013b4.ssl.cf1.rackcdn.com/747853/2/check/tox-py35/8c5a67e/zuul-info/zuul-info.ubuntu-xenial.txt , that looks like old image indeed | 07:39 |
| ianw | https://zuul.opendev.org/t/openstack/build/44b70e26385144ae87dca0589bfde2fc/log/zuul-info/zuul-info.ubuntu-bionic.txt#4 | 07:39 |
| ianw | heh, yeah, that's it | 07:40 |
| AJaeger | yes, passed now - anybody wants to +2A 747853? | 07:50 |
| *** DSpider has joined #opendev | 07:54 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** priteau has joined #opendev | 08:03 | |
| *** fressi has quit IRC | 08:08 | |
| *** hashar has joined #opendev | 08:16 | |
| *** fressi has joined #opendev | 08:18 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Update hashicorp jobs file with correct title https://review.opendev.org/747853 | 08:21 |
| *** fressi has quit IRC | 08:31 | |
| *** sshnaidm|bbl is now known as sshnaidm | 08:36 | |
| *** fressi has joined #opendev | 08:51 | |
| *** Gyuseok_Jung has quit IRC | 09:04 | |
| *** dtantsur|afk is now known as dtantsur | 09:05 | |
| *** calcmandan has quit IRC | 09:12 | |
| *** calcmandan has joined #opendev | 09:13 | |
| *** Gyuseok_Jung has joined #opendev | 09:38 | |
| *** tkajinam has quit IRC | 09:54 | |
| *** tkajinam has joined #opendev | 09:55 | |
| *** ajya has joined #opendev | 10:24 | |
| *** mgagne has quit IRC | 10:39 | |
| openstackgerrit | Merged opendev/irc-meetings master: Change Masakari Meeting time https://review.opendev.org/747819 | 11:13 |
| openstackgerrit | Merged opendev/irc-meetings master: Change Masakari Meeting channel https://review.opendev.org/748795 | 11:13 |
| *** Gyuseok_Jung has quit IRC | 11:59 | |
| *** mattd01 has joined #opendev | 12:01 | |
| openstackgerrit | Haïkel Guémar proposed zuul/zuul-jobs master: Add-sshkey: ensure that ssh home directory exists https://review.opendev.org/749278 | 12:43 |
| openstackgerrit | Thierry Carrez proposed opendev/system-config master: Explain "why opendev" on opendev.org index page https://review.opendev.org/748263 | 12:54 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add-sshkey: ensure that ssh home directory exists https://review.opendev.org/749278 | 13:44 |
| openstackgerrit | Gage Hugo proposed openstack/project-config master: Add openstack/openstack-helm-deployments https://review.opendev.org/748304 | 14:15 |
| *** johnavp1989 has joined #opendev | 14:29 | |
| *** sshnaidm has quit IRC | 14:34 | |
| fungi | #status log deleted unused cinder volumes in rax-dfw control plane tenant: wiki-dev.openstack.org/main01 (55db9e89-9cb4-4202-af88-d8c4a174998e), review-dev.openstack.org/main01 (66fea64f-9220-4c53-8988-deb32477ada7), static.openstack.org/main01 (0c0c7fb5-146b-4ecf-baf6-4ac9eaa4f277) | 14:37 |
| openstackstatus | fungi: finished logging | 14:37 |
| *** Dmitrii-Sh7 has joined #opendev | 14:41 | |
| openstackgerrit | Merged openstack/project-config master: Add openstack-ansible/os_senlin role https://review.opendev.org/748683 | 14:43 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Add os_senlin to zuul projects https://review.opendev.org/748693 | 14:43 |
| *** Dmitrii-Sh has quit IRC | 14:48 | |
| *** Dmitrii-Sh7 is now known as Dmitrii-Sh | 14:48 | |
| *** Gyuseok_Jung has joined #opendev | 14:51 | |
| *** sshnaidm has joined #opendev | 14:53 | |
| *** qchris has quit IRC | 14:57 | |
| *** mlavalle has joined #opendev | 14:59 | |
| *** hashar is now known as hasharAway | 15:02 | |
| *** yoctozepto has quit IRC | 15:02 | |
| *** yoctozepto has joined #opendev | 15:03 | |
| *** qchris has joined #opendev | 15:09 | |
| *** dtantsur is now known as dtantsur|afk | 15:18 | |
| *** fressi has quit IRC | 15:24 | |
| *** hasharAway is now known as hashar | 15:25 | |
| *** sshnaidm is now known as sshnaidm|bbl | 15:50 | |
| *** fressi has joined #opendev | 15:53 | |
| *** fressi has quit IRC | 15:59 | |
| clarkb | zuul.o.o's zuul-web image hasn't updated yet, I figured I'd let the normal machinery do the pull then we can restart zuul-web to pick up the build dashboard fix as well as the fix for bolding pipeline names | 16:07 |
| openstackgerrit | Merged openstack/project-config master: Retire the devstack-plugin-zmq project https://review.opendev.org/745344 | 16:17 |
| clarkb | we should double check these new projects and retirements aren't affected by the gerritlib and jeepyb updates from yesterday | 16:18 |
| openstackgerrit | Merged opendev/base-jobs master: Set file modes explicitly https://review.opendev.org/748478 | 16:20 |
| openstackgerrit | Merged openstack/project-config master: Retire devstack-plugin-pika project https://review.opendev.org/745342 | 16:21 |
| clarkb | fungi: we shoul watch the file modes change in bsae jobs carefully. In particular there is some concern that afs content may change modes | 16:23 |
| fungi | yeah, i expected that might be the case | 16:24 |
| clarkb | I didn't realize it got a new ps after my -1 otherwise I probably would've WIP'd it but I think this was alwways going to be the merge strategy (rip the bandaid off and observce) | 16:24 |
| fungi | i'm around to help beat on or undo it | 16:24 |
| openstackgerrit | Merged openstack/project-config master: Add neutron-tempest-plugin-scenario-ovn job to Neutron dashboard https://review.opendev.org/748223 | 16:25 |
| clarkb | infra-root the zuul-web image is now two hours old on zuul.o.o. Any objections to me restarting zuul-web there now? | 16:45 |
| *** sshnaidm|bbl is now known as sshnaidm | 16:46 | |
| clarkb | don't see any objections, restarting it now | 16:52 |
| fungi | go for it | 16:53 |
| clarkb | johnsom: I just updated our zuul-web deployment to include the revert of the build dashboard stuff that you noticed. It should look like the old setup now | 16:53 |
| fungi | sorry, back again, was trying to scarf down some lunch real quick | 16:54 |
| clarkb | no worries | 16:54 |
| johnsom | clarkb Thank you! | 16:54 |
| *** priteau has quit IRC | 16:59 | |
| clarkb | https://review.opendev.org/#/admin/projects/openstack/openstack-ansible-os_senlin,branches lgtm so I think the jeepyb changes haven't regressed at least | 17:07 |
| clarkb | still to be determined if setting a non master HEAD on a new project is functional | 17:07 |
| fungi | we probably ought to test that at some point | 17:08 |
| *** sshnaidm is now known as sshnaidm|afk | 17:11 | |
| *** mattd01 has quit IRC | 17:19 | |
| *** hashar is now known as hasharAway | 17:22 | |
| clarkb | ya, I half expect we'll get volunteers at some point based on various conversations in various groups, but if not then we can always do it ourselves | 17:24 |
| *** lpetrut has quit IRC | 17:26 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Remove dependency on pkg_resources https://review.opendev.org/748737 | 17:35 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Remove dependency on pkg_resources https://review.opendev.org/748737 | 17:38 |
| fungi | huh, my gertty is hanging and logging a bunch of 404 errors... apparently it wants to fetch refs from https://review.opendev.org/p/ | 17:43 |
| fungi | which we recently disabled | 17:44 |
| fungi | i don't seem to have configured it to look there | 17:44 |
| *** Gyuseok_Jung has quit IRC | 17:44 | |
| *** sgw has quit IRC | 17:44 | |
| *** dirk has quit IRC | 17:44 | |
| *** Gyuseok_Jung has joined #opendev | 17:46 | |
| *** sgw has joined #opendev | 17:46 | |
| *** dirk has joined #opendev | 17:46 | |
| *** auristor has quit IRC | 17:47 | |
| *** noonedeadpunk has quit IRC | 17:48 | |
| johnavp1989 | hello all. is it proper to use the six package to achieve python2/3 compatability in opendev projects? | 17:49 |
| johnavp1989 | the python3 wiki suggests that it is but i just wanted to make sure | 17:50 |
| fungi | johnavp1989: we don't really decide what's proper and what isn't for something like that | 17:51 |
| clarkb | johnavp1989: it is a useful tool for that yes. Sometimes a simple try except is sufficent though and we dont need a whole new dep | 17:51 |
| *** noonedeadpunk has joined #opendev | 17:51 | |
| clarkb | will be context dependent | 17:51 |
| fungi | also depends a bit on what you mean by "opendev projects" | 17:51 |
| johnavp1989 | specifically the openstacksdk | 17:51 |
| fungi | johnavp1989: okay, so you're talking about an official openstack project, not just anything hosted in the opendev collaboratory. the openstack tc and individual project leaders are who tend to make those sorts of standardization decisions | 17:53 |
| johnavp1989 | looks like the six library was recently removed as a dependency so i guess that kind of answers my question | 17:53 |
| johnavp1989 | okay makes sense. thanks for the info | 17:54 |
| fungi | johnavp1989: you may have more luck asking in #openstack-sdks | 17:55 |
| *** auristor has joined #opendev | 18:00 | |
| fungi | okay, solved my gertty issue. apparently /p/ is a baked-in default, setting "git-url: https://opendev.org/" in the server section allowed me to stop using that | 18:01 |
| fungi | worth keeping in mind if anyone else reports issues | 18:01 |
| clarkb | good to catch these issues early | 18:02 |
| clarkb | any idea why it didnt fail sooner? | 18:02 |
| corvus | tbh, i'm not entirely sure why we don't have a /p ? | 18:02 |
| corvus | this works: git clone https://gerrit-review.googlesource.com/p/gerrit | 18:02 |
| fungi | clarkb: i think it was triggered by me trying to open a change for a new project, and gertty tried to clone that new project from the /p/ url | 18:03 |
| clarkb | corvus: paladox and the docspoint out /p is used for polygerrit dashboarding in newer gerrit | 18:04 |
| fungi | corvus: yeah, i think paladox said something about wmf's gerrit having some fancy rewrite rules so that git operations will still work, for backward compatibility? | 18:04 |
| fungi | maybe gerrit-review is doing something similar | 18:05 |
| corvus | gerrit-review doesn't appear to redirect | 18:05 |
| paladox | yeh, we do rewrites for /p | 18:05 |
| fungi | basically if it looks like a git operation then rewrite to something else on the backend, and otherwise hand off /p requests to polygerrit | 18:05 |
| corvus | git clone https://gerrit.wikimedia.org/r/p/operations/puppet does redirect | 18:05 |
| paladox | https://github.com/wikimedia/puppet/blob/production/modules/gerrit/templates/apache.erb#L117 | 18:06 |
| fungi | maybe gerrit-review is doing some transparent rewrites to accomplish the same | 18:06 |
| clarkb | does upstream's gerrit expose the dashboard feature? | 18:07 |
| clarkb | if not that may also explain the difference | 18:08 |
| corvus | it does look like gerrithub doesn't have a /p | 18:08 |
| corvus | so perhaps googlesource did implement a workaround | 18:09 |
| paladox | i don't think they implemented a workaround. | 18:10 |
| corvus | paladox: cloning from /p works for googlesource | 18:11 |
| paladox | oh | 18:11 |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Remove /p from default git-url https://review.opendev.org/749375 | 18:13 |
| fungi | i'm also not opposed to adding some fancy dance to rewrite or redirect it, but we had hoped we could get away without doing that | 18:13 |
| *** mattd01 has joined #opendev | 18:14 | |
| corvus | i'm just thoroughly confused by what is expected at this point | 18:14 |
| corvus | i have whiplash from 10 years of adding and removing /p from the url | 18:15 |
| fungi | me too, really | 18:15 |
| clarkb | the big issue with doing rewrites is we'd need to manage git branches in another location. doable but if the api is going away it is simpler to stop | 18:15 |
| corvus | (because this is absolutely not even the second time this has happened) | 18:15 |
| corvus | i do wish googlesource reflected the state of the product though; that it doesn't is really the most confusing thing | 18:16 |
| clarkb | thats "normal" though. They dont run with a bunch of things you'd expect in yhe wild aiui | 18:17 |
| clarkb | like no dwl db back when that was a thing | 18:17 |
| clarkb | *sql db | 18:17 |
| corvus | as a user, i can't tell that | 18:17 |
| corvus | but verifying that the api works against the canonical upstream implementation seems like, all things being equal, a pretty good sanity check | 18:18 |
| corvus | but maybe i should just use gerrithub for that in the future | 18:18 |
| clarkb | aiui every single guice hook is there not because open source gerrit has multiple implementations but because google has an alternative implementation. Its definitely not ideal | 18:20 |
| openstackgerrit | Merged openstack/project-config master: Add openstack/openstack-helm-deployments https://review.opendev.org/748304 | 18:36 |
| openstackgerrit | Merged openstack/project-config master: Update Ansible roles for ATOS and Thales HSMs https://review.opendev.org/748028 | 18:50 |
| *** diablo_rojo has joined #opendev | 18:56 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Remove dependency on pkg_resources https://review.opendev.org/748737 | 19:03 |
| *** openstackgerrit has quit IRC | 19:21 | |
| * clarkb finds a sandwich | 19:53 | |
| fungi | corvus: for completeness, i looked at the build failure for the opendev page update, and it's importing setuptools in a virtualenv built as part of the job, so fetches latest setuptools | 19:54 |
| fungi | wouldn't have been solved by the image pin either way | 19:54 |
| fungi | (in addition to having run long after we built images with setuptools<50) | 19:55 |
| johnavp1989 | can anyone direct me on how i link a gerrit change to storyboard? | 19:55 |
| fungi | johnavp1989: best way is if there is already a task for the project which that change is implementing, you add a "Task: #" footer in the commit message but if you just want to refer generally to a story and not a particular task, it's "Story: #" in a commit message footer | 19:56 |
| fungi | if you refer to a specific task id number, then automation should also update the task state and insert a comment on the story linking back to the review for you | 19:58 |
| *** hasharAway has quit IRC | 19:59 | |
| fungi | johnavp1989: https://docs.openstack.org/contributors/common/git.html#tags (we probably should have called that section "footers" instead, to disambiguate them from actual git tag objects) | 20:00 |
| fungi | i'll push a change to the contributor guide to make that better | 20:01 |
| johnavp1989 | okay thanks that's simple enough. i guess i now need to figure out what a task is in storyboar dha | 20:02 |
| fungi | johnavp1989: if you look at an open story, like https://storyboard.openstack.org/#!/story/2008093 , you'll see there's a "tasks" section. in the case of that particular story it has only one task, with an id of 40794 | 20:04 |
| fungi | that task is in "todo" state right now, but if you pushed a change to gerrit with "task: 40794" on a line by itself at the end of the commit message (like immediately after the change-id line), then that task's state will automatically change to "review" and a comment will get added with a hyperlink to the change you pushed | 20:06 |
| fungi | johnavp1989: does that make sense? | 20:06 |
| johnavp1989 | fungi: yes it does. thank you | 20:08 |
| fungi | my pleasure. don't hesitate to ask any other usage questions you might have regarding these tools | 20:09 |
| johnavp1989 | Gerrit has posted to the storyboard now which is what i was hoping for. Will do, I appreciate the help. | 20:10 |
| fungi | you're welcome! | 20:15 |
| *** andrewbonney has quit IRC | 20:18 | |
| *** mattd01 has quit IRC | 20:35 | |
| *** auristor has quit IRC | 20:53 | |
| clarkb | I think we're still failing with setuptools 50.0.1 as expected | 20:58 |
| clarkb | er as ianw expected anyway | 20:58 |
| *** openstackgerrit has joined #opendev | 21:00 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Install older setuptools in puppet apply jobs https://review.opendev.org/749394 | 21:00 |
| clarkb | infra-root ^ I think that will fix the ask and opendev.org changes' test failures | 21:01 |
| clarkb | not sure if we prefer to use the system distutils via the env var or just pin setuptools (I've pinned setuptools there) | 21:01 |
| ianw | i guess a pin, because it seems unlikely anyone is updating those jobs to thing that would require newer setuptools in the long run anyway | 21:03 |
| clarkb | wfm | 21:05 |
| clarkb | diablo_rojo: I have submitted the form, can you check that you got it when you have a chance | 21:05 |
| clarkb | that change is running the two jobs that are failing elsewhere too so we should actually know if it helps | 21:06 |
| clarkb | fungi: corvus https://review.opendev.org/#/c/749394/1 does pass testing if you have a moment for it. I'll recheck the opendev.org fornt page change and reapprove the ask.o.o read only change once that lands | 21:20 |
| diablo_rojo | clarkb, got it! | 21:21 |
| openstackgerrit | Merged opendev/system-config master: Install older setuptools in puppet apply jobs https://review.opendev.org/749394 | 21:35 |
| * clarkb rechecks things | 21:36 | |
| *** DSpider has quit IRC | 21:56 | |
| openstackgerrit | Merged opendev/system-config master: Make ask.openstack.org read-only https://review.opendev.org/746497 | 22:11 |
| *** tosky has quit IRC | 22:43 | |
| clarkb | ianw: I think we've got sad ansibles on bridge again | 22:53 |
| clarkb | ianw: what was your debugging process of that before? | 22:53 |
| ianw | umm, first probably is the stuck ssh processes should indicate what's not responding | 22:54 |
| ianw | 104.130.159.19 | 22:54 |
| ianw | elasticsearch02.openstack.org. | 22:54 |
| fungi | elasticsearch02 | 22:54 |
| clarkb | elasticsearch02.openstack.org. yup | 22:54 |
| fungi | fweeeeep | 22:55 |
| ianw | then for p in `ps -aef | grep 'ansible-playbook' | awk '{print $2}' `; do sudo kill -INT $p; done | 22:55 |
| ianw | then fix whatever's wrong :) | 22:55 |
| clarkb | we should fix elasticsearch02 probably first? | 22:55 |
| fungi | and indeed, it's unresponsive for ssh | 22:55 |
| clarkb | in this case I expect we can just reboot it | 22:55 |
| fungi | responds to ping, but yeah it's likely croaked | 22:56 |
| clarkb | I'll ask for a hard reboot of elasticsearch02 ? | 22:56 |
| fungi | go for it | 22:57 |
| clarkb | I wish I knew how to make ansible timeout | 22:58 |
| clarkb | I swear we configure it to timeout | 22:58 |
| clarkb | but its like that is a post tcp connection establishment timeout | 22:58 |
| clarkb | sort of related: my laptop says man isn't installed anymore | 23:00 |
| clarkb | wat | 23:01 |
| clarkb | also I was hoping killing the ssh control processes would cause the asible playbooks to clean up but it seems like it doesn't | 23:02 |
| *** auristor has joined #opendev | 23:03 | |
| clarkb | alright the only running ansible-playbooks processes are current (I think) | 23:08 |
| clarkb | this iteration through should update ask assuming we don't hit other hosts that are unhappy now | 23:08 |
| openstackgerrit | Kendall Nelson proposed openstack/project-config master: Updates Message on PR Close https://review.opendev.org/742194 | 23:44 |
| clarkb | remote-puppet-else is next | 23:46 |
| * clarkb practices patience | 23:46 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!