Tuesday, 2020-09-01

ianwyeah, i think 35 jobs only run on xenial, and i think 3.5 is the worst affected by the new setuptools00:03
openstackgerritGage Hugo proposed openstack/project-config master: Add openstack/openstack-helm-deployments  https://review.opendev.org/74830400:52
*** diablo_rojo has quit IRC00:54
*** DSpider has quit IRC01:07
*** wolsen has quit IRC03:04
*** bhagyashris is now known as bhagyashris|away05:23
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Update hashicorp jobs file with correct title  https://review.opendev.org/74785305:43
*** fressi has joined #opendev05:44
fricklerinfra-root: py35 still seems to show the same error https://zuul.opendev.org/t/zuul/build/8c5a67e79c694844a209e75865c4549f06:16
ianwfrickler: was that the one i just rechecked?06:16
fricklerianw: that was the one AJaeger amended just half an hour ago06:17
*** lpetrut has joined #opendev06:33
*** andrewbonney has joined #opendev07:13
*** tosky has joined #opendev07:21
ianwfrickler: hrm, https://zuul.opendev.org/t/zuul/build/579dfd55aaea4a64abe6e61cb96cc75d  passed though.  maybe just an old node?07:27
fricklerianw: that's possible, I've wondered whether there's an easy way to find out which image a node was based on, maybe we could add that to the job output?07:35
* AJaeger rechecks 74785307:37
ianwfrickler: umm, i think it is, it dumps the dib-build.txt header or something somehwere07:37
fricklerianw: ah, right, top of https://8abbfecd00d9996fdb0c-5c4643ce01a9b304087712e3e08013b4.ssl.cf1.rackcdn.com/747853/2/check/tox-py35/8c5a67e/zuul-info/zuul-info.ubuntu-xenial.txt , that looks like old image indeed07:39
ianwheh, yeah, that's it07:40
AJaegeryes, passed now - anybody wants to +2A 747853?07:50
*** DSpider has joined #opendev07:54
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** priteau has joined #opendev08:03
*** fressi has quit IRC08:08
*** hashar has joined #opendev08:16
*** fressi has joined #opendev08:18
openstackgerritMerged zuul/zuul-jobs master: Update hashicorp jobs file with correct title  https://review.opendev.org/74785308:21
*** fressi has quit IRC08:31
*** sshnaidm|bbl is now known as sshnaidm08:36
*** fressi has joined #opendev08:51
*** Gyuseok_Jung has quit IRC09:04
*** dtantsur|afk is now known as dtantsur09:05
*** calcmandan has quit IRC09:12
*** calcmandan has joined #opendev09:13
*** Gyuseok_Jung has joined #opendev09:38
*** tkajinam has quit IRC09:54
*** tkajinam has joined #opendev09:55
*** ajya has joined #opendev10:24
*** mgagne has quit IRC10:39
openstackgerritMerged opendev/irc-meetings master: Change Masakari Meeting time  https://review.opendev.org/74781911:13
openstackgerritMerged opendev/irc-meetings master: Change Masakari Meeting channel  https://review.opendev.org/74879511:13
*** Gyuseok_Jung has quit IRC11:59
*** mattd01 has joined #opendev12:01
openstackgerritHaïkel Guémar proposed zuul/zuul-jobs master: Add-sshkey: ensure that ssh home directory exists  https://review.opendev.org/74927812:43
openstackgerritThierry Carrez proposed opendev/system-config master: Explain "why opendev" on opendev.org index page  https://review.opendev.org/74826312:54
openstackgerritMerged zuul/zuul-jobs master: Add-sshkey: ensure that ssh home directory exists  https://review.opendev.org/74927813:44
openstackgerritGage Hugo proposed openstack/project-config master: Add openstack/openstack-helm-deployments  https://review.opendev.org/74830414:15
*** johnavp1989 has joined #opendev14:29
*** sshnaidm has quit IRC14:34
fungi#status log deleted unused cinder volumes in rax-dfw control plane tenant: wiki-dev.openstack.org/main01 (55db9e89-9cb4-4202-af88-d8c4a174998e), review-dev.openstack.org/main01 (66fea64f-9220-4c53-8988-deb32477ada7), static.openstack.org/main01 (0c0c7fb5-146b-4ecf-baf6-4ac9eaa4f277)14:37
openstackstatusfungi: finished logging14:37
*** Dmitrii-Sh7 has joined #opendev14:41
openstackgerritMerged openstack/project-config master: Add openstack-ansible/os_senlin role  https://review.opendev.org/74868314:43
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Add os_senlin to zuul projects  https://review.opendev.org/74869314:43
*** Dmitrii-Sh has quit IRC14:48
*** Dmitrii-Sh7 is now known as Dmitrii-Sh14:48
*** Gyuseok_Jung has joined #opendev14:51
*** sshnaidm has joined #opendev14:53
*** qchris has quit IRC14:57
*** mlavalle has joined #opendev14:59
*** hashar is now known as hasharAway15:02
*** yoctozepto has quit IRC15:02
*** yoctozepto has joined #opendev15:03
*** qchris has joined #opendev15:09
*** dtantsur is now known as dtantsur|afk15:18
*** fressi has quit IRC15:24
*** hasharAway is now known as hashar15:25
*** sshnaidm is now known as sshnaidm|bbl15:50
*** fressi has joined #opendev15:53
*** fressi has quit IRC15:59
clarkbzuul.o.o's zuul-web image hasn't updated yet, I figured I'd let the normal machinery do the pull then we can restart zuul-web to pick up the build dashboard fix as well as the fix for bolding pipeline names16:07
openstackgerritMerged openstack/project-config master: Retire the devstack-plugin-zmq project  https://review.opendev.org/74534416:17
clarkbwe should double check these new projects and retirements aren't affected by the gerritlib and jeepyb updates from yesterday16:18
openstackgerritMerged opendev/base-jobs master: Set file modes explicitly  https://review.opendev.org/74847816:20
openstackgerritMerged openstack/project-config master: Retire devstack-plugin-pika project  https://review.opendev.org/74534216:21
clarkbfungi: we shoul watch the file modes change in bsae jobs carefully. In particular there is some concern that afs content may change modes16:23
fungiyeah, i expected that might be the case16:24
clarkbI didn't realize it got a new ps after my -1 otherwise I probably would've WIP'd it but I think this was alwways going to be the merge strategy (rip the bandaid off and observce)16:24
fungii'm around to help beat on or undo it16:24
openstackgerritMerged openstack/project-config master: Add neutron-tempest-plugin-scenario-ovn job to Neutron dashboard  https://review.opendev.org/74822316:25
clarkbinfra-root the zuul-web image is now two hours old on zuul.o.o. Any objections to me restarting zuul-web there now?16:45
*** sshnaidm|bbl is now known as sshnaidm16:46
clarkbdon't see any objections, restarting it now16:52
fungigo for it16:53
clarkbjohnsom: I just updated our zuul-web deployment to include the revert of the build dashboard stuff that you noticed. It should look like the old setup now16:53
fungisorry, back again, was trying to scarf down some lunch real quick16:54
clarkbno worries16:54
johnsomclarkb Thank you!16:54
*** priteau has quit IRC16:59
clarkbhttps://review.opendev.org/#/admin/projects/openstack/openstack-ansible-os_senlin,branches lgtm so I think the jeepyb changes haven't regressed at least17:07
clarkbstill to be determined if setting a non master HEAD on a new project is functional17:07
fungiwe probably ought to test that at some point17:08
*** sshnaidm is now known as sshnaidm|afk17:11
*** mattd01 has quit IRC17:19
*** hashar is now known as hasharAway17:22
clarkbya, I half expect we'll get volunteers at some point based on various conversations in various groups, but if not then we can always do it ourselves17:24
*** lpetrut has quit IRC17:26
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Remove dependency on pkg_resources  https://review.opendev.org/74873717:35
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Remove dependency on pkg_resources  https://review.opendev.org/74873717:38
fungihuh, my gertty is hanging and logging a bunch of 404 errors... apparently it wants to fetch refs from https://review.opendev.org/p/17:43
fungiwhich we recently disabled17:44
fungii don't seem to have configured it to look there17:44
*** Gyuseok_Jung has quit IRC17:44
*** sgw has quit IRC17:44
*** dirk has quit IRC17:44
*** Gyuseok_Jung has joined #opendev17:46
*** sgw has joined #opendev17:46
*** dirk has joined #opendev17:46
*** auristor has quit IRC17:47
*** noonedeadpunk has quit IRC17:48
johnavp1989hello all. is it proper to use the six package to achieve python2/3 compatability in opendev projects?17:49
johnavp1989the python3 wiki suggests that it is but i just wanted to make sure17:50
fungijohnavp1989: we don't really decide what's proper and what isn't for something like that17:51
clarkbjohnavp1989: it is a useful tool for that yes. Sometimes a simple try except is sufficent though and we dont need a whole new dep17:51
*** noonedeadpunk has joined #opendev17:51
clarkbwill be context dependent17:51
fungialso depends a bit on what you mean by "opendev projects"17:51
johnavp1989specifically the openstacksdk17:51
fungijohnavp1989: okay, so you're talking about an official openstack project, not just anything hosted in the opendev collaboratory. the openstack tc and individual project leaders are who tend to make those sorts of standardization decisions17:53
johnavp1989looks like the six library was recently removed as a dependency so i guess that kind of answers my question17:53
johnavp1989okay makes sense. thanks for the info17:54
fungijohnavp1989: you may have more luck asking in #openstack-sdks17:55
*** auristor has joined #opendev18:00
fungiokay, solved my gertty issue. apparently /p/ is a baked-in default, setting "git-url: https://opendev.org/" in the server section allowed me to stop using that18:01
fungiworth keeping in mind if anyone else reports issues18:01
clarkbgood to catch these issues early18:02
clarkbany idea why it didnt fail sooner?18:02
corvustbh, i'm not entirely sure why we don't have a /p ?18:02
corvusthis works: git clone https://gerrit-review.googlesource.com/p/gerrit18:02
fungiclarkb: i think it was triggered by me trying to open a change for a new project, and gertty tried to clone that new project from the /p/ url18:03
clarkbcorvus: paladox and the docspoint out /p is used for polygerrit dashboarding in newer gerrit18:04
fungicorvus: yeah, i think paladox said something about wmf's gerrit having some fancy rewrite rules so that git operations will still work, for backward compatibility?18:04
fungimaybe gerrit-review is doing something similar18:05
corvusgerrit-review doesn't appear to redirect18:05
paladoxyeh, we do rewrites for /p18:05
fungibasically if it looks like a git operation then rewrite to something else on the backend, and otherwise hand off /p requests to polygerrit18:05
corvusgit clone https://gerrit.wikimedia.org/r/p/operations/puppet  does redirect18:05
fungimaybe gerrit-review is doing some transparent rewrites to accomplish the same18:06
clarkbdoes upstream's gerrit expose the dashboard feature?18:07
clarkbif not that may also explain the difference18:08
corvusit does look like gerrithub doesn't have a /p18:08
corvusso perhaps googlesource did implement a workaround18:09
paladoxi don't think they implemented a workaround.18:10
corvuspaladox: cloning from /p works for googlesource18:11
openstackgerritJames E. Blair proposed ttygroup/gertty master: Remove /p from default git-url  https://review.opendev.org/74937518:13
fungii'm also not opposed to adding some fancy dance to rewrite or redirect it, but we had hoped we could get away without doing that18:13
*** mattd01 has joined #opendev18:14
corvusi'm just thoroughly confused by what is expected at this point18:14
corvusi have whiplash from 10 years of adding and removing /p from the url18:15
fungime too, really18:15
clarkbthe big issue with doing rewrites is we'd need to manage git branches in another location. doable but if the api is going away it is simpler to stop18:15
corvus(because this is absolutely not even the second time this has happened)18:15
corvusi do wish googlesource reflected the state of the product though; that it doesn't is really the most confusing thing18:16
clarkbthats "normal" though. They dont run with a bunch of things you'd expect in yhe wild aiui18:17
clarkblike no dwl db back when that was a thing18:17
clarkb*sql db18:17
corvusas a user, i can't tell that18:17
corvusbut verifying that the api works against the canonical upstream implementation seems like, all things being equal, a pretty good sanity check18:18
corvusbut maybe i should just use gerrithub for that in the future18:18
clarkbaiui every single guice hook is there not because open source gerrit has multiple implementations but because google has an alternative implementation. Its definitely not ideal18:20
openstackgerritMerged openstack/project-config master: Add openstack/openstack-helm-deployments  https://review.opendev.org/74830418:36
openstackgerritMerged openstack/project-config master: Update Ansible roles for ATOS and Thales HSMs  https://review.opendev.org/74802818:50
*** diablo_rojo has joined #opendev18:56
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Remove dependency on pkg_resources  https://review.opendev.org/74873719:03
*** openstackgerrit has quit IRC19:21
* clarkb finds a sandwich19:53
fungicorvus: for completeness, i looked at the build failure for the opendev page update, and it's importing setuptools in a virtualenv built as part of the job, so fetches latest setuptools19:54
fungiwouldn't have been solved by the image pin either way19:54
fungi(in addition to having run long after we built images with setuptools<50)19:55
johnavp1989can anyone direct me on how i link a gerrit change to storyboard?19:55
fungijohnavp1989: best way is if there is already a task for the project which that change is implementing, you add a "Task: #" footer in the commit message but if you just want to refer generally to a story and not a particular task, it's "Story: #" in a commit message footer19:56
fungiif you refer to a specific task id number, then automation should also update the task state and insert a comment on the story linking back to the review for you19:58
*** hasharAway has quit IRC19:59
fungijohnavp1989: https://docs.openstack.org/contributors/common/git.html#tags (we probably should have called that section "footers" instead, to disambiguate them from actual git tag objects)20:00
fungii'll push a change to the contributor guide to make that better20:01
johnavp1989okay thanks that's simple enough. i guess i now need to figure out what a task is in storyboar dha20:02
fungijohnavp1989: if you look at an open story, like https://storyboard.openstack.org/#!/story/2008093 , you'll see there's a "tasks" section. in the case of that particular story it has only one task, with an id of 4079420:04
fungithat task is in "todo" state right now, but if you pushed a change to gerrit with "task: 40794" on a line by itself at the end of the commit message (like immediately after the change-id line), then that task's state will automatically change to "review" and a comment will get added with a hyperlink to the change you pushed20:06
fungijohnavp1989: does that make sense?20:06
johnavp1989fungi: yes it does. thank you20:08
fungimy pleasure. don't hesitate to ask any other usage questions you might have regarding these tools20:09
johnavp1989Gerrit has posted to the storyboard now which is what i was hoping for. Will do, I appreciate the help.20:10
fungiyou're welcome!20:15
*** andrewbonney has quit IRC20:18
*** mattd01 has quit IRC20:35
*** auristor has quit IRC20:53
clarkbI think we're still failing with setuptools 50.0.1 as expected20:58
clarkber as ianw expected anyway20:58
*** openstackgerrit has joined #opendev21:00
openstackgerritClark Boylan proposed opendev/system-config master: Install older setuptools in puppet apply jobs  https://review.opendev.org/74939421:00
clarkbinfra-root ^ I think that will fix the ask and opendev.org changes' test failures21:01
clarkbnot sure if we prefer to use the system distutils via the env var or just pin setuptools (I've pinned setuptools there)21:01
ianwi guess a pin, because it seems unlikely anyone is updating those jobs to thing that would require newer setuptools in the long run anyway21:03
clarkbdiablo_rojo: I have submitted the form, can you check that you got it when you have a chance21:05
clarkbthat change is running the two jobs that are failing elsewhere too so we should actually know if it helps21:06
clarkbfungi: corvus https://review.opendev.org/#/c/749394/1 does pass testing if you have a moment for it. I'll recheck the opendev.org fornt page change and reapprove the ask.o.o read only change once that lands21:20
diablo_rojoclarkb, got it!21:21
openstackgerritMerged opendev/system-config master: Install older setuptools in puppet apply jobs  https://review.opendev.org/74939421:35
* clarkb rechecks things21:36
*** DSpider has quit IRC21:56
openstackgerritMerged opendev/system-config master: Make ask.openstack.org read-only  https://review.opendev.org/74649722:11
*** tosky has quit IRC22:43
clarkbianw: I think we've got sad ansibles on bridge again22:53
clarkbianw: what was your debugging process of that before?22:53
ianwumm, first probably is the stuck ssh processes should indicate what's not responding22:54
clarkbelasticsearch02.openstack.org. yup22:54
ianwthen for p in `ps -aef | grep 'ansible-playbook' | awk '{print $2}' `; do sudo kill -INT $p; done22:55
ianwthen fix whatever's wrong :)22:55
clarkbwe should fix elasticsearch02 probably first?22:55
fungiand indeed, it's unresponsive for  ssh22:55
clarkbin this case I expect we can just reboot it22:55
fungiresponds to ping, but yeah it's likely croaked22:56
clarkbI'll ask for a hard reboot of elasticsearch02 ?22:56
fungigo for it22:57
clarkbI wish I knew how to make ansible timeout22:58
clarkbI swear we configure it to timeout22:58
clarkbbut its like that is a post tcp connection establishment timeout22:58
clarkbsort of related: my laptop says man isn't installed anymore23:00
clarkbalso I was hoping killing the ssh control processes would cause the asible playbooks to clean up but it seems like it doesn't23:02
*** auristor has joined #opendev23:03
clarkbalright the only running ansible-playbooks processes are current (I think)23:08
clarkbthis iteration through should update ask assuming we don't hit other hosts that are unhappy now23:08
openstackgerritKendall Nelson proposed openstack/project-config master: Updates Message on PR Close  https://review.opendev.org/74219423:44
clarkbremote-puppet-else is next23:46
* clarkb practices patience23:46

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!