Thursday, 2020-07-23

openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-pip: Clarify situation with ensure_pip_from_packages_with_python2  https://review.opendev.org/74257101:01
ianwso the merge API is "POST /repos/:owner/:repo/merges"01:21
ianwthis comes under "contents" https://developer.github.com/v3/apps/permissions/#permission-on-contents01:21
ianw#status log removed the r/w requirements for "contents" on the github zuul app ( https://github.com/apps/opendev-zuul)01:24
openstackstatusianw: finished logging01:24
ianwRead access to code01:24
ianwRead access to administration and metadata01:24
ianwRead and write access to commit statuses, issues, and pull requests01:24
ianwthat looks right01:25
*** sgw has quit IRC02:32
*** sgw has joined #opendev02:48
*** sgw has quit IRC02:52
*** sgw has joined #opendev02:57
*** sgw has quit IRC03:40
*** sgw has joined #opendev03:40
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Pre-install python3 for CentOS  https://review.opendev.org/74186803:51
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] Drop dib-python requirement from several elements  https://review.opendev.org/74187703:51
*** xiaolin has quit IRC03:58
*** sgw has quit IRC04:15
*** sgw has joined #opendev04:15
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Pre-install python3 for CentOS  https://review.opendev.org/74186804:37
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] Drop dib-python requirement from several elements  https://review.opendev.org/74187704:37
*** marios has joined #opendev05:21
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Pre-install python3 for CentOS  https://review.opendev.org/74186806:25
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] Drop dib-python requirement from several elements  https://review.opendev.org/74187706:25
*** qchris has quit IRC06:51
*** qchris has joined #opendev07:05
*** DSpider has joined #opendev07:11
*** tosky has joined #opendev07:22
*** dougsz has joined #opendev07:35
*** fressi has joined #opendev07:49
*** elod_off is now known as elod07:54
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:03
*** jaicaa has quit IRC08:27
*** jaicaa has joined #opendev08:29
*** tristan888 has joined #opendev10:06
*** qchris has quit IRC10:15
*** dtantsur|afk is now known as dtantsur10:29
*** zbr is now known as zbr|ruck10:30
*** sshnaidm|afk is now known as sshnaidm|off11:11
*** bhagyashris is now known as bhagyashris|afk11:54
*** tristan888 has quit IRC12:07
*** bhagyashris|afk is now known as bhagyashris12:22
*** qchris has joined #opendev12:57
*** qchris has quit IRC13:10
*** qchris has joined #opendev13:14
fungiinfra-root: just a heads up, i'm investigating several openstack release job failures from ~4 hours ago which suggest afs write errors... i think they may point to problems with specific executors, though the sample size is too small to say: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016064.html13:15
fungithe two tarball write failures both ran from ze11, the docs failure from ze1013:16
fungize02, ze10 and ze11 have all unexpectedly rebooted within the past 24 hours, it looks like13:16
fungize11 also logged a couple of "afs_pag_wait() PAG throttling triggered" lines in dmesg toward the end of the bootspam13:17
fungithough that might just be a benign warning. there were no errors in dmesg around the times of the write permission problems recorded in the failed builds13:18
fungioh, though it also logged "clock went backwards, not PAG throttling" and dmesg times are >10 minutes earlier from the times syslog records the same entries13:26
fungiso i wonder if something got confused by a giant time jump during boot13:27
fungize10 has an eve larger discrepancy14:14
fungis/eve/even/14:14
fungize02 is not as extreme, but still several minutes difference14:15
clarkbthey are currently out of sync?14:17
clarkbthey are xenial so not using systemd's ntp service iirc. They should be ntpd? if that is the case wemay need to force them to make a jump to current time?14:18
funginope, ntpd is doing its job just fine14:19
fungiaccurate at least to the second14:19
clarkboh I see it moved time by big amount in boot14:19
fungijust wondering if the large time jump at start could have caused issues for afs or something14:19
clarkbif afs's cache entries use system sourced timestamps I could see that14:20
fungii still need to manually test afs writes from them. trying to juggle this and yardwork before the heat advisory takes effect again14:20
clarkbfs checkvolumes may help if so?14:20
clarkbthat made mirrors happy about the wheel cache dirs recwntly14:21
fungiyeah, i mean, all three build failures happened within an hour of one another now 5+ hours ago, so i don't even know for sure if the issue persists14:21
fungii also haven't dug in the ze10 executor debug log yet to see what the actual error from the docs publication post_failure was14:23
dtantsurfolks, does anyone know if gertty is still developed? it doesn't seem to be compatible with python 314:24
fungidtantsur: can you expand on "doesn't seem to be compatible with python 3"?14:26
dtantsurfungi: tries to call decode() on strings. crashes when opening any project. I'm writing a story as we speak.14:26
fungistrange, i'm using it with python3 as we speak14:27
fungiwhat minor version of the interpreter?14:27
dtantsur3.814:28
dtantsurfungi: https://storyboard.openstack.org/#!/story/200795114:28
fungii normally run it under 3.5 but i'm testing with 3.8 on a newer machine now14:29
dtantsurI *think* I used to use it with Python 3 as well, and probably that was 3.6 on the previous fedora14:29
*** mlavalle has joined #opendev14:44
clarkbdid ply update to change the types there?14:45
clarkblast release was 2018 unlikely14:46
fungidtantsur: i'm currently exercising gertty 1.6.0 on python 3.8.4 and it's not crashing for me14:53
fungisame versions of ply and urwid you're using too14:55
fungidtantsur: i'm running from within a venv rather than with pip install --user, could that make a difference?14:56
fungii can open a project, open a review, check a diff, start a review comment, search with message:string and find changes, et cetera14:58
dtantsurokay, lemme try with a venv14:58
fungii manually compile my python interpreters instead of using distro packages for them, but i doubt that makes a difference here14:58
dtantsurnope, crashes the same for me Oo14:58
clarkbsame versions of gertty?14:59
dtantsurpip freeze: http://paste.openstack.org/show/796250/14:59
dtantsurlemme try with 3.6 now14:59
fungithis is my pip freeze: http://paste.openstack.org/show/79625115:00
dtantsuronly urllib3 is different15:00
dtantsurcrashes the same in a 3.6 venv O__o15:00
fungiyour urllib is barely newer than mine, but other than that identical pip freeze too15:01
dtantsurI wonder if my gertty.yaml has something that triggers the crash15:01
dtantsurchange-list-query: >15:02
dtantsur    status:open15:02
dtantsur    NOT message:DNM15:02
dtantsur    NOT message:"DO NOT MERGE"15:02
dtantsurfungi: try the last part of the query ^^^ it triggers the issue for me15:04
fungiError: 'str' object has no attribute 'decode'15:05
dtantsurso it's the same as https://storyboard.openstack.org/#!/story/200648415:05
fungibingo!15:05
dtantsurapparently the quotation marks trigger it15:06
dtantsur`NOT message:DNM` works normally15:06
fungiyep, i concur15:07
fungii also get the same behavior on python 3.5 where i normally run gertty15:08
openstackgerritJames E. Blair proposed opendev/base-jobs master: Add ensure-pip to python release pre-run playbook  https://review.opendev.org/74269415:44
corvusclarkb, fungi: ^ blocking zuul release (see #zuul)15:45
clarkbshould it gi before bindep? I'm checking in that now15:47
clarkbI think it should since the bindep role also uses pip, though in this case it seems to find our preinstalled venv bindep? so is actually fine in either order for us15:49
clarkbI've approvedut based on that15:50
fungiit's easy to reorder that if we decide we should15:53
*** marios is now known as marios|out15:55
*** zbr|ruck is now known as zbr16:02
fungiclarkb: corvus: ansible-lint on that change says "roles/ensure-pip/tasks/main.yaml:1: [E304] Environment variables don't work as part of command"16:05
fungium, what?16:05
fungialso that only showed up in the gate pipeline16:05
corvusi don't think we ran check to completion on that16:06
fungiahh16:06
fungiright, we enqueue direct to gate and preempt check16:06
clarkbit means that the command module doesnt apply env vars to the command execution env. need shell for that. I'm guessing this is anew check because ensure-pip haslanded elsewhere just fine?16:07
fungiso that much is explained16:07
corvus304 is not ignored in zuul-jobs, so yeah, i'm thinking this may be new16:07
fungihttps://docs.ansible.com/ansible-lint/rules/default_rules.html16:08
fungiversion added: historic16:08
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: DNM: test lint on ensure-pip  https://review.opendev.org/74269916:08
corvusfungi: that seems to unsubstatiate that theory16:08
fungii live to unsubstantiate theories16:09
fungibut yeah, i'm baffled16:09
corvusi mean, that's not a command task, that's shell, and env vars sure do work on shell16:10
corvusi reprod that locally16:11
fungiseems like a really thing to false positive match on16:12
fungier, a really strange thing16:12
clarkbalso there are no env vars?16:12
zbrthat is a very weird error to see, trying to reproduce locally16:12
clarkbor is it finding them in the script itself?16:12
zbrafaik 304 will only appear if it finds "command" and never on shell stuff.16:13
corvusclarkb: yeah i think it's in the script16:13
corvuslike it's supposed to look for "command: ENV=foo bar" and tell you that doesn't work16:13
zbris using latest linter?16:13
corvusare we doing anything like copying the zuul ansible modules in to the path?16:14
corvusbecause we just fixed zuul's overridden command module16:14
corvusso maybe it's tripping on that16:14
zbryou are using ansible-lint<4 --- basically enjoy it!16:15
clarkbANSIBLE_ACTION_PLUGINS = {envsitepackagesdir}/zuul/ansible/base/actiongeneral16:15
clarkbcorvus: in tox.ini ^ I think that could be related16:15
zbrbugs fixed years ago16:15
corvuszbr: have you identified a bug?16:15
*** fressi has quit IRC16:15
zbrdo a cat test-requirements.txt16:15
zbrline 1616:15
clarkbwe moved command out of action general into action and action trusted16:15
clarkbso maybe if we change that to action or action trusted it will be happy?16:15
corvuszbr: i believe you that we're running ansible-lint < 4.16:16
corvuszbr: are you saying you know this is a bug that is fixed in 4?16:16
zbrcorvus: this is what i am trying to tell you16:16
clarkboh except it is installing zuul from pacakge nto source I think16:17
clarkband we don't have a packaged version with the fix yet?16:17
zbrplease look at https://pypi.org/project/ansible-lint/#history16:17
clarkbzbr: just because there are releases doesn't mean this is a bug that has been fixed. We should investigate that16:17
*** fressi has joined #opendev16:18
zbrit did16:18
clarkbcan you point to that?16:18
clarkbI think that is what corvus is asking about16:18
zbri run it locally, on same change, and no 304 was seen16:18
corvuszbr: do you know what version we're using in zuul-jobs?16:18
zbryes, latest pre-release16:19
corvuslinters-requirements.txt:ansible-lint>=4.3.0a1,<516:19
corvusah we have a separate file for it16:19
zbrwhich means latest (a5)16:19
zbrcopy same condition, it will work fine16:19
corvusi'll look into updating it then16:20
zbryou may want to add skips for 70116:20
corvusand 20416:20
corvusand 10616:20
zbrwell, can copy the config itself too if you want16:20
zbr206 should be fixed instead16:21
corvusi'm not gonna waste time on it :)16:21
zbri will add meta skip soon named "galaxy", which will skip those related to galaxy16:21
zbrto make your life easier ;)16:22
corvusoh sorry, yeah i'll fix 206, i misread :)16:22
zbrin fact 4.3.0a5 is likely to become GA next week, we are only polishing it these days, preparing config changes, improvind docs.16:24
zbrAlso the current docs are located at https://ansible-lint.readthedocs.io/en/latest/ -- the ones on ansible website were not outdated in an year or so16:25
openstackgerritJames E. Blair proposed opendev/base-jobs master: Add ensure-pip to python release pre-run playbook  https://review.opendev.org/74269416:26
*** fressi has quit IRC16:26
*** diablo_rojo_phon has joined #opendev16:30
*** dougsz has quit IRC16:30
clarkbonce things settle down on the zuul side I'm going to start prepping an etherpad for tomorrows gerrit project renaming (and followup on any needed changes for that etc)16:35
clarkbone thing that comes to mind is last time we did this we had a problem because we ran manage-projects with a sequence of projects.yaml states. Do we have opinions on whether or not the changes to projects.yaml and friends should be squashed? I believe that if we properly stop ansible the expectation is that our automation will do the right thing?16:36
clarkbthings to think about, but I'll not dig in until after zuul is happy16:36
*** dtantsur is now known as dtantsur|afk16:37
fungiyeah, once the openstack release team meeting concludes i'll get back to troubleshooting those afs-related build failures16:37
fungiand it just did16:37
clarkbalso I feel pulled in a hundred different directions so going to try and stay focused on the zuul side of things as that seems the most important at the moment16:38
openstackgerritMerged opendev/base-jobs master: Add ensure-pip to python release pre-run playbook  https://review.opendev.org/74269416:47
clarkbcorvus: ^16:48
corvuswoot16:48
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Update upload-logs-s3 with access/secret key variable documentation  https://review.opendev.org/74271016:52
*** marios|out has quit IRC17:01
*** markmcclain has quit IRC17:14
*** markmcclain has joined #opendev17:16
AJaegerconfig-core, infra-root, FYI, I'm on vacation the next 10 days and will go offline tomorrow night.18:31
corvusAJaeger: enjoy!18:32
fungiAJaeger: sounds great--hope you have a good time!18:32
AJaegerI'll try my best - thanks!18:33
fungijust to wrap up the earlier questions around afs write failures from ze11, i could not reproduce them. i was able to write to the same tree from ze11 (though i did it using my kerberos credentials, not the base job's)18:41
*** shtepanie has joined #opendev18:53
openstackgerritMerged zuul/zuul-jobs master: Reinstate docker tag cleanup  https://review.opendev.org/74246118:58
mnaserAJaeger: have fun!!19:24
clarkbinfra-root notes going up at: https://etherpad.opendev.org/p/gerrit-2020-07-2419:34
clarkbI'll write up the rename recording change shortly too19:34
openstackgerritClark Boylan proposed opendev/project-config master: Add record for project renames on July 24, 2020  https://review.opendev.org/74273119:39
clarkbinfra-root I think https://etherpad.opendev.org/p/gerrit-2020-07-24 and its associated changes are ready for reviewand the content is now complete19:45
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role  https://review.opendev.org/74273219:46
AJaegerclarkb: want to add a WIP to https://review.opendev.org/#/c/738979/ ? I'll remove mine now so I don't forget.19:48
clarkbAJaeger: you can leave yours as us gerrit admins can remove it too :)19:49
AJaegerclarkb: Ok, then I leave it19:49
clarkband you should enjoy your vacation. I'm not sure what the istuation is like in your part of the wrold, but I was able to relax at home and get some things done around here when I took a week off. Not quite the same btu still worthwhile19:50
fungiwell, we *can* delete it but i think we have to delete the +2 at the same time19:50
clarkbfungi: there is no +219:50
clarkbso I think its fine19:50
fungion 738979?19:50
clarkboh the +2 was just added19:50
AJaegerclarkb, gmann : -1 on https://review.opendev.org/#/c/739286 - could you check my comment, please?19:52
clarkblooking19:52
AJaegerclarkb: travelling inside Germany, it seems safe to travel again - with lots of masks ;)19:53
clarkbAJaeger: oh good point on your comment, though I think we can also merge as is and figure it out after the rename if necessary. Let's see what gmann says19:53
clarkbI saw the official jobs removal and that made sense and I attributed the other thing to that as well19:54
clarkbbtu the other thing is separate19:54
clarkbgmann: ^ please take a look so that we are ready for tomorrow19:54
AJaegerclarkb: yep, we could. I could also patch quickly either way...19:54
AJaegerclarkb: removed the -1 and left a comment that a followup is fine as well.19:56
clarkbthanks19:56
clarkbI've double checked the fixes I found from the last rename and it seems we've merged them so I think we are good from that side of things20:01
openstackgerritMerged zuul/zuul-jobs master: Disable base role testing that runs code on localhost  https://review.opendev.org/74253720:04
openstackgerritMerged zuul/zuul-jobs master: Collect partial subunit files  https://review.opendev.org/74252720:04
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils  https://review.opendev.org/74273620:06
*** DSpider has quit IRC20:15
corvusclarkb, fungi, ianw: https://github.com/pyca/cryptography/issues/5339#issuecomment-663193431  i'll etherpad a reply to this20:19
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role  https://review.opendev.org/74273220:25
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils  https://review.opendev.org/74273620:25
fungicorvus: ack, i  also worry a little about our current arm64 test capacity depending on what their test volume realistically looks like20:39
fungibut i expect we'll have to see it for ourselves20:42
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role  https://review.opendev.org/74273220:44
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils  https://review.opendev.org/74273620:44
corvusclarkb, fungi, ianw: https://etherpad.opendev.org/p/aQ_-tmssILd6UXInsOTT20:50
corvusi apparently am having less trouble writing today than yesterday :)20:50
fungithat makes one of us! ;)20:50
clarkbthat looks good. Maybe mention that our arm resources currently come from a single donor which makes their long term viability perhaps slightly less solid than our x86 options? That said linaro has done an excellent job and is involved and I'm not too worried20:52
corvushow many clouds we got now?20:53
clarkbfor arm its one20:54
corvustotal20:54
clarkblinaro, rax, inap, vexxhost, limestone, airship-citycloud (sort of when it owrks), ovh I think that is it?20:55
fungidepending on whether you want to count regions20:55
corvusi'll say "6+" :)20:55
fungialso openedge presently offline but back soon20:55
clarkbopendedge is offline ya20:56
clarkbbut intended to return20:56
corvusThe ARM resources also come from a single donor (whereas we have 6+ clouds providing x86 nodes); we consider that a risk, but the provider, Linaro, has been fairly reliable.20:56
corvusclarkb: added that sentence20:56
fungilgtm20:56
clarkbyup lgtm as well20:56
corvusokay, i'll send20:56
fungithanks!20:57
corvusdone20:59
fungialso just glad there's someone out there who uses parenthetical expressions as often as i do20:59
fungii treat english as a lisp variant20:59
corvusthere should be a name for the (closing parethetical smiley :)21:07
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role  https://review.opendev.org/74273221:07
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils  https://review.opendev.org/74273621:07
corvusfungi: also -- it must be said -- i read a lot of victorian novels --- which is probably where i developed my fondness for en-dashes (and em-dashes)!21:08
fungiit happens21:19
fungiand yeah, the place where emoticons and punctuation meet... therein lies chaos21:20
clarkbI think that is why everyone uses animated gifs now21:21
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role  https://review.opendev.org/74273221:22
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils  https://review.opendev.org/74273621:22
*** JayF has quit IRC21:44
*** JayF has joined #opendev21:45
gmannclarkb: AJaeger i can keep doc job in that patch itself, I am not sure if we can do it in in-repo (at least fast)  as this repo acl group members are empty -https://review.opendev.org/#/admin/groups/994,members21:47
openstackgerritGhanshyam Mann proposed openstack/project-config master: Rename transparency-policy from openstack/ to osf/ namespace  https://review.opendev.org/73928621:49
gmannclarkb: AJaeger ^^ updated.21:49
clarkbgmann: thanks21:49
JayFIs there a way to subscribe to a project in gerrit? If I want to get an email about every new change posted to a project? I feel like it should be simple, but I can't find docs or figure it out in the web ui.22:06
JayFIn typical fashion; I've been looking for 15 minutes and find it right after asking. In case someone finds this later in logs: top right corner, click your name -> Settings -> "Watched Projects"22:07
clarkbJayF: yes, gerrit calls it watching and you do it via https://review.opendev.org/#/settings/projects22:07
JayFThank you :)22:08
clarkbah you found it too :)22:08
clarkbalternatively you can use gertty and set up project subscriptions in it as well22:08
JayFThat always happens. I give up, ask, then decide to look again and it's like I'm magnetized to the answer.22:08
*** tosky has quit IRC22:59
*** mlavalle has quit IRC23:31
*** shtepanie has quit IRC23:32

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!