Tuesday, 2021-06-22

opendevreviewIan Wienand proposed opendev/system-config master: centos-mirror: add dry run mode  https://review.opendev.org/c/opendev/system-config/+/79736400:08
opendevreviewIan Wienand proposed opendev/system-config master: centos-mirror: exclude ppc64le  https://review.opendev.org/c/opendev/system-config/+/79736500:08
*** ykarel|away is now known as ykarel04:45
*** ricolin_ is now known as ricolin05:05
*** marios is now known as marios|ruck05:15
*** ysandeep|out is now known as ysandeep06:11
ianwso it turns out importing the db from the old trove instance has some sort of primary key clash @ https://bugs.chromium.org/p/gerrit/issues/detail?id=1469006:33
ianwsee if that has an easy answer, if not, treating it as ephemeral data is an option i guess06:34
ianwbut, in general the deployment of the local db, startup of gerrit, etc. is all good on review02.  so we're making progress and getting a little closer to cut over06:40
ianwi hope to have a full checklist for review @ https://etherpad.opendev.org/p/gerrit-upgrade-2021 in short order06:40
*** gthiemon1e is now known as gthiemonge06:46
opendevreviewMerged openstack/project-config master: Enable publish-openstack-python-tarball job  https://review.opendev.org/c/openstack/project-config/+/79704906:51
*** rpittau|afk is now known as rpittau07:06
*** jpena|off is now known as jpena07:37
opendevreviewGuillaume Chauvel proposed zuul/zuul-jobs master: Update run-buildset-registry for readability  https://review.opendev.org/c/zuul/zuul-jobs/+/76244707:45
*** ykarel is now known as ykarel|lunch07:50
opendevreviewchandan kumar proposed openstack/project-config master: Add publish-to-pypi job template for tripleo-operator-ansible  https://review.opendev.org/c/openstack/project-config/+/79741508:48
*** ykarel|lunch is now known as ykarel09:08
*** dviroel|out is now known as dviroel11:20
*** jpena is now known as jpena|lunch11:36
*** ykarel_ is now known as ykarel12:02
*** prometheanfire is now known as Guest18412:25
*** osmanlicilegi is now known as Guest18712:25
*** ChanServ changes topic to "OpenDev is a space for collaborative Open Source software development | https://opendev.org/ | channel logs http://eavesdrop.openstack.org/irclogs/%23opendev/"12:29
*** jpena|lunch is now known as jpena12:31
*** marios|ruck is now known as marios|ruck|call13:10
*** marios|ruck|call is now known as marios|ruck13:25
TheJuliaOut of curiosity, I've been seeing an increase in the number of random post_failures on ironic jobs. Are these tracked/trended at the opendev level? Is there anyway we can make post-failure less faily?13:26
*** marios|ruck is now known as marios|ruck|call13:34
opendevreviewTristan Cacqueray proposed opendev/gear master: Prevent AttributeError: 'Server' object has no attribute 'statsd'  https://review.opendev.org/c/opendev/gear/+/79745113:42
fungiTheJulia: if they're truly random, then probably not. if there are some patterns you can discern, that might give us a clue as to what can be made more robust. we do generally track counts of each result type for each pipeline and project in our grafana via statsd reporting from the scheduler, but that's just raw counts/times. the builds api/webui on https://zuul.opendev.org can be queried for13:51
fungimore detail around what jobs are getting what results, but if you want to look at the logs of any you find there we only keep a month of those13:51
TheJuliafungi: primarily in ironic we're seeing stuff like https://zuul.opendev.org/t/openstack/build/d3159791bcca4ac89aceddbec6c43611 where we're wondering if the path detection script is just failing oddly somehow https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-subunit-output/files/find-testr.sh13:53
TheJuliarpittau: ^^13:54
rpittaujust checked a succesful job and the command has only one line13:54
rpittauthe failing ones has 213:54
rpittausince the script is taking stdout_lines[0] it ends up with an empty entry in the second case13:55
rpittauhere I mean -> https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-subunit-output/tasks/main.yaml#L6813:55
opendevreviewRodolfo Alonso proposed openstack/project-config master: Make explicit the network backend used in the CI jobs  https://review.opendev.org/c/openstack/project-config/+/79745413:59
*** marios|ruck|call is now known as marios|ruck14:00
fungiTheJulia: rpittau: comparing the succeeding and failing examples, i see the successful one runs `/opt/stack/tempest/.tox/venv/bin/stestr last --subunit >>/tmp/subunit...` but the failing one runs just `last --subunit >>/tmp/subunit...`14:01
fungimaybe it's testr detection breaking14:01
rpittaufungi: yeah, because the stdout output is on 2 lines and the first one is empty14:02
fungirpittau: aha, yep, just spotted that as well, now i see what you were saying about two lines earlier14:03
rpittaufungi: yeah, sorry, maybe a bit confusing :)14:03
fungiso it's this script: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-subunit-output/files/find-testr.sh14:05
fungithe last loop in that script seems to guard against returning empty lines14:06
fungisuggesting it's some other command run prior to that emitting a blank line, maybe14:07
rpittaufungi: I' mstarting to think the issue is not on the bash script but in the ansible module, what if we add a strip() after stdout?14:20
rpittauoh that's stdout_lines actually14:22
fungiyes, the stdout from running that script14:25
fungithe script seems to try to avoid emitting blank lines, so it would be good to figure out where the blank is coming from...14:26
*** ykarel is now known as ykarel|away14:35
*** marios|ruck is now known as marios|ruck|call14:44
rpittaufungi: maybe use a filter to create a new list from stdout_lines and exclude empty entries ?14:44
*** marios|ruck|call is now known as marios|ruck14:44
fungirpittau: yeah, i mean ways to work around it are numerous, the bigger question is where the blank is coming from. i'm not personally a fan of code which accumulates a pile of workarounds for unidentified misbehaviors14:50
fungiit's clear that the `type` command can emit blank lines when queried about a nonexistent name, but the script has a filter of sorts already to deal with that14:51
fungimaybe that test -n isn't working as intended? maybe the entry itself has an embedded newline for some reason? and under what circumstances does it arise?14:52
opendevreviewMerged zuul/zuul-jobs master: Update run-buildset-registry for readability  https://review.opendev.org/c/zuul/zuul-jobs/+/76244714:53
fungior could something else run as a part of that script be generating blank lines before it ever reaches that loop?14:53
*** ysandeep is now known as ysandeep|out14:57
*** Guest184 is now known as prometheanfire15:10
*** gthiemon1e is now known as gthiemonge15:38
rpittaufungi: I agree on the "pile of workarounds", just this little issue is getting annoying as it breaks hours of successful CI runs.15:38
rpittauAnyway, as we saw it looks like what's breaking here is that stdout of the command contains a newline, I think it's ansible prepending that for some reason and "unfortunately" not all the times, kind of hard to verify, I don't see debug ansible info in the jobs15:38
fungirpittau: we do collect the ansible json stream as well and publish it next to the text output log, maybe it will have more clues15:40
rpittauoh didn't see that!15:40
rpittauyeah, there you go15:41
rpittau"stdout": "\r\n/opt/stack/tempest/.tox/venv/bin/stestr\r\n"15:41
rpittauI blame ansible15:41
fungiwell, that could just be a representation of what's coming out of the script15:42
fungiwe could add some debugging from the script to stderr i suppose, which shouldn't conflict with parsing the stdout, but since this seems to occur infrequently it would be hard to leverage without merging it to the role15:43
*** marios|ruck is now known as marios|out16:00
*** rpittau is now known as rpittau|afk17:11
gmannclarkb: fungi Need some help on ELK service help things, mainly two question17:24
gmannclarkb: fungi 1. ELK +e-r are used in opendev as well as in other open infra project also right not just openstack17:24
fungi(clarkb is out for the next few days, but i'm happy to answer)17:24
fungigmann: i don't know about "used" but we index logs from any builds which match the logstash filters we've got loaded17:25
fungiso we're potentially indexing logs for more than just openstack projects17:25
gmannok which is from all the projects opendev support17:25
fungithough openstack's ci data accounts for the vast majority of what we're indexing17:25
gmannfungi: 2nd question is about skillset requirement for maintainers, we listed 'developers and system administrators ' in this  https://governance.openstack.org/tc/reference/upstream-investment-opportunities/2021/community-infrastructure-elk-maintainer.html17:27
fungiwell, not necessarily "all projects." not all open infrastructure foundation projects use zuul, and of the ones which do some of them don't use zuul exclusively and also have other ci systems. but also not all the projects hosted in opendev and using opendev's zuul deployment are part of an open infrastructure project, we have plenty of wholly independent projects we're hosting as well17:27
gmannI see17:27
gmannI think that skillset requirement is correct, we still need developer skill not only system admin/ops only as we need to maintain code for e-r especially 17:28
fungithe skillset is going to need to be people capable of designing and running a replacement system from the ground up, i think. the current deployment is going to be more work to update to a maintainable state than it would be to recreate from scratch with new technologies17:28
fungiwe have basic design documents explaining how the current implementation was intended to work, but a replacement wouldn't necessarily need to follow the same architecture (and should probably differ on a number of key points really)17:29
gmannok, for ELK we can say developer skill not required as such but for e-r we need developer on code-maintenance part17:31
fungii can't really speak to "developers" vs "systems administrators" since i don't think there's a clear distinction. at least when i first entered the workforce you needed to be able to write software to be any good at administering systems, and if you couldn't admin things then writing software was especially challenging17:32
*** jpena is now known as jpena|off17:32
fungia good systems administrator is a developer and a good developer is a systems administrator. they're one and the same17:32
fungiin this particular case it's systems software, so if you don't understand systems then the software isn't going to make sense17:33
gmannyeah, that's more right interpretation. 17:33
fungialso e-r and elk are implementation details, any of that can be changed in the redesigned system17:34
fungiand yeah, the systems-administrators-are-developers idea has been rebranded "devops" in recent years, but isn't even remotely a new idea, it's how things have worked for many decades17:34
gmannyeah, devops is right term I can add.17:35
fungithe tools have changed, but the basic skills haven't17:35
gmannthanks fungi for clarifying the requirement. I think this is enough info for now.  17:35
*** iurygregory_ is now known as iurygregory18:39
opendevreviewJeremy Stanley proposed opendev/infra-manual master: Switch docs from referencing Freenode to OFTC  https://review.opendev.org/c/opendev/infra-manual/+/79753119:37
opendevreviewIldiko Vancsa proposed opendev/infra-manual master: Update the documentation with the recent IRC changes  https://review.opendev.org/c/opendev/infra-manual/+/79753219:44
fungii'm stepping out to meet some folks for dinner, but should be back in a couple of hours tops19:45
*** dviroel is now known as dviroel|out20:59
opendevreviewJames E. Blair proposed openstack/project-config master: Add zuul/gating.dev repo  https://review.opendev.org/c/openstack/project-config/+/79754322:01
opendevreviewJames E. Blair proposed openstack/project-config master: Adding gating.dev zone repo  https://review.opendev.org/c/openstack/project-config/+/79754422:06
opendevreviewJames E. Blair proposed openstack/project-config master: Adding gating.dev zone repo  https://review.opendev.org/c/openstack/project-config/+/79754422:08
opendevreviewJames E. Blair proposed opendev/system-config master: Add gating.dev zone to ADNS  https://review.opendev.org/c/opendev/system-config/+/79754522:11
ianwfungi: if you have a sec to loko over https://review.opendev.org/c/opendev/system-config/+/797365 and the files it deletes in the referenced dry-run log, that would be great22:13
ianwthat's the ppc removal 22:13
fungiaha, thanks!22:15
opendevreviewJames E. Blair proposed openstack/project-config master: Adding gating.dev zone repo  https://review.opendev.org/c/openstack/project-config/+/79754422:18
ianwthis patch review table created by gerrit is very weird22:23
ianwi can not see how it would not be setting the primary key constraints correctly22:24
ianwalthough we should have executed https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/schema/MariaDBAccountPatchReviewStore.java#5722:25
fungicould it be the dump from our old db which is the problem?22:25
ianwwell this is using the inbuilt gerrit tool to migrate that db22:25
fungimaybe it had a different primary key in the past and migrations didn't undo it?22:25
fungii will admit to still finding relational databases a bit of witchcraft22:26
ianwi can't see what is making our table22:26
corvusianw: i like "show create table foo" for things like this too22:27
ianwwe must have created the "generic table" as it's file_name is a varchar(4096) -> https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/schema/JdbcAccountPatchReviewStore.java#19822:28
ianwnot the mariadb version, which is varchar(255) -> https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/server/schema/MariaDBAccountPatchReviewStore.java#6222:29
corvusianw: there's a "file_id" column in mysql which is apparently part of the primary key?22:29
corvusand that is omitted from the mariadb schema22:29
ianwyeah, and not in the above queries that should be creating the table22:30
corvusso, to be clear, it seems like a straightforward case where the old schema does not match the new one.  i think in this case a migration script should ignore duplicate key errors on insert and continue22:32
ianwyeah, i wonder when it changed.  they must have moved from having a separate auto inc for the primary key to making the primary key that combo of fields22:33
corvus(and we know they've played fast and loose with this table in upgrades before; it's not surprising (but disappointing) to me that our schema doesn't match what's in the source code)22:34
ianwi'm guessing this has to do with our old database not using innodb tables22:40
ianwCreation of the account_patch_reviews table failed on mysql with the22:40
ianw    file_name column being set to varchar(4096), see [1].22:40
ianw    mysql has a limit of 767 bytes max key length for InnoDb tables.22:40
ianwsince our trove db has a file_name column of 4096 it must be different22:41
fungiso was it myisam on the trove instance then?22:41
ianwit's still unclear to me why it chose a different primary key22:42
ianw(on the old trove side)22:42
fungii still strongly suspect it's a (perhaps lost to the annals of time) gap in migrations for gerrit22:43
corvusi mean, it didn't choose it... it's very likely what fungi said22:43
fungiwe've been carrying over the same databases since, i dunno, gerrit 2.1?22:43
fungithey didn't take upgrades nearly as seriously once upon a time22:43
corvusthey really don't care about this table; they also say if anything goes wrong with it, just drop it and start over22:44
fungipart of our upgrade plan was also "if something goes wrong with that db, we'll just wipe it"22:44
fungiit's really just checkmarks for filenames in reviews to signal whether you've "seen" them22:45
corvusmy personal opinion: it's worth spending about 30 minutes total trying to port the data over; otherwise i'd drop it :)22:45
fungii can't remember the last time i've actually relied on that feature in gerrit22:45
fungiif it had a switch to turn the feature off which allowed you to not need a separate database to track those checkmarks, i'd vote for disabling it22:46
fungiianw: different topic, but before i go pushing a fix, do you agree this is incorrect? https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/limnoria/templates/meetingLocalConfig.py.j2#L422:53
fungishould be https://meetings.opendev.org/meetings/ yeah?22:54
fungijust making sure i swing my hammer in the right spot22:54
diablo_rojoSo I was digging around looking for where in system config the general opendev ML is setup? I see where the openstack and kata ones are, but not the non project specific ones. 22:56
diablo_rojoAny pointers?22:56
fungidiablo_rojo: in the host_vars i pointed you to on the ml thread?22:57
fungioh i see what you're asking. the "openstack" file i linked is what you're looking for22:58
fungiwe have 5 different mailing list "sites" on the lists.openstack.org server22:58
fungiit's not an openstack-specific server, it just dates from a time when all our servers were named in the openstack.org domain22:58
diablo_rojooh! I missed the reply sorry!22:59
fungithe lists.katacontainers.io server is the odd one out, it only serves that one site, at some point we'll fold it into the multi-site server23:00
diablo_rojoAh fair enough. I will look a little harder. 23:00
fungiwell, hopefully my reply explains exactly where to add what you're wanting23:01
diablo_rojoGot it. Sorry!23:01
fungiif it isn't clear, happy to add further details23:01
diablo_rojoI dont actually see the reply on the thread in my inbox for some reason, but I will go look at the archives.23:02
fungifeel free to follow up to the ml thread you started or ask in here, either's fine by me23:02
fungidiablo_rojo: http://lists.opendev.org/pipermail/service-discuss/2021-June/000258.html23:03
fungimaybe your mail provider has something against me23:03
diablo_rojoPerhaps.. though its weird I got Mohammed, but not yours. 23:05
diablo_rojoAll good now. I found the opendev section. I really don't think a new mailman site is required. 23:05
fungithis is why i run my own mailserver. i don't like the idea of some company deciding what messages i do and don't get to see23:05
fungiespecially a company with ulterior motives which views me as a product rather than a customer23:06
corvusincidentally, it's pretty easy to set up zuul jobs to publish openedx courses23:07
diablo_rojofungi, its not normally an issue, but yeah.. a bit arbitrary can be frustrating. 23:07
diablo_rojocorvus, oh that is good to know.23:07
fungiit might even be a cool demo to set up23:09
diablo_rojoYes it would!23:10
fungihaving more folks in academia get exposure to the wonders of zuul would be awesome23:10
fungiupstream institute as an open edx course could also be interesting23:12
opendevreviewKendall Nelson proposed opendev/system-config master: Add mailing list for FLOSS MOOC  https://review.opendev.org/c/opendev/system-config/+/79754923:16
ianwfungi: sorry, back from drop-off.  umm, yes although it should redirect23:16
ianwselect t.change_id, t.patch_set_id, t.account_id, t.file_name from account_patch_reviews as t group by t.change_id, t.patch_set_id, t.account_id, t.file_name having count(*) > 1;23:30
ianwERROR 126 (HY000): Incorrect key file for table '/var/tmp/#sql_2c2_0.MYI'; try to repair it23:30
opendevreviewMerged openstack/project-config master: Add zuul/gating.dev repo  https://review.opendev.org/c/openstack/project-config/+/79754323:31
opendevreviewMerged openstack/project-config master: Adding gating.dev zone repo  https://review.opendev.org/c/openstack/project-config/+/79754423:31
fungiianw: well, it doesn't redirect, but i'll add one. we have a redirect for eavesdrop.openSTACK.org but not eavesdrop.openDEV.org, at least not yet anyway23:56
opendevreviewJeremy Stanley proposed opendev/system-config master: Correct the meeting base URL for our meetbot  https://review.opendev.org/c/opendev/system-config/+/79755023:56
ianwyou're right.  now i think about it, i think we don't need eavesdrop.opendev.org at all23:57
fungihowever now we have a number of meeting logs with urls for the latter in them23:57
ianwyeah, that's unfortunate and i guess we need to redirect them.23:58
ianwor we could just edit the meeting logs23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!