Tuesday, 2020-06-09

ianwclarkb: wrt https://review.opendev.org/#/c/733034/ there's no puppet5 anywhere, is there?00:07
clarkbianw correct yhe upgrade from 4 to 5 is significant aiui and part of why we decided to pivot00:10
ianwok i might split out the removal of that testing bit; *finally* we have rpms for openafs on x86-centos7 and x86/aarch64 centos800:10
ianwwhich means we can now publish the wheels00:11
*** tkajinam has quit IRC00:18
openstackgerritmelanie witt proposed opendev/elastic-recheck master: Add query for ServersNegativeTestJSON bug 1882608  https://review.opendev.org/73439700:20
openstackbug 1882608 in OpenStack Compute (nova) "DELETE fails with HTTP 500, StaleDataError: UPDATE statement on table 'instance_mappings' expected to update 1 row(s); 0 were matched" [Undecided,New] https://launchpad.net/bugs/1882608 - Assigned to melanie witt (melwitt)00:20
*** tkajinam has joined #opendev00:25
openstackgerritIan Wienand proposed opendev/system-config master: ARM64 openafs role tests  https://review.opendev.org/73303400:28
openstackgerritIan Wienand proposed opendev/system-config master: Remove Puppet 5 testing  https://review.opendev.org/73439800:28
openstackgerritIan Wienand proposed opendev/system-config master: Integration tests: update debian stable to Buster  https://review.opendev.org/73439900:28
ianwthat's the stack00:29
*** tkajinam_ has joined #opendev00:33
*** tkajinam has quit IRC00:33
openstackgerritIan Wienand proposed opendev/system-config master: ARM64 openafs role tests  https://review.opendev.org/73303400:37
*** Meiyan has joined #opendev00:58
fungiianw: if you get a sec, rackspace support mailed us a bit ago about an outage for the grafana-MySQL trove instance01:09
fungimay want to make sure all's well with the service01:10
*** xiaolin has joined #opendev01:10
ianwfungi: seems ok, no errors i can see01:12
fungicool, thanks. they may end up migrating it to another host, if they do there will probably be a second e-mail01:14
fungiat least that's how these usually go01:15
openstackgerritMark T. Voelker proposed opendev/irc-meetings master: Update InteropWG Meeting  https://review.opendev.org/73440201:16
*** xiaolin has quit IRC01:58
mordredianw: I think stuff there is safe to land yes?01:59
mordredianw: (your afs stack)02:00
mordredianw: +2 from me on the stack - and +A on the first one02:01
ianwmordred: yep, i think thanks; everything can install openafs and get to /afs/openstack.org ... so02:02
ianwshould be ok!02:02
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Do not fail in a venv when activate_this.py is not found  https://review.opendev.org/73171903:13
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Remove virtualenv activation  https://review.opendev.org/73441403:13
*** xiaolin has joined #opendev03:15
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Remove virtualenv activation  https://review.opendev.org/73441403:27
*** xiaolin has quit IRC03:36
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Remove virtualenv activation  https://review.opendev.org/73441403:36
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Remove virtualenv activation  https://review.opendev.org/73441403:42
openstackgerritMerged opendev/system-config master: openafs-client: arch fixes  https://review.opendev.org/73304003:52
*** factor has joined #opendev04:35
*** ykarel|away is now known as ykarel04:42
*** ysandeep|away is now known as ysandeep05:22
openstackgerritMerged openstack/diskimage-builder master: Update readme to clarify an ubuntu bionic image is built  https://review.opendev.org/73399005:23
openstackgerritMerged openstack/diskimage-builder master: Add back pep8 and tarball jobs  https://review.opendev.org/73376805:25
*** sgw has quit IRC05:42
*** hashar has joined #opendev05:53
openstackgerritMerged openstack/diskimage-builder master: Cleanup py27 and docs support  https://review.opendev.org/73376205:56
openstackgerritIan Wienand proposed openstack/project-config master: Drop pip-and-virtualenv from images  https://review.opendev.org/73442806:32
*** ravsingh has joined #opendev07:25
*** tosky has joined #opendev07:32
hrwianw and rest of team: thanks for working on wheels07:48
*** rpittau|afk is now known as rpittau07:54
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** DSpider has joined #opendev08:04
*** ysandeep is now known as ysandeep|afk08:04
*** xiaolin has joined #opendev08:19
*** sshnaidm|afk is now known as sshnaidm08:22
openstackgerritAndreas Jaeger proposed openstack/diskimage-builder master: Switch to newer openstackdocstheme and reno versions  https://review.opendev.org/73249008:23
*** ykarel is now known as ykarel|lunch08:31
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Add support for CentOS 8 Stream  https://review.opendev.org/73408308:51
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Add support for CentOS 8 Stream  https://review.opendev.org/73408308:52
*** priteau has joined #opendev09:00
openstackgerritRiccardo Pittau proposed openstack/diskimage-builder master: Add .eggs to gitignore  https://review.opendev.org/73446909:02
*** hashar has quit IRC09:06
*** ykarel|lunch is now known as ykarel09:15
openstackgerritRiccardo Pittau proposed openstack/diskimage-builder master: Convert multi line if statement to case  https://review.opendev.org/73447909:18
*** xiaolin has quit IRC09:26
*** ysandeep|afk is now known as ysandeep09:39
*** xiaolin has joined #opendev09:42
openstackgerritMerged openstack/diskimage-builder master: Do not fail in a venv when activate_this.py is not found  https://review.opendev.org/73171909:54
*** ykarel is now known as ykarel|afk10:00
*** Meiyan has quit IRC10:00
openstackgerritMerged opendev/irc-meetings master: Update InteropWG Meeting  https://review.opendev.org/73440210:05
*** rpittau is now known as rpittau|bbl10:11
openstackgerritMerged openstack/diskimage-builder master: Drop six usage  https://review.opendev.org/73376510:28
*** priteau has quit IRC10:29
openstackgerritMerged openstack/diskimage-builder master: Debuntu: add apt-transport-https  https://review.opendev.org/71678810:33
*** ykarel|afk is now known as ykarel10:35
*** tkajinam_ has quit IRC10:39
*** ykarel is now known as ykarel|mtg10:43
*** priteau has joined #opendev10:49
openstackgerritSimon Westphahl proposed openstack/diskimage-builder master: Use kpartx option to update partition mappings  https://review.opendev.org/72882410:55
openstackgerritSagi Shnaidman proposed zuul/zuul-jobs master: Add ansible collection roles  https://review.opendev.org/73036011:03
*** ykarel|mtg is now known as ykarel11:06
openstackgerritSagi Shnaidman proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible  https://review.opendev.org/73458411:12
*** priteau has quit IRC11:16
*** tkajinam has joined #opendev11:32
*** ravsingh has quit IRC11:59
*** rpittau|bbl is now known as rpittau12:11
openstackgerritRiccardo Pittau proposed openstack/diskimage-builder master: Convert multi line if statement to case  https://review.opendev.org/73447912:27
*** lpetrut has joined #opendev12:29
*** sgw has joined #opendev12:33
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Add support for CentOS 8 Stream  https://review.opendev.org/73408312:38
*** Eighth_Doctor is now known as Conan_Kudo13:01
*** Conan_Kudo is now known as Eighth_Doctor13:03
*** Eighth_Doctor is now known as Eleventh_Doctor13:07
*** Eleventh_Doctor is now known as Eighth_Doctor13:08
*** priteau has joined #opendev13:12
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Add support for CentOS 8 Stream  https://review.opendev.org/73408313:32
*** tkajinam has quit IRC13:49
openstackgerritMasayuki Igawa proposed opendev/irc-meetings master: Update QA office hour time  https://review.opendev.org/73461214:02
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: tox: allow tox to be upgraded  https://review.opendev.org/69005714:20
*** ravsingh has joined #opendev14:31
*** lpetrut has quit IRC14:40
openstackgerritJames E. Blair proposed opendev/system-config master: WIP: add Zookeeper TLS support  https://review.opendev.org/72030214:57
openstackgerritSagi Shnaidman proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible  https://review.opendev.org/73458414:57
*** ravsingh has quit IRC15:03
*** xiaolin has quit IRC15:05
*** olaph has joined #opendev15:20
*** hashar has joined #opendev15:40
clarkbgitea 1.12 continues to be updated, I wonder how they decide when the release is ready15:41
clarkbmordred: ^ you don't happen to know do you?15:41
mordredclarkb: I do not know15:41
clarkbmaybe I need to set up discord :/15:41
clarkbmordred: it seems my fix for the src/ thing isn't actually applied I think because we have no working inventory to apply it with15:42
clarkbmordred: can you double check that and I guess we can edit ansible.cfg directly?15:42
mordredclarkb: oh - yeah - we might just have to fix the config file by hand15:42
mordredclarkb: https://github.com/go-gitea/gitea/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.12.015:43
clarkb(I noticed because LE renewals are still not running)15:43
mordredclarkb: perhaps they continue to land things related to that milestone?15:43
clarkbmordred: that would make sense15:43
clarkbmordred: did you want to edit the ansible cfg?15:43
clarkb(I feel like you have a better grasp of that than I do)15:43
mordredclarkb: sure!15:43
mordredclarkb: done15:44
mordredclarkb: want me to re-run le?15:45
mordredclarkb: (I'm re-running teh install-ansible playbook to make sure it's all working)15:46
clarkbmordred: ya le would be good to run too15:46
*** ykarel is now known as ykarel|away15:46
openstackgerritJames E. Blair proposed opendev/system-config master: Fake zuul_connections for gate  https://review.opendev.org/73092915:47
mordredclarkb: done15:58
clarkband etherpad has a new cert15:59
* clarkb finds breakfast16:02
openstackgerritMonty Taylor proposed opendev/system-config master: Run zuul-executor using docker  https://review.opendev.org/73396716:09
openstackgerritMonty Taylor proposed opendev/system-config master: Cleanup old direct install files  https://review.opendev.org/73463616:09
openstackgerritMonty Taylor proposed opendev/system-config master: Revert "Cleanup old direct install files"  https://review.opendev.org/73463716:09
*** rpittau is now known as rpittau|afk16:09
mordredcorvus: for that ^^ to work I think we're gonna need openafs-krb5 in the executor image. do we think that's too much to add to the zuul/zuul-executor images?16:11
openstackgerritEmilien Macchi proposed openstack/project-config master: Retire Paunch  https://review.opendev.org/73464016:14
mordredcorvus: or alternately we could make a downstream image in system-config - but I worry that the CD story would suffer a bit if we did16:14
fungiclarkb: there's a weechat plugin for discord, but be aware that unsanctioned clients are essentially against the discord tos and can get your account terminated. such a lovely chat platform they have there16:19
mordredclarkb: when I interact with the gitea people on discord I've just used the web interface16:19
mordredclarkb: https://discord.com/channels/322538954119184384/53766868617396226016:20
fungithey expose a full-featured api for writing bots to do whatever you like in channels there, but expressly state that they can't be used for connecting your own client interfaces16:20
fungisomething they refer to as "self-bots"16:20
openstackgerritClark Boylan proposed opendev/system-config master: Be explicit about using python3 in docker images  https://review.opendev.org/73464716:32
clarkbinfra-root ^ what I'm finding skimming through our use of python, python2 and python3 is that it isn't always immediately clear what `python` means. I've got several changes that do some version of ^ now to make that a lot more clear and hopefully that will make the python2 cases stand out a lot more over time16:32
corvusmordred: this is pretty borderline -- but zuul does explicitly support the openafs pag sysctl, so i think maybe including it is probably okay.  it's a pretty small package.16:34
corvusmordred: if things get out of hand down the line, we could always borrow a page from debian's exim playbook: "zuul/zuul-executor" and "zuul/zuul-executor-heavy" :)16:35
corvuswhere we would include the kitchen sink in heavy16:36
corvusbut i don't think we're there yet16:36
mordredcorvus: yeah - I agree with both things16:42
mordredcorvus: openafs-krb5 is all we need? or do we also need openafs-client?16:42
*** hashar has quit IRC16:43
*** hashar has joined #opendev16:46
corvusmordred: i don't think we need the client16:49
corvusmordred: hopefully it isn't package-required by openafs-krb5 though...16:49
corvusmordred: are we going to do the seccomp thing?16:50
corvusmordred: sorry, i lost track a bit of where we were last weekend -- what did you learn when you tried it out for realz?16:51
mordredcorvus: I do not believe we need the seccomp16:58
mordredcorvus: I ran the various afs things in a container on ze01 and they worked, and the tokens weren't visible from other containers16:59
AJaegerinfra-root, are we ready to drop pip-and-virtualenv from our images? ianw pushed https://review.opendev.org/734428 for this, please review16:59
mordredcorvus: the container image is still on ze01 if you want to verify16:59
mordredcorvus: also - the executor is still shut down on ze01 so that we wouldn't bork executor afs in any way - so if you want to verify, cool - if not, we should probably restart the executor there17:00
mordredcorvus: also - the client isn't package reuqired by openafs-krb5 - although things like the fs command are in client and not krb517:03
corvusmordred: okay, i'll try some testing there too17:13
mordredcorvus: image is zuultest17:13
mordredcorvus: that's the zuul-executor image with openafs-krb5 and openafs-client both installed in it17:13
mordred(client because I was using it to do fs la things)17:14
*** mlavalle has joined #opendev17:19
openstackgerritMerged zuul/zuul-jobs master: test-playbooks: avoid warnings with shell/command  https://review.opendev.org/73160517:20
*** priteau has quit IRC17:21
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Enable linting of test-playbooks  https://review.opendev.org/73147117:24
*** hashar has quit IRC17:25
fungitroubleshooting refstack/openstackid interactions this morning, it's come to my attention that we have the expired addtrust ca cert in 21 separate places in our hiera... i wonder if we should try to clean that up17:27
fungiapparently requests on older versions of python chokes if there's an expired intermediate served, even if it's not needed to validate the server's cert17:27
clarkbfungi: we should cross check against unneeded hiera data due to LE usage too17:28
fungii expect many of these are in now unreferenced hostvars/groupvars, yes17:29
clarkbit wouldn't surprise me if we've left old cert data around even after switching to LE for some domains17:29
clarkbbut ya I think cleaning up unused certs and updating those that have the expired one is a good thing17:29
fungiwe absolutely have. there's one in group_vars/etherpad.yaml for example17:29
mordredyeah - I'm certain that the hiera data has _tons_ of old unused cruft17:30
clarkbmordred: on eavesdrop we haev 3 many week old accessbot containers still running17:31
clarkbmordred: I've noticed this as part of my python3 audit17:31
clarkbcan those be killed?17:31
fungiethercalc, etherpad, files, git-server, openstackid-dev, openstackid, storyboard, survey, translate, wiki, zuul-scheduler, ask, firehose, refstack, static17:31
fungi(7 hits in group_vars/files.yaml)17:32
clarkbetherpad, all of files, zuul-scheduler, and static should all be LE now I think17:33
clarkbfiles and static were combined into static.opendev.org17:33
clarkband in that process got converted to LE I believe (but we should double check each domain separately)17:33
fungimore so, i'm wondering if there's a way we can programmatically identify unused hostvars/groupvars for cleanup17:37
*** ysandeep is now known as ysandeep|away17:49
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: tox: allow tox to be upgraded  https://review.opendev.org/69005717:54
AJaegerinfra-root, the openstack tenant has still two config errors. These will be fixed by https://review.opendev.org/#/c/731690/ - what do you think of force merging that? The other branches have been fixed18:05
clarkbAJaeger: in the past I've asked people to stop running the broken jobs when making those updates18:06
clarkbthen we don't need to force merge and they can reenable the jobs when they are working18:06
clarkbif that isn'y possible for some reason we can force merge18:06
AJaegerclarkb: let me try that first...18:06
*** sshnaidm is now known as sshnaidm|afk18:07
mordredclarkb: yes - although I think we should maybe think about what to do about hung accessbot processes18:40
mordredclarkb: I *think* in the past those have been because of network connectivity and accessbot itself gets stuck in some sort of retry/wait loop18:41
*** iurygregory has quit IRC18:43
openstackgerritGhanshyam Mann proposed openstack/project-config master: Rename interop repos from openstack/ to osf/ namespace  https://review.opendev.org/73466918:47
*** iurygregory has joined #opendev18:47
clarkbmordred: connectivity to freenode? maybe we can add a timeout to the initial connection attempt and if that raises just exit and try again another time?18:53
mordredclarkb: yeah - have you deleted the containers yet? we could strace what's going on18:56
clarkbmordred: I have not18:58
clarkb(figured there was potential for people wanting to debug furhter)18:58
mordredyea - I'm poking real quick - seems laggy18:59
mordredclarkb: cannot exec in a stopped state: unknown18:59
mordredclarkb: I think they're just sad19:00
openstackgerritRafael Folco proposed openstack/diskimage-builder master: [DNM] test: qemu-img convert cache=writethrough  https://review.opendev.org/73467719:12
*** lpetrut has joined #opendev19:20
openstackgerritMonty Taylor proposed opendev/system-config master: Run zuul-executor using docker  https://review.opendev.org/73396719:26
clarkbfungi: I did think about programmtically identifying them but I think it is difficult because you'd need to identify if puppet code also needs removal to be completely accurate20:01
ianwinfra-root: if i could get one more eye on https://review.opendev.org/#/q/status:open+project:opendev/system-config+branch:master+topic:arm64-openafs that will fix up all the openafs clients20:01
clarkbfungi: that said grepping the names in system-config should get us really closed then we can give it an eyeball test?20:01
fungiclarkb: a good point20:01
fungii suppose any variables not referenced somewhere in system-config are probably fair game, however these are scoped to specific systems and therefore may not actually be used by those systems any longer even though a match is turned up for the same variable name in use for other systems20:02
clarkbI'm going to grab lunch now then will try to get through all the various changes we've linked through in the last hour or so20:03
fungithat's the part i'm not sure how to work out easily20:03
*** lpetrut has quit IRC20:23
clarkbianw: can you check my comment on https://review.opendev.org/#/c/733034/7 I'm not sure how important that is given plans around updating nb0320:24
ianwclarkb: ahhh yes, fair point.  i can add that back.  it is for sure headed for extinction but not quite yet20:26
ianwclarkb: although, actually we never had the test did we20:27
clarkboh maybe not20:28
ianwnodes: name: puppet4 ... groups: name: puppet3 ... what is that test doing?20:29
clarkbit may be testing an upgrade form puppet 3 to 420:30
ianwohhh i see, it runs openafs & puppet3 test on the base node, and just the puppet4 test on the "puppet4" node20:30
ianwspeaking of nb03, i'm not having much luck talking to it ...20:31
ianwneither is cacti20:33
ianwit was SHUTOFF ... i do not know why?20:35
ianwthere's nothing but some nulls in the log, which suggests it didn't stop cleanly i guess20:36
ianw#status log started nb03.openstack.org that had shutoff20:37
openstackstatusianw: finished logging20:37
clarkbI've added some annotations to https://etherpad.opendev.org/p/rax-dns-openstack-org20:38
openstackgerritIan Wienand proposed opendev/system-config master: Add puppet3 tests to xenial arm64  https://review.opendev.org/73469820:38
ianwclarkb: ^ that should update from that comment20:38
clarkbthanks I've approved the parent and +2'd the child20:39
clarkbfungi you have a fungitest.openstack.org in the dns zone not sure if that can be cleaned up or nto but you may want ot annotate it either way20:40
clarkbI've not approved https://review.opendev.org/#/c/734428/1 to give it a chance for last lookover, but I think we've crossed the period of time where we said we would make the change?20:42
clarkbinfra-root ^ that is the change to switch base images to ones with less pip and virtualenv in them20:42
corvusmordred: what i observe testing afs in docker on ze01:  without seccomp=unconfined:  host and all docker containers share a keyring.  each zuul-bwrap instance gets its own keyring which is not shared with either host or docker containers.  with seccomp=unconfined, i saw the same behavior.  i think that means we should be gtg with afs on the executor, with the caveat that we should remember the20:47
corvushost==docker container issue.  honestly, considering the way we're using docker (host networking, bind-mount volumes) that's probably pretty intuitive for us.20:47
openstackgerritMerged opendev/system-config master: Remove Puppet 5 testing  https://review.opendev.org/73439820:59
openstackgerritMerged opendev/system-config master: Integration tests: update debian stable to Buster  https://review.opendev.org/73439920:59
clarkbianw: left some nits on https://review.opendev.org/#/c/728739/3 and it needs a rebase but overall lgtm21:00
ianwclarkb: thanks, will go over soon21:01
ianw... "ERROR! the role 'kerberos-client' was not found in" ... it seems something else is up with the wheel jobs now21:08
*** larainema has quit IRC21:11
ianwit's not including the system-config roles ... hmm21:14
clarkbianw: things were moved around21:15
clarkbpossible you have to use base/foo instaed of foo now?21:15
ianwyeah i think i see where i've dropped it ... i moved the wheel build jobs into o-z-j so we could speculatively test changes to them, but i appear to have dropped the roles inclusion on the publish side which remains in p-c for periodic and afs key purposes21:21
ianwi will fix *that* and then ... we should have all the wheels! :)21:22
ianwapparently the opendev meeting is early enough for me that my phone's new "digital wellbeing" thing thinks i should be asleep and puts my phone into greyscale mode ...21:23
ianw"reducing the attention-grabbing nature of many app icons" ... and causing me great confusion until i figured out what it was :)21:25
openstackgerritIan Wienand proposed openstack/project-config master: Wheel publish jobs: include system-config roles  https://review.opendev.org/73470421:30
ianwclarkb: ^ should do it21:30
clarkbI've found the first definite "its python2 and it will be painful" tool: meetbot21:31
fungiluckily, "we have a spec"21:38
clarkbya all that is on the etherpad now21:39
fungii'd love to use py3k as a solid driver for that spec21:42
*** rajinir has quit IRC21:42
*** rajinir has joined #opendev21:42
clarkbinfra-root if you get a chance https://review.opendev.org/#/q/topic:explicit-python3 has some minor cleanups to make the python3 checking more clear21:52
clarkbwhat I'm finding is a good chunk of things are running under python3 but called `python`. Another chunk are python3 capable (at least as reported by unittests or other usages) but we run under python2. And then another chunk are simply not python3 able like meetbot21:54
clarkbwhat is encouraging with that is with some minor config management change we may be able to make a good chunk of these python321:54
openstackgerritClark Boylan proposed opendev/system-config master: Cleanup old puppet management of release-volumes.py  https://review.opendev.org/73471122:04
clarkbianw: ^ noticed that while looking at things. You don't have a similar clenaup already do you?22:04
ianwclarkb: lgtm, i don't think so22:09
*** tkajinam has joined #opendev22:43
*** tosky has quit IRC22:59
clarkbfungi: I'll send a notice to service-announce@lists.opendev.org that we're gonna have a short gerrit outage for project renames at 2100UTC Friday23:09
fungigood idea23:09
fungithough i expect nobody will care at 21z on a friday23:10
fungibetter safe than sorry23:10
*** DSpider has quit IRC23:14
openstackgerritTim Burke proposed opendev/bindep master: Tolerate missing bindep.txt when using --profiles  https://review.opendev.org/73471923:26
clarkbfastly is apparently having problems due to maybe a lcoud provider issue? in any case we may see that as pypi failures23:27
*** ysandeep|away is now known as ysandeep23:42
openstackgerritMerged opendev/system-config master: ARM64 openafs role tests  https://review.opendev.org/73303423:53

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!