Friday, 2020-06-12

openstackgerritMerged opendev/system-config master: Test etherpad with testinfra
mordredI'm not going to dns that - I'm also going to snapshot it and tear it down and stuff00:14
mordredbut it's there for now00:14
openstackgerritIan Wienand proposed opendev/bindep master: Use abstracted virtualenv_command from ensure-pip
openstackgerritIan Wienand proposed opendev/bindep master: Use abstracted virtualenv_command from ensure-pip
openstackgerritIan Wienand proposed opendev/bindep master: Add centos 8 and focal testing
ianwdropping "--verbose" from pip during the wheel build dropped the build log output from 500+mb bz2 compressed (2gb uncompressed) to 1.2mb compressed00:44
*** Meiyan has joined #opendev00:56
mordredianw: WOW01:01
*** seongsoocho has quit IRC01:11
*** vblando has quit IRC01:11
openstackgerritMerged openstack/diskimage-builder master: Remove virtualenv activation
*** vblando has joined #opendev01:19
*** mnasiadka has quit IRC01:19
*** mnasiadka has joined #opendev01:23
*** seongsoocho has joined #opendev01:28
*** ravsingh has joined #opendev03:01
openstackgerritIan Wienand proposed opendev/system-config master: Acutally run system-config arm64 test on an arm64 node
ianw^ that explains why i thought that openafs was working on arm64 xenial, when it actually wasn't on the wheel builder04:14
ianwbut ... we have wheels! ... everything except xenial arm64 (fixed by is populated04:16
openstackgerritIan Wienand proposed opendev/bindep master: Add centos 8 and focal testing
openstackgerritIan Wienand proposed opendev/bindep master: Cull the test bindep file
*** prometheanfire has quit IRC04:28
openstackgerritIan Wienand proposed openstack/project-config master: Add all python versions to bindep tox testing
*** prometheanfire has joined #opendev04:49
*** prometheanfire has quit IRC05:05
*** prometheanfire has joined #opendev05:15
*** sgw has quit IRC05:44
*** ravsingh has quit IRC06:02
ianwfungi: there's a few bindep changes @ that will be helpful to avoid virtualenv issues06:26
jrosseri'm just trying out the centos-7-plain nodes and getting node_failure, this is my change
jrosserthe ubuntu-bionic-plain ones have started running ok06:34
openstackgerritIan Wienand proposed opendev/system-config master: AFS: add note on volume creation servers
openstackgerritIan Wienand proposed opendev/system-config master: Add tool to export Rackspace DNS domains to bind format
ianwjrosser: that is embarrassing, i appear to have overlooked centos-7-plain in
ianwif someone would like to add it, it's just copy paste from centos-8-plain but change 8 to 707:02
openstackgerritJonathan Rosser proposed openstack/project-config master: Add missing centos-7-plain disk images
*** xiaolin has quit IRC07:44
*** xiaolin has joined #opendev07:46
yoctozeptokolla has a problem with infra ubuntu mirrors since 2020-06-10 afternoon (UTC) - trying to install sqlite3 gets us sqlite3 : Depends: libsqlite3-0 (= 3.22.0-1ubuntu0.3) but 3.22.0-1ubuntu0.4 is to be installed - example mirror
yoctozepto(ubuntu bionic)07:51
*** hashar has joined #opendev07:56
*** iurygregory has joined #opendev07:58
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** iurygregory has quit IRC08:02
*** tosky has joined #opendev08:04
frickleryoctozepto: I saw that locally yesterday, too, seems some image got updated faster than the mirror, let me check update logs08:05
fricklerinfra-root: grafana shows ubuntu.mirror being 9 days old, sadly I'm on a different workstation currently and cannot access my ssh key, can someone else have a look, please?08:11
yoctozeptofrickler: thanks for checking08:17
*** tkajinam has quit IRC08:23
*** DSpider has joined #opendev08:25
openstackgerritChris MacNaughton (icey) proposed openstack/diskimage-builder master: Stop bringing the test environment into the mocks
*** xiaolin has quit IRC08:52
*** elod has quit IRC08:57
*** elod has joined #opendev09:03
*** priteau has joined #opendev09:13
*** xiaolin has joined #opendev09:28
*** donnyd has joined #opendev09:36
*** priteau has quit IRC09:39
*** priteau has joined #opendev09:41
*** xiaolin has quit IRC09:53
*** Meiyan has quit IRC09:57
*** nautics889 has joined #opendev10:40
*** ravsingh has joined #opendev11:19
yoctozeptoinfra-root, frickler: any luck with that ubuntu mirror? please let me know if this is a bigger issue as we might be better off just using public ubuntu mirrors in the meantime11:22
nautics889Hello, recently I've encountered an unexpected filter behavior in my OpenStack Horizon admin's dashboard. I deployed Horizon (train release), created an admin account and created two instances in two different project. After that i discovered that filter by project name at /admin/instances/ works some wrong way, it is clearly represented at this screenshot — . Instead of11:31
nautics889dropping results that don't match filter string it just fetches all and return the whole lookup. I started debugging and discovered that actually by pressing of "filter" button it calls ``filter`` method eventually, and it defined at horizon/tables/action exactly in ``FilterAction`` class ( As you can see it is11:31
nautics889just a mock-method without filtering implementation. Does it mean that i should implement filtering on my own?11:31
*** hashar has quit IRC11:45
AJaegernautics889: we cannot help with OpenStack install, this is the channel for developing the infrastructure OpenStack and other repos use for collaboration. Best to ask in #openstack or a more specific channel like #openstack-horizon12:20
*** lpetrut has joined #opendev13:15
*** ravsingh has quit IRC13:29
fungifrickler: yoctozepto: i can start looking into the ubuntu mirror problem now13:31
fungiCould not lock the VLDB entry for the volume 536870949. VLDB: vldb entry is already locked13:32
yoctozeptofungi: kafs again?13:33
fungii don't think kafs is involved here at all, this is a cronjob on an ubuntu xenial instance13:34
fungibut the volume lock on the read only replica looks like the only problem. /afs/ says the content updated 2020-06-12T12:40:55,821215441+00:00 so we probably have current packages in the read-write volume13:35
fungimost likely a vos release took so long its kerberos ticket timed out13:35
fungias soon as i can confirm there's not one in progress i'll delete the stale volume lock and start one with -localauth directly on the fileserver13:36
corvusfungi: i thought we switched all the vos releases to run on a server and use -localauth13:38
corvusor is that just the ones on the new server?13:39
*** auristor has quit IRC13:40
fungithat's just the new server, so rsync-based mirroring scripts like rsync13:41
fungiwe haven't yet done the work to port the reprepro configuration to the newer server13:41
fungianyway, i've confirmed there is no current transaction in progress for that volume13:41
fungiand have now released the lock on the mirror.ubuntu volume13:42
fungii'm holding a flock for that mirror on the mirror-update server in a root screen session so it won't try to update right now13:42
fungiand have vos release running with -localauth on afs01.dfw.o.o13:42
fungiin a root screen session there13:42
fungiStarting ForwardMulti from 536870950 to 536870950 on (full release).13:43
fungithis may take a while13:43
fungii'll keep tabs on it13:43
fungi#status log full release of mirror.ubuntu afs volume is underway ina root screen session on, mirror update flock is held in a root screen session on mirror-update.openstack.org13:44
openstackstatusfungi: finished logging13:44
*** ravsingh has joined #opendev13:45
corvusit's probably too late to delete the nodepool images, since the report goes back to 6-1013:48
*** auristor has joined #opendev13:49
corvusyoctozepto: it's hard to predict how long this will take to catch up, but it could take most of the day.13:50
yoctozeptofungi, corvus: thanks for handling :-) it's fine as long as it unblocks us for the weekend and next week13:52
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: ensure-tox: Avoid Reinstalling tox venv if present
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: ensure-twine: Avoid Reinstalling twine if present
fungiyoctozepto: out of curiosity, what's the environment in which libsqlite3-0 3.22.0-1ubuntu0.4 is being found in your failures? our most recent ubuntu-bionic image still has 3.22.0-1ubuntu0.3 because it's built using our mirrors14:26
fungi(we build our images using our mirrors to avoid these sorts of mismatches)14:27
fungii'm curious to see one of the failed build logs14:28
fungiout most recent image build was
fungier, our most recent14:29
yoctozeptofungi: in-container obviously, logs in a moment14:30
yoctozeptofungi: example log
fungiyoctozepto: were the container images built from our mirrors? or some other package source?14:31
*** mlavalle has joined #opendev14:32
yoctozeptofungi: a different wording log:
yoctozeptofungi: package source is your mirrors but the base images are straight from official ones14:32
yoctozeptofungi: so it's official ubuntu bionic image14:32
yoctozeptofungi: with repos replaced for our installs14:32
fungiahh, okay, so the problem in this case is that the container base images are newer than our mirror state14:34
fungiunrelated to our nodepool images14:34
fungicorvus: ^ do rolling back images in nodepool wouldn't have helped anyway14:34
fungier, so14:34
fungithis is making more sense now14:35
fungithanks for the additional detail, yoctozepto!14:35
fungialso explains why the problem wasn't much more widespread14:36
*** lpetrut has quit IRC14:36
yoctozeptofungi: yw :-)14:38
*** sgw has joined #opendev14:38
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add test-build-container-image-release job
openstackgerritJames E. Blair proposed opendev/system-config master: WIP: add Zookeeper TLS support
mnaserclarkb: i see you put a +2 down but the rename hasn't been done (i can search for project:x/whitebox-tempest-plugin)14:53
mnaseri'm unsure if the rename was actually done or not?14:53
mnaser(i saw AJaeger dropped their -W too without much commentary so just wanted to know -- bc we have a governance change that's depnded on this for a while so yeah)14:54
corvusmnaser: it's scheduled for later today:
mnaseri really should get on the opendev mailing lists.  i'll register today14:54
mnasercorvus: thank you :)14:55
corvusnp :)14:55
corvusmnaser: i'm guessing AJaeger removed W because of the scheduled time (very late in his tz); i'll add mine now14:56
mnasercorvus: ah, that makes total sense now.  always a few steps ahead of us :)14:57
* mnaser would have not thought to do that14:57
corvus(just to be clear, this is a blocking -W i'm talking about :)14:57
clarkbI reviewed the change in preparation for today's renaming. +2'd it because I believe it is ready to go as soon as we are ready15:00
fungii'm seeing if i can work out how to port our reprepro mirror setup from puppet to ansible, and then i'll double-check stuff for the rename maintenance15:06
openstackgerritMerged openstack/project-config master: Deprecate Paunch
fungigrr, this is going to be a lot of erb to j2 loop conversions15:08
fungiand conditionals. not just substitutions15:08
*** ravsingh has quit IRC15:09
AJaegercorvus: thanks for adding your -W; I should have pinged here - didn't want to block it...15:20
AJaegerclarkb: regarding the interop changes: No job is dropped, I double checked. Only the sync to github is dropped.15:21
clarkbAJaeger: hrm I thought pypi publishing was dropped for one or two repos15:27
AJaegerclarkb: it was not there initially - openstack/interop has only the github sync for example15:31
AJaegerso, 2 out of four repos had pypi publishing - and still have it afterwards. All 4 drop the github sync15:31
clarkbgot it15:32
openstackgerritClark Boylan proposed openstack/project-config master: Use https apt mirrors for image builds
clarkbfungi: ^ your comment about using the mirrors to build the images led me to ^. I've based it on ianw's pip and virtualenv change to avoid conflicts, but I figure thats a good hygiene change15:39
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add test-build-container-image-release job
fungiclarkb: good idea!15:44
openstackgerritJames E. Blair proposed opendev/system-config master: WIP: add Zookeeper TLS support
corvusfungi, clarkb, mordred: our work on the zuul testing has paid off -- ^ that change/job is now producing sufficient logs to indicate that the zk tls change is not working and why.  i've made 2 corrections so far based on it.15:46
corvusmind you, the job still succeeds because we don't yet have a way to test that zuul "works".  but we can at least check the logs for startup errors.15:47
mordredcorvus: \o/15:47
clarkbcorvus: cool, I'm glad the sort of simplified zuul install is useful in some way15:47
fungido we have a good example of using loop to iterate over an associative array using both the keys and values from it?15:47
clarkbcorvus: maybe for a testing aspect we can look for data in zk itself?15:47
clarkblook for side effects essentially15:47
clarkbfungi: I think the LE roles have stuff like that15:48
corvusfungi: try zuul.conf.j215:48
fungiooh, will do15:48
corvusespecially the end -- the connections list15:48
clarkboh in jinja not ansible, ya zuul is probably better than LE for that15:48
clarkbcorvus: for example doesn't zuul create the /zuul path?15:48
corvusclarkb: good idea, that should let us at least check the nodepool side -- and yes, now that you mention it, i think there's a /zuul too :)15:49
fungiwell, the j2 example will come in handy as well, i'm going to need to do some similar fancy things in erb conversion15:49
fungibut i was actually looking for ansible, yes15:49
corvusah sorry, i saw erb->j2 earlier and got stuck on that; i see you said "loop" earlier.  i read that as a concept, not a keyword.  communicating is hard.15:50
fungiloop: "{{ query('dict', mydict) }}"15:51
fungiand then refer to item.key and item.value15:51
fungithat seems to be the gist of it anyway15:51
mordredI wanna say there's a with_items?16:16
mordredfungi: so, loop: "{{ query('dict', mydict) | dict2items }}" I suppose16:17
mordredfungi: oh - wait - your query() thing is alsoa way to do that? neat16:18
fungiapparently. at least it's how the letsencrypt roles are using it16:18
fungias an example16:20
fungithough that one only references item.key and not item.value16:20
fungithis one apparently has a nested dict:
fungithough it's using the dict2items filter16:22
*** nautics889 has quit IRC17:19
*** nautics889 has joined #opendev17:23
johnsomHello OpenDev folks. The Octavia team has tagged the Ocata and Pike branches as EOL here: I think the next step is to remove the branches from git. What do I need to do to facilitate that?17:40
fungijohnsom: at the moment, that's a manual action taken by a stable maint or release team representative, or can also be performed by one of our gerrit admins17:41
fungithe next version of gerrit we're now very close to upgrading to will give us the ability to better control branch deletion permissions via acl, which should make automating that in release management tooling a lot easier/safer17:42
fungi(the version we're on now was the first to add an api call for it)17:43
johnsomfungi Ok, Sean was asking if I had requested the deletion last night, which led me here. Lol17:43
fungicool, yeah i can do that if everyone's ready for it17:47
fungijust double-checking stuff before i accidentally delete the wrong thing ;)17:48
johnsomfungi I think we are ready. The eol tags are applied, so we should be good to go17:49
johnsomI appreciate that!17:49
smcginnisfungi: Unless that was recently added, release managers do not have those rights.17:50
fungismcginnis: we usually temporarily added a release manager to a group with those permissions17:51
smcginnisUsually? I've been aware of the last 5 or so times this has come up, and it's always been a matter of asking someone here to do that step. But if there's a different way we should be handling it, let me know.17:52
fungi#status log deleted openstack/python-octaviaclient branch stable/pike (previously d4a5507c99430a7efb4a0ab83a47ca48937b23cf) as requested by johnsom and smcginnis17:52
openstackstatusfungi: finished logging17:52
smcginnisI could see adding a flag to our deliverables file that would still keep the branch, but tell the automation to actually perform a delete of the branch.17:52
fungismcginnis: it's varied, so i meant "usually when it wasn't done by a gerrit admin" (the times where a release manager or stable branch manager has agreed to process a list of them)17:52
smcginnisSo in other words when Tony did a lot of clean up. ;)17:53
fungi#status log deleted openstack/octavia branch stable/ocata (previously c2fdffc3b748f8007c72e52df257e38756923b40) as requested by johnsom and smcginnis17:53
openstackstatusfungi: finished logging17:54
fungi#status log deleted openstack/octavia branch stable/pike (previously 2976a7f0f109e17930db8a61136526ead44ea7e5) as requested by johnsom and smcginnis17:54
openstackstatusfungi: finished logging17:54
fungijohnsom: smcginnis: those were the only three branches you needed deleting, right?17:54
fungi(two for octavia and one for the client)17:55
*** nautics889 has quit IRC17:56
johnsomfungi Correct, thank you17:56
fungiyou're welcome!17:56
*** nautics889 has joined #opendev17:57
johnsomfungi Do we need to do anything for them on github? Or will that sync eventually?18:01
*** priteau has quit IRC18:06
fungijohnsom: the way those mirror jobs are written at the moment, the only trigger when new changes merge to those repositories, but after the next octavia or client change lands those will hopefully clear out18:19
johnsomOk, cool18:20
*** ysandeep|away is now known as ysandeep18:39
openstackgerritSagi Shnaidman proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible
clarkbya we would add tonyb as a gerrit admin temporarily18:41
fungistuffing my face real quick and then i'll double-check that we've got everything ready for the maintenance18:57
corvusstill t-2h right?19:04
*** ysandeep is now known as ysandeep|away19:06
clarkbcorvus: yup19:06
clarkbinfra-root got updated with auristor's feedback and would be a good piece of documentation to not lose if anyone else has a moment to review that19:17
corvusthat is decidedly out of season19:21
clarkbcorvus: here too. Not completely abnormal for us though19:21
corvusmmm petrichor19:22
clarkbI have vivid memories of the summer I worked as an intern at Garmin because I didn't have a car and it was july and pouring rain19:23
fungiraining here too. luckily i saw the forecast and got caught up on yardwork earlier in the week. nws says to expect 6 solid days of wet19:26
openstackgerritMerged opendev/system-config master: AFS: add note on volume creation servers
fungiso one of the implementation details for our reprepro mirroring is that we add the upstream package index signing keys to the local keyring for the user invoking reprepro... i assume we don't have anything like this i can crib from the ansible roles in system-config yet19:52
clarkbfungi: I don't think so19:54
clarkbwe add things to the apt keyring but not user keyrings19:54
fungithis looks like it will do what we need, but is there a more official ansible stdlib option (other than using the file and command modules i guess)?
clarkbfungi: I think no considering that is open19:56
fungiso do folks think using that third party gpg-import module is preferable to just freestylin' it ourselves?19:58
clarkbfungi: maybe we can make a simple freestyling module in system-config's role set19:58
clarkbthen we don't have to bother with collections and galaxy and all of those changes19:58
fungiwell, we may not even need a module, we could just shell out to gpg --import19:59
fungibasically wondering if doing it with file and command tasks is better/worse20:00
clarkbshelling out is probably best20:01
clarkbit ensures the toolchain is present and functional20:01
fungicool, i can do that. it'll just be a few tasks with a loop and a list of keys anyway20:02
fungi(plus the files containing the key material we want imported of course)20:02
fungiis there a common pattern for feeding files shipped by a role into a command task without explicitly writing the file to the system somewhere first?20:07
clarkbyou can load things like json and yaml into ansible from files as vars20:07
clarkbor put the content as a defaults var directly20:07
clarkbI guess it doesn't have to be a default either can be set as a regular var20:08
fungiyeah, right now we put all of the public .asc files corresponding to the archive signing keys in our repository directly, but we *could* stick them in yaml i suppose20:09
fungii'm trying not to reinvent too much of the current setup, to avoid breaking so much that untangling all the problems i introduce becomes more of a challenge20:10
fungithis change is already upwards of 40 files modified20:10
fungiand a lot of control logic translated from ansible to puppet and erb to j220:11
fungiso it's absolutely going to have a lot wrong with it initially, no doubt20:12
* fungi is beginning to see why nobody has had time to work on this yet20:12
clarkbfungi: if you keep the old puppet in place and don't add the new cron jobs immediately we can probably sanity check everything looks good (even by manually running things) before doing the switch over20:13
clarkbthat doesn't make it easier but reduces the scaryness of large changes20:13
fungihrm, yeah i guess i could do this without file moves/deletes, and hold back the cron additions20:13
fungigood idea20:14
fungianyway, we're at t-0:45 so shifting gears to maintenance checklist20:15
fungishould we go ahead and flip the big red switch to stop deployments?20:15
fungi(and remind me what the file is we have to touch on bridge.o.o to do that)20:16
openstackgerritJames E. Blair proposed opendev/system-config master: WIP: add Zookeeper TLS support
fungii'm going to start recording maintenance activity for this in but also moving to #opendev-meeting and will startmeeting opendev-maint in there20:19
openstackgerritMerged zuul/zuul-jobs master: Add test-build-container-image-release job
-openstackstatus- NOTICE: The Gerrit service on is going offline momentarily at 21:00 UTC for project rename maintenance, but should return within a few minutes:
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Fix upload-docker-image when in release pipeline
openstackgerritJames E. Blair proposed opendev/system-config master: Fix rename playbook after zuul user rename
*** paladox has quit IRC21:28
*** paladox has joined #opendev21:33
openstackgerritMerged openstack/project-config master: Add whitebox-tempest-plugin under QA
openstackgerritMerged openstack/project-config master: Rename interop repos from openstack/ to osf/ namespace
openstackgerritMerged opendev/project-config master: Record 20200612 project renames
openstackgerritJames E. Blair proposed opendev/system-config master: WIP: add Zookeeper TLS support
openstackgerritMerged zuul/zuul-jobs master: Fix upload-docker-image when in release pipeline
-openstackstatus- NOTICE: gerrit is being taken offline for emergency cleanup, will return to service again shortly21:58
*** nautics889 has quit IRC22:01
*** rchurch has joined #opendev22:31
*** mlavalle has quit IRC22:41
openstackgerritJeremy Stanley proposed opendev/system-config master: Correct "ansbile" typos
openstackgerritMonty Taylor proposed opendev/system-config master: Add utility script to disable ansible
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add tests for upload-docker-image
mordredclarkb, fungi: ^^22:42
mordred(the system-config patch from me)22:42
openstackgerritMonty Taylor proposed opendev/system-config master: Add utility script to disable ansible
-openstackstatus- NOTICE: The Gerrit service on is available again22:47
*** tosky has quit IRC23:25
*** DSpider has quit IRC23:25
openstackgerritMonty Taylor proposed opendev/system-config master: Add utility script to disable ansible
fungimirror.ubuntu vos release is still in progress23:46
fungicontinuing to watch it23:46
openstackgerritMerged opendev/system-config master: Correct "ansbile" typos

Generated by 2.17.2 by Marius Gedminas - find it at!