Friday, 2020-03-27

ianwwe could add a role to the base jobs to dump a config file00:02
ianwconfig file /home/zuul/.config/virtualenv/virtualenv.ini missing (change via env var VIRTUALENV_CONFIG_FILE)00:03
clarkbwill virtualenv read a global config?00:03
clarkbah ya that could work00:03
mordredooh. I like the idea of putting things in to a config00:20
openstackgerritIan Wienand proposed opendev/base-jobs master: virtualenv-override : add download for virtualenv seeder
openstackgerritIan Wienand proposed opendev/base-jobs master: virtualenv-override : add download for virtualenv seeder
openstackgerritIan Wienand proposed opendev/base-jobs master: virtualenv-override: add to base pre playbook
ianwclarkb / fungi / mordred : ^ i think that is the series to add to base-test, then i can test it, then add to base00:36
clarkbianw: it failed linting due to a proper error (I noted it inline if you don't want ot look at log files)00:52
ianwclarkb: argh, thanks00:55
openstackgerritIan Wienand proposed opendev/base-jobs master: virtualenv-config : add download for virtualenv seeder
openstackgerritIan Wienand proposed opendev/base-jobs master: virtualenv-config: add to base pre playbook
ianwi did run the role locally, but screwed up the name00:55
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script"
ianwcorvus: i still think the pypi issues are causing havoc, but i went and looked closely and only removed the parts of the unicode change that were related to the download script; that still seems generically useful to test00:58
*** diablo_rojo has quit IRC01:01
*** diablo_rojo has joined #opendev01:02
*** diablo_rojo has quit IRC01:06
openstackgerritGhanshyam Mann proposed openstack/project-config master: Replace python-charm-jobs to py3 job
openstackgerritMerged opendev/base-jobs master: virtualenv-config : add download for virtualenv seeder
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] Add Fedora 31 support and test jobs
ianwAJaeger / frickler : as the next people to maybe see this, for your consideration -> will fix a range of virtualenv problems due to it vendoring a bad version of setuptools (see for links)04:53
ianwi've linked to a test run there too04:53
ianwif anyone turns up talking about script permissions that's likely it04:53
openstackgerritMerged opendev/base-jobs master: virtualenv-config: add to base pre playbook
*** ralonsoh has joined #opendev07:08
*** dpawlik has joined #opendev07:11
*** factor has joined #opendev07:31
*** ykarel has joined #opendev07:42
*** ysandeep|rover has joined #opendev07:43
*** lpetrut has joined #opendev08:06
*** elod has quit IRC08:13
*** rpittau|afk is now known as rpittau08:23
*** ykarel is now known as ykarel|lunch08:24
*** ysandeep|rover is now known as ysandeep|rover|l08:33
*** elod has joined #opendev08:57
*** ysandeep|rover|l is now known as ysandeep|rover08:59
*** ykarel|lunch is now known as ykarel09:00
*** tosky has joined #opendev09:29
openstackgerritJens Harbott (frickler) proposed openstack/project-config master: Add Shrews to alumni
*** roman_g has joined #opendev10:20
openstackgerritThierry Carrez proposed opendev/infra-specs master: POC Xwiki farm at
ttx(rebased on top of the jitsi-meet spec merge)10:29
*** ysandeep|rover is now known as ysandeep|rov|brb10:41
AJaegerinfra-root, cliff and openstacksdk have been tagged for new release - thanks, ttx10:52
*** ysandeep|rov|brb is now known as ysandeep|rover11:03
AJaegerSo, on zuul-jobs: py35 failure is fixed, py27 is failing differently now (stestr 3.0.0 problem?)11:11
openstackgerritMerged opendev/irc-meetings master: Move Large Scale SIG meeting hour
fricklerAJaeger: oh, I think mtreinish discussed that in -qa yesterday. 3.0.0 got published w/o proper python-require markers11:16
fricklerso as a workaround we would have to add an explicit cap11:16
frickleruntil he does the same juggling that sdk just finished11:17
mtreinishI'm waiting for gmann to respond with the results of his test to see if requires-python still works, if not I'll push out a 3.0.1 with the metadata switched to python-requires11:19
mtreinishwe probably should also upgrade pbr in that case too (since it uses requires-python)11:20
*** rpittau is now known as rpittau|bbl11:32
openstackgerritMerged opendev/system-config master: Add IRC bots to OpenInfra Labs channels
*** ysandeep|rover is now known as ysandeep|rov|mtg12:00
openstackgerritMerged opendev/system-config master: Build gerrit 2.14 as well
*** elod_ has joined #opendev12:20
openstackgerritSorin Sbarnea proposed openstack/project-config master: Add opendev/pre-commit-hooks
*** ysandeep|rov|mtg is now known as ysandeep|rover12:37
AJaegerfrickler: let me push a cap for now...12:38
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Cap stestr version
*** hashar has joined #opendev12:44
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Cap stestr version
*** elod has quit IRC12:53
*** elod_ is now known as elod12:53
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: install-kubernetes: fix missing package
*** openstackstatus has quit IRC13:01
*** openstack has joined #opendev13:04
*** ChanServ sets mode: +o openstack13:04
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Cap stestr version
*** rpittau|bbl is now known as rpittau13:10
openstackgerritMerged opendev/system-config master: Add links to dev docs on gitea
*** roman_g has quit IRC13:18
*** hashar_ has joined #opendev13:20
*** hashar has quit IRC13:20
*** hashar_ is now known as hsahar13:26
*** hsahar is now known as hashar13:26
*** ysandeep|rover is now known as ysandeep|away13:39
openstackgerritMerged opendev/system-config master: Remove gitea doc links
openstackgerritMerged openstack/diskimage-builder master: Add python-stow-versions element
*** roman_g has joined #opendev13:54
openstackgerritDonny Davis proposed openstack/project-config master: Open Edge has proved to be stable - bumping to 40
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Remove bashate from test-requirements
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script"
openstackgerritMerged zuul/zuul-jobs master: Cap stestr version
AJaegerinfra-root, system-config and zuul-jobs repos are fixed, changes merge again...14:15
AJaegerUntil the next fire ;)14:15
openstackgerritMonty Taylor proposed opendev/system-config master: Base run jobs needing containers on containers base
openstackgerritMerged openstack/project-config master: Open Edge has proved to be stable - bumping to 40
openstackgerritMerged zuul/zuul-jobs master: Remove bashate from test-requirements
openstackgerritMerged zuul/zuul-jobs master: ensure-tox: use python3 by default
openstackgerritMerged zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script"
*** ykarel is now known as ykarel|afk14:34
*** mlavalle has quit IRC14:43
openstackgerritMerged opendev/system-config master: Add meetpad server
openstackgerritMerged opendev/system-config master: Build jitsi-meet images
openstackgerritMerged opendev/system-config master: jitsi-meet: open etherpad on join
*** mlavalle has joined #opendev15:03
*** Open10K8S has joined #opendev15:05
*** _mlavalle_1 has joined #opendev15:22
*** _mlavalle_1 has quit IRC15:22
*** mlavalle has quit IRC15:25
clarkbre opendev docs links I think the next step in that process is to publish the infra manual without the openstack theme15:30
clarkbwe can probably use a preset sphinx theme unless someone thinks they may be able to opendev a theme (I doubt I'd make much progress on that)15:30
fungialabaster is probably fine for now. it will let us specify a logo image in the sphinx config as a stop-gap15:33
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: install-kubernetes: fix missing package, fix the client.key file path
*** ykarel|afk is now known as ykarel15:36
clarkbcool I'll try that15:37
mnaserinfra-root: i just caught this watching logs on submit-logstash-jobs (which won't be seen in logs cause it happens after swift upload)15:49
mnaserTraceback (most recent call last):\n  File \"/var/lib/zuul/builds/c3565c6c4d91424fa052629cb5021d2e/work/tmp/ansible_submit_log_processor_jobs_payload_6mhcqyrl/\", line 201, in main\n  File15:49
mnaser\"/var/lib/zuul/builds/c3565c6c4d91424fa052629cb5021d2e/work/tmp/ansible_submit_log_processor_jobs_payload_6mhcqyrl/\", line 108, in submitJobs\n  File \"/usr/lib/zuul/ansible/2.9/lib/python3.5/site-packages/gear/\", line 1485, in submitJob\n    conn = self.getConnection()\n  File15:49
mnaser\"/usr/lib/zuul/ansible/2.9/lib/python3.5/site-packages/gear/\", line 1284, in getConnection\n    raise NoConnectedServersError(\"No connected Gearman servers\")\ngear.NoConnectedServersError: No connected Gearman servers\n15:49
mnaserExecutor: ze01.openstack.org15:50
fungiare we seeing the same on any other executors?15:50
corvuswell, there's a growing backlog of logstash jobs15:51
corvusso something's getting there15:51
corvusmaybe this is tenant-specific?15:52
corvusmnaser: was that in the zuul tenant rather than openstack?15:52
mnasercorvus: oh yes, that was a job in zuul (zuul/zuul-jobs change)15:52
mordredI'm seeing a ton of red across jobs that shouldn't have any red in openstack tenant - I'm just now starting to look15:52
mordred"nova-compute: no process found" and similar seem to be common across them15:53
corvusmnaser: i don't know if we expect that to work outside of openstack (i mean, maybe we expect it to work and it accidentally broke?  or maybe we don't expect it to work and we never cared to clean up the error?  i'll try to figure out)15:53
clarkbcorvus: mnaser the gearman server is up but running 'status' against it isn't returning very quickly15:53
clarkbcorvus: it should work outside of openstack thee are no acls on it15:53
corvusclarkb: that's probably not surprising if there's 800k jobs15:53
clarkbcorvus: ya I'm guessing that downstream something broke backing up the pipelines and node gearman has fallen over15:54
clarkbwe'll probably need to restart geard and then look at logstash and elasticsearch15:54
clarkbI can do that15:54
corvusclarkb: recent messages are in logstash15:54
corvusso it's not completely stopped15:54
clarkbthere is a good chances that elasticsearch ran out of disk again15:54
clarkbno disk utilization seems fine15:55
mnasersorry i've opened a can of worms on friday :(15:55
clarkband cluster reports greeness15:55
fungimaybe something started logging massive amounts of data and that's increased the volume beyond our capacity to process it?15:57
clarkb8/20 logstash processees are running. Possible something went through and OOMed the rest. I'll start by ensuring those logstash side processes are running then we can restart geard15:58
clarkbfungi: ya its looking like something filled the pipe ^15:58
mordredhahaha. the devstack errors I'm seeing are my fault - I added that pip check run and now it's throwing me "hacking 1.1.0 has requirement flake8<2.7.0,>=2.6.0, but you have flake8 2.5.5."16:02
mordredbecause that's important for devstack16:02
* mordred moves to requirements16:02
clarkbmnaser: thanks for the report. All logstash pipeline processors should be up and running now16:14
clarkbinfra-root remote: Use alabaster theme16:29
clarkb(gerritbot not respoinding as it doesn't have the new name yet )16:29
clarkbAJaeger: ^ your thoughts much appreciated too :)16:30
*** ykarel is now known as ykarel|away16:31
AJaegerclarkb: I miss the "next page" button - but for now, that's a simple and clean change, so let's do it.16:38
openstackgerritAdam Coldrick proposed opendev/storyboard master: Drop Python 2.7 support
*** dpawlik has quit IRC16:41
openstackgerritJames E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad
corvusclarkb, fungi: ^ could you do a speedy re-review of that?  i think it hit a startup race in the gate16:44
openstackgerritJames E. Blair proposed opendev/system-config master: Correct launch readme link
fungicorvus: sure thing, i was just about to the point where i could take a look at those failures in case you hadn't yet16:46
corvusmeanwhile, i'll spin up the server16:49
clarkbmordred and fungi beat me to it16:49
*** smcginnis has quit IRC16:50
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: install-kubernetes: fix missing package, fix the client.key file path
openstackgerritJames E. Blair proposed opendev/ master: Add meetpad hostnames
openstackgerritJames E. Blair proposed opendev/system-config master: Add meetpad01 to inventory
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: install-kubernetes: fix missing package, fix the client.key file path
clarkbI'm only 10 minutes late and already the changes are approved :)17:13
openstackgerritMerged opendev/ master: Add meetpad hostnames
*** hashar has quit IRC17:17
mnaserso.. about the repo freeze17:20
mordredoh - I think we're ready to move forward with that17:20
mnaser can gladly be a testbed if you want17:21
mordredclarkb: what do you think about manually running latest ansible against review - and then manually running the manage-projects from the container (should still work even with gerrit running directly)17:22
clarkbmordred: not sure I understand that last bit17:22
clarkbyou mean without restarting gerrit on new image it should work?17:23
clarkbthat seems like a good plan17:23
mordredwe don't exec inside the image - we just docker run17:23
corvusand everything is bind mounted and host networking, so should be the same.  ++17:23
mordredso - I think the steps are a) manual run of service-review with disabled removed to run it against review and get config files up to the latest and wrapper scripts and whatnot17:25
mordredthen log in to review, do a docker pull and then run manage-projects and watch the logs for errors17:25
mordred(it should be a no-op run right now)17:26
clarkbwill it not dockerpull in ansible already?17:27
clarkbbut ya that soundsgood17:27
clarkbespecially the noop run first17:27
mordrednot currently - we should probalby add that - right now we're counting on docker-compose to pull for us17:27
mordredbut doing a pull in manage-projects playbook is likely a good idea17:27
fungiassuming the docker run doesn't fetch new project-config, it ought to no-op17:27
mordredI've made a copy of service-review.yaml as service-review-mt.yaml - removed the disabled - and I'm going to run it now17:29
mordredunless y'all want me to screen and run it17:29
fungiup to you17:32
mordredyou know - I'm in screen - if anyone wants to watch17:32
mordredI'm teed up - but will wait a sec in case anyone want  to watch along17:33
fungii'm connected to the currently established root screen session, but i'm guessing that's not yours17:33
fungilooks like it was fiddling with firewall rules17:33
fungioh, or... you're on bridge17:33
funginot review.o.o17:33
mordredroot@bridge:/opt/system-config# ansible-playbook playbooks/service-review-mt.yaml17:33
mordredshould be the last thing there17:34
mordredlet me _also_ get in a screen on review for the next bit17:34
fungiokay, found it17:34
fungii also closed out the root screen session on review.o.o since it seemed to have been abandoned there for smoe time17:34
fungisome time17:34
mordredI've made a new one on review :)17:36
mordredk. I will now run the ansible on bridge17:36
fungiconnected to both now17:37
corvusmordred: will the ansible run on bridge run manage-projects because of recent config changes?17:37
mordredcorvus: it shouldn't17:38
* clarkb finds root screen17:38
corvuswhy not? because there were no changes or it's not configured to?17:38
mordredcorvus: manage-projects is not in the gerrit role - it's a separate playbook17:38
mordredalso - we haven't landed that change yet17:38
corvusk.  sounds like it's been thought through.  :)17:38
mordredI'm going to go to review.o.o root screen and run manage-projects17:39
mordreddoing the docker-compose pull first17:39
fungiright, didn't want to incorporate it until it saw a dry-run under close observation17:40
mordredthat's missing --host=net17:40
mordredbother - those changes are in teh run-manage-projects change17:40
mordredsince review is still in the emergency file - that should be safe to land17:41
mordredand then we can try this again17:41
clarkblooking at that list of mounts where is the cache?17:41
*** dpawlik has joined #opendev17:41
mordred/opt/lib/jeepyb I think17:41
clarkbya ls'ing there looks like that is it17:42
clarkbmordred: I've approved that change17:43
fungiheh, double-tap17:43
clarkbmordred: note, I think we want to only run manage-projects when the file changes? or at least that was the old behavior17:43
clarkbwith the caching its probably fast enough to run often though17:43
mordredclarkb: yeah - let's implement that before we take it out of emergency17:44
mordredof course, this would be a great candidate to run from zuul in post on project-config changes eventually17:45
*** diablo_rojo has joined #opendev17:46
openstackgerritMerged opendev/system-config master: Use our jitsi-meet image for meetpad
fungii think we ran it when gerrit/projects.yaml or anything under gerrit/acls changed, right?17:47
mordredyeah - but probably safe to run when anything in project-config changes?17:48
clarkbor anything in project-config/gerrit17:49
fungiprobably safe, but maybe wasteful if it takes a long time to comlpete17:49
fungiyeah, anything under the gerrit tree is a good compromise17:49
fungithe only other thing in there is the notify_impact.yaml file right now17:49
mordredwell - I might have to re-add the puppet logic of copying files from /opt/project-config to /home/gerrit217:49
mordredbecause right now it's just bind-mounting in /opt/project-config17:49
mordredbut - you know - that's not that horrible :)17:50
clarkbI wasn't around last friday, is the version of project-config on review.o.o up to date with the two project renames?17:50
openstackgerritMerged opendev/system-config master: Add meetpad01 to inventory
clarkbwe'll need it to be in order to properly noop17:50
mordredyeah - it should be. let's check17:52
*** dpawlik has quit IRC17:52
fungii thought it wasn't because of the emergency file entry17:56
fungilooks up to date in /opt though17:56
mordredyeah - cause I just re-ran the service playbook which pulled the latest (because of notify-impact.yaml)17:57
fungithat 'splains it17:58
*** smcginnis has joined #opendev18:15
noonedeadpunkhey everyone. Do you have any info when diskimage-builder is going to be tagged? As we're kinda waiting for opportinuty to make use of recently merged python-stow-versions element:)18:22
clarkbnoonedeadpunk: ianw typically makes the dib releases and it is a weekend right now in australia. Maybe ping ianw and see if a release can be made on monday australia time?18:26
noonedeadpunkclarkb: ok thanks! I guess he has just been pinged :p18:28
*** nmickus has joined #opendev18:30
rm_workhey, wanted to make a job from a template non-voting, but looks like "voting: false" doesn't work in the templates section -- what is the strategy to resolve this? can we provide the job name directly in a jobs section and have it pull the right job from the template automatically or something?18:32
*** lpetrut has quit IRC18:33
clarkbrm_work: you need to remove the template and list the jobs normally18:33
rm_workok so yeah, that does work?18:33
clarkbya if you remove the template18:33
rm_workjust take the job name and list it in the jobs section instead of using the template?18:33
rm_workdo we lose anything by removing the template?18:33
clarkbrm_work: all the jobs in the template. You can list them directly though18:33
noonedeadpunkclarkb: I guess it's possible to leave template but just metnion single job as non-voting among jobs?18:33
rm_workI THINK in this case it's only one?18:34
rm_workyeah I wondered if overriding just that job was an option18:34
noonedeadpunkand that's will override behaviour of template?18:34
openstackgerritMerged opendev/system-config master: Run manage-projects on gerrit servers
openstackgerritMerged opendev/puppet-openstackid master: added www. cname on vhost file
clarkbnoonedeadpunk: I want to say that doesn't work (people have tried ti and been confused with the result)18:34
clarkbrm_work: you can also update the template globally fi the chagne should be global18:34
rm_workah hmmmm18:35
rm_workwe may own that template actually, now that i think about it, so possibly an option lol18:35
rm_workthough I think it'd be preferrable to do just an override18:35
noonedeadpunkclarkb: ah, yes, we were disabling job inside tempalte I guess..18:36
noonedeadpunkclarkb: but actually no18:37
noonedeadpunkthat worked for us18:37
noonedeadpunkand openstack-ansible-deploy-aio_distro_metal_tempestconf-centos-7 job is part of openstack-ansible-deploy-aio_metal-jobs tempalte18:37
clarkbnoonedeadpunk: got it18:38
noonedeadpunk(I meant ubuntu-bionic job not tempestconf-centos-7)18:39
fungiyeah, it's worth trying to do that with a variant, as long as the job in the template isn't marked final18:44
clarkbfungi: maybe that was why I remembered it not working18:44
fungi(that may even work with final jobs, can't remember if that blocks variants or just parenting)18:44
clarkbmordred: the change langed I think that means we should've promoted a new image?18:48
clarkboh no we needed new ansible in this case right?18:48
clarkbI'm about to find lunch but can help more after18:48
mordredclarkb: yeah - it's new ansible we need18:52
mordredclarkb: so I'll wait for the next pulse to update us18:53
mordredso that I'm not fighting it18:53
*** rpittau is now known as rpittau|afk18:57
*** roman_g has quit IRC19:03
*** ralonsoh has quit IRC19:16
mordredinfra-root: ok. system-config is updaed - I'm going to run the playbook again19:19
AJaegermordred: can I remove my WIP from the new repo requests?19:25
mordredfungi: ok. that went well - I think it's time to switch to review and try manage-projects19:25
mordredAJaeger: almost19:25
AJaeger(those still need another core to review)19:25
openstackgerritRonelle Landy proposed zuul/zuul-jobs master: Add var to allow jobs to skip centos8 deps install
mordredfungi: k. that all looks ok - I'm going to run manage-projects19:26
fungilooks like it's doing its thing19:27
fungivery verbosely19:27
mordredwow. that's doing a lot of things19:27
mordredit wasn't happy about opendev/infra-manual19:27
*** nmickus has left #opendev19:28
mordredfungi: hrm19:29
fungiso i think it's having trouble fetching the meta/config ref maybe?19:30
mordredoh - look - it tried to create a new empty repo19:31
*** diablo_rojo has quit IRC19:31
mordreddid we miss renaming the repo in /opt/lilb/jeepyb?19:31
clarkbyou shouldnt need to update the cache iirc19:31
clarkbit should see the project is already there and populate the cache from gerrit automatically19:32
clarkb(I'm still eating my curry)19:33
*** diablo_rojo has joined #opendev19:33
mordredI bet you anything it's another utf-8 thing19:33
mordred        if output.strip() != "project.config" or status != 0:19:33
mordredok. that was silly, I have started a new screen19:35
openstackgerritMerged zuul/zuul-jobs master: install-kubernetes: fix missing package, fix the client.key file path
mordredhrm. why isn't hat working ...19:36
mordredoh. it did19:36
mordredyup. it's returning bytes19:37
openstackgerritMonty Taylor proposed opendev/jeepyb master: Return strings from run_command
mordredok. that patch ^^ fixes it19:42
clarkbmordred: maybe double chwck we dont have similiar problems where we interface with gerritlib?19:42
clarkbthere was semi recent changes ti gerritlib to deal with unicode and python319:43
clarkbI cant remember if we went all bytes or all strings19:43
mordredclarkb: we seemed to find infra-manual in the list that time - so the acls work at least worked19:43
mordredbut looking19:43
clarkbI can help in afew too. finishing up lunch19:43
*** ysandeep|away has quit IRC19:44
*** ysandeep has joined #opendev19:45
mordred <-- there's the log from the most recent run19:45
*** elod has quit IRC19:46
*** elod has joined #opendev19:46
mordredclarkb: fwiw - I edited jeepyb in place with the above patch inside of a manage-projects container and it fixed that instance19:46
*** diablo_rojo has quit IRC19:51
fungiUnable to freeze job graph: Job system-config-build-image-python-builder not defined19:55
fungithat's blocking the jeepyb fix from getting tested19:56
clarkbit needs a -37 at the end19:56
clarkbor 3.7?19:56
fungidid we wedge that from another tenant?19:56
clarkbfungi: it got converted in opendev/system-config and not updated on this side I think19:57
clarkbmordred: I've double checked gerritlib and the change that was made was to go to unicode which explains the mismatch I think19:57
clarkbmordred: so ya your fix is proper19:57
fungioh, yep, it's fixed in opendev/system-config but not yet in openstack/project-config19:58
clarkbfungi: I think we might need to update that job in project-config19:58
fungiis the fix already proposed?19:58
fungii can if not19:58
clarkbnot that I know of19:58
fungion it19:58
fungido we want -3.7 or -3.8? seems we have both defined19:59
clarkb3.7 in this case I htink20:00
clarkbits our default still, but we have 3.8 to transition to eventually20:00
mordredso the 3.8 image is there to allow us to do that on a consumer-by-consumer basis20:00
openstackgerritJeremy Stanley proposed openstack/project-config master: Update Python image builder job name
fungisee if that seems right20:02
mordredlooks perfect- thanks!20:02
*** diablo_rojo has joined #opendev20:07
openstackgerritMerged openstack/project-config master: Update Python image builder job name
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add vexxhost/smokeping_prober-helm
mordredI have rechecked the jeepyb change20:25
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add vexxhost/smokeping_prober-helm
openstackgerritMonty Taylor proposed opendev/gerritlib master: Run gerritlib integration tests with python3
mordredclarkb: ^^20:34
clarkb+2 do we want to get results on that before attempting production rn again?20:35
mordredwell - I bet we'll get results on it before the images are finished being built20:36
mordredand - I *bet* it'll fail20:36
mordredsince it should need the jeepyb change to work for manage-projects to work20:36
mordredbut let's see :)20:36
mordredif it does, that'll be a great indication of it being a good test20:36
mordredclarkb: oh - did you see the results from the don't-install-test-requirements in #openstack-qa ?20:37
clarkbI did not20:37
mordredtl;dr - it worked - and actually fixed some broken nv jobs20:38
mordredsince they were failing on issues installing things from test-reuqirements :)20:38
mordredthings like something something nose20:38
mordredanywa - thought you'd enjoy that :)20:39
fungiresults are in20:46
fungigerritlib-jeepyb-integration FAILURE in 7m 52s20:46
clarkbsame fetch config exception20:48
clarkbthats good and consistent :)20:48
mordredlet me depends-on it20:49
openstackgerritMonty Taylor proposed opendev/gerritlib master: Run gerritlib integration tests with python3
mordredclarkb: if we don't watch out - we're going to end up with an actually tested system :)20:51
openstackgerritMonty Taylor proposed opendev/system-config master: Add an image pull to manage-projects playbook
openstackgerritMonty Taylor proposed opendev/jeepyb master: Fix fsck_command for python3
openstackgerritMonty Taylor proposed opendev/gerritlib master: Run gerritlib integration tests with python3
mordredfound another issue - there's the jeepyb patch and I updated the gerritlib dependson21:01
fungigetting sidelines by dinner but will try to resume reviewing once i eat, if there's still stuff in progress for this21:03
mordredit's almost certain there will be21:03
clarkbyay testing21:03
mordredclarkb: I'm glad you made that integration test - although I'm a little sad it's in gerritlib and not in jeepyb :)21:04
clarkbmordred: ya it was largely a "put this somehwere" thing21:04
clarkbmordred: maybe we should make a copy with a new name in jeepyb then delete the old one in gerritlib21:04
mordredif you'd put it in jeepyb I'd be saying "I wish you'd put it in gerritlib"21:04
mordrednah - by the time we're done with this it'll be fine and we'll never touch it again21:05
clarkbmordred: more bugs
clarkbare you already looking at that or should I?21:12
mordredclarkb: oh fun21:13
mordredwat? TypeError: 'filter' object is not subscriptable21:14
clarkbmordred: ya we filter in gerritlib21:15
clarkband I'm guessing that returns an iterable not a list in python321:15
clarkbso we can either do list(filter()) in gerritlib or list() the return from gerritlib21:15
clarkbchanging gerritlib will requires a release21:15
clarkband ya filter() returns an iterator21:15
clarkbmaybe we should do both things? and then we can remove the jeepyb fix later when gerritlib is released?21:16
openstackgerritMonty Taylor proposed opendev/jeepyb master: More python3 fixes
mordredclarkb: oh - sorry - I didn't see that - I stuck in a next21:17
mordredI think we shoudl do the list() wrap21:18
fungiis there a common topic for these? i've lost track of the outstanding patches21:18
openstackgerritClark Boylan proposed opendev/gerritlib master: Return lists from listing functions
mordrednext doesnt' work on a list21:18
clarkbmordred: ^ there is the list() wrap on the gerritlib side21:18
mordredlet me remove the jeepyb next()21:18
clarkbmordred: maybe do it in jeepyb too for now?21:18
clarkbmordred: we consume it from releases so will want to make a release21:18
clarkbit being gerritlib21:18
clarkbfungi: I didn't use one21:18
clarkbmordred: is there a topic I should use?21:18
openstackgerritMonty Taylor proposed opendev/jeepyb master: More python3 fixes
mordredclarkb: gerrit-docker maybe?21:19
clarkbmordred: there are several other places in jeepyb where we call gerrit\.list.*21:19
mordredwhy did we get unable to freeze job graph?21:19
clarkbmordred: I think you want to do the same update on those21:20
openstackgerritMonty Taylor proposed opendev/jeepyb master: More python3 fixes
clarkbfungi: topic:gerrit-docker set21:20
openstackgerritMonty Taylor proposed openstack/project-config master: Use versioned jobs in the gate too
mordredclarkb, fungi: ^^ we missed the gate jobs before. doh21:21
mordredI'm going to squash the jeepyb patches now since we have to run it again anyway21:22
clarkbmordred: k21:22
openstackgerritMonty Taylor proposed opendev/jeepyb master: More python3 fixes
openstackgerritMonty Taylor proposed opendev/gerritlib master: Run gerritlib integration tests with python3
mordredok. squashed, abandoned the other and redid the depends-on21:24
mordredlet's see how far we get this time :)21:24
mordredclarkb: oh - did you see noonedeadpunk asking about a dib release earlier?21:25
mordredwe've got the new stow-python element in there21:25
clarkbmordred: I did, ianw typically manages those so asked that we ping ianw for a inw time zone monday release21:25
clarkbkids preschool is going to try distance learning via zoom21:27
clarkbI wish the tech world would conglomerate around open tools a bit more so that by the time they trickle down into schools we don't end up using such things21:27
clarkbmaybe I can convince them to use meetpad (though etherpad probably not super use ful for them) and they can be our load testers >_>21:28
mordredwaiting for tempest-slow-py3 is like watching paint dry21:29
mordredclarkb: gerritlib integration passed!21:30
clarkbI think everything I didn't write has my +2, fungi if you have time to review looks like some stuff is happy now21:31
mordredclarkb: oh - also this
openstackgerritMonty Taylor proposed opendev/system-config master: Base run jobs needing containers on containers base
clarkbmordred: check comment on that one, I think its fine but someting to consider21:33
openstackgerritMonty Taylor proposed opendev/system-config master: Remove unused gerrit puppet things
mordredclarkb: yeah - I *think* it should still work liek if we update that image name in a host_var21:35
clarkbmordred: ya but not if at role include time for main role include?21:35
mordredbut I agree - for now we should almost certainly be fine21:35
clarkbits probably fine in most cases21:35
openstackgerritMonty Taylor proposed openstack/project-config master: Add openstacksdk and python-openstackclient to integrated
mordredclarkb: I'm torn on that ^^21:50
mordredit's maybe the wrong idea21:50
mordredbut - liek right now I'm having to wait for a devstack change to land so I can click +A because I can't depends-on and +A21:51
clarkbhrm a depends on would enforce order but not do the fancy testing I guess21:51
clarkbbeing able to test things together like that seems desireable21:51
mordredwell - it's that it won't enqueue it into the gate21:51
mordredtesting thinngs together totally works21:52
mordredbut only at check21:52
clarkbya then you hvae to wait for things to merge on the other side first21:52
mordredfor gate - brcause devstack is in a shared queue, a depends-on from openstacksdk won't enqueue openstacksdk into the gate21:52
mordredso - today that patch seems liek a good idea21:52
mordredbut - maybe most days it's a bad idea21:52
mordredbecause it is fairly rare that openstacksdk depends on one of the other projects21:53
clarkbother way around though21:53
mordredyeah. it might just be the correct thing to do regardless of whether it might be annoying sometimes21:53
openstackgerritJeremy Stanley proposed opendev/puppet-openstackid master: Add SSL options in www subdomain redirect vhost
mordredfungi, corvus: is ready to go is tested with
fungiokay, i think i'm caught up on other fires and questions... looking now to see what needs reviewing for the gerrit stuff22:02
mordredactually - should go first22:02
mordredfungi: so - actually - hold off on the others - the gate won't work for jeepyb until that's in22:03
mordredwe fixed check earlier and not gate :)22:03
fungidid i not get all the occurrences of that job name? i thought i had searched for them22:03
mordredI blame trolls22:04
fungii fixed the name of the build job, that's fixing the name of the upload job22:06
funginow i get it22:06
fungiwe stopped at the first error, so it didn't complain about the other job name22:06
mordredfun right?22:07
fungiand we can recheck 715527 once 715556 merges22:07
fungi(since it's in a config repo)22:07
openstackgerritMerged openstack/project-config master: Use versioned jobs in the gate too
clarkbI rechecked 71552722:17
mordred is ready for +A22:17
fungias did mordred22:17
mordredme goes for bikeride - hopefully patches will be landed by the time I'm back and we can try this all again :)22:19
clarkbenjoy. I'm not bike riding today because it is cold and wet and miserable. And indoor bike time is just not the same22:20
corvusanyone care to try a meetpad?
openstackgerritMerged opendev/puppet-openstackid master: Add SSL options in www subdomain redirect vhost
clarkbI typically jitsi from my phone, but I can get headset set up and do it on real computer22:38
clarkbgive me a couple minutes to climb under desk and plug stuff in22:38
openstackgerritJames E. Blair proposed opendev/system-config master: Correct jitsi-meet config file
clarkbI can hear you corvus22:42
clarkbcan you hear me? that may just be local mic problems if not22:42
clarkbalso i get warnings about etherpad cookies not being enabled and it really wnting cookies22:42
corvusi can not hear you22:43
corvusclarkb: i have gotten the cookies thing sometimes, and sometimes not22:43
clarkbok let me sort out my mic22:43
corvussounds like it's not just a local problem and it may need some debugging...22:43
clarkbcorvus: ya chrome can't get my mic :)22:46
clarkbI got the cookie thing again fwiw in chrome22:46
corvuswell bummer22:46
clarkbthe url thing in chrome says it can access my mic but then jitis itself seems confused22:47
clarkbI'm going to see if I can determine which device it thinks it is using22:47
corvusi think it's set up to proxy etherpad through the nginx container.  that seems a little weird to me, and potentially a source of problems, but maybe it's necessary.22:47
clarkbthe little pop up says it can't use it for "unknown reason"22:47
clarkbwas the feedback me?22:48
corvusnope, me, sorry22:48
clarkbcorvus: WaG87Fvi2lqlcSFuzHQg22:57
openstackgerritMerged opendev/system-config master: Base run jobs needing containers on containers base
clarkbcorvus: /opt/etherpad-lite/etherpad-lite/src/static/js/pad_editbar.js23:03
clarkbcorvus: in toggleDropdowns23:03
johnsomI have an oddball question about gerrit/zuul. If I wanted to setup another category similar to a third party CI, but the job is in other OpenStack repositories (maybe x/ repos) how would I go about doing that?23:10
johnsomWe have third party drivers that would like to run non-voting jobs, live vmware, etc. that for example are under x/vmware-nsx. We would like to distinguish that these are not supported/managed by the octavia team.23:12
johnsomWould just need to setup a "third party" pipeline?23:13
clarkbjohnsom: isn't that third party ci?23:13
clarkb(maybe I've latched onto the vmware testing which I don't think we can do upstream too much)23:14
johnsomThe difference is it's already an opendev job, so the trigger would be our gerrit as opposed to an external system listening and sending results back.23:14
fungijobs run in our ci system, non-voting but also somehow visually separate from "official" non-voting jobs23:14
johnsomBoth cases are current/former neutron sub-projects23:15
johnsomThey implement drivers for Octavia (among other things there)23:16
mordredclarkb, corvus: I cannot see or hear anything23:16
fungitechnically it would be possible to have a separate pipeline triggered like check but which didn't vote... or which voted on a new, different gerrit review label23:16
mordredin case there is anything to see or hear23:16
clarkbmordred: your browser should ask you for permission to use your mic and or camera23:17
mordredyup. it did - and I granted it - and my camera light is illuminated23:17
clarkbmordred: if you don't allow it then it has a sad, if it didn't do that click in the icon area left of url bar and see if you can tell it to do the things that way23:17
johnsomfungi Yeah, that sounds like what we would like. Basically an opendev third party. lol23:17
clarkbfungi: ya I think thats the trick is it can't vote or we'd need a separte account23:18
corvusjohnsom, fungi: it's possible, but to really make it work we need another gerrit account, so it's quite a bit of work.23:18
corvuswe abandoned an earlier similar effort because it didn't seem worth it at the time23:18
fungijohnsom: any reason why just having them be normal non-voting jobs is a problem?23:18
johnsomI am fine with "can't vote" for now. I don't see that happening anytime in the near future.23:18
clarkbmordred: we can see you fwiw23:18
mordredI can see myself in the settings popup23:19
fungilots of projects have non-voting jobs which they consider purely advisory23:19
mordredbut I can't see myself in my little window on the main page23:19
mordredthere it is23:19
johnsomfungi Yeah, the concern is it doesn't distinguish them as third party, so doesn't give the clue that these aren't project team supported drivers.23:20
johnsomSince we don't own the job definition, we can't distinguish them using the name either.23:20
fungicorvus: i wasn't thinking as far as reporting under a separate account, we already isolate reports from separate pipelines in different comments (and group them in our javascript overlay, though no idea how the pipeline grouping translates to the checks api in 3.x)23:21
fungibut i agree it's still additional work to maintain23:21
fungialso no idea if that's visually distinct enough to address johnsom's concerns23:22
clarkbfungi: ya it could vote in a verified2 category23:27
clarkbor if we want it to vote in verified need a second account23:28
fungiit could just not leave a +1 or -1 on any label and only report success or failure, right?23:28
clarkbI think that will clear our existing votes?23:29
mordredclarkb, corvus:
fungioh, if it doesn't vote on a label that resets the vote, right23:30
fungiand if it did vote on a different label, that would need to somehow preserve any existing vote on the verified label23:30
fungiso yeah, i don't think it's actually an option after all23:31
fungizuul would need some feature to know which labels needed to remain untouched when reporting from a particular pipeline and then reinclude the prior values if they're present23:32
clarkbfungi: I think it may only set values for the category it is told to write to23:33
clarkbso it may already ignore other pipelines23:33
fungiother labels you mean?23:33
fungihrm, would need testing23:33
johnsomI am assuming this isn't testing I could do in our repo on a patch. Like I can't just define my own pipeline right?23:35
clarkbjohnsom: correct, pipelines are "trusted" zuul config so have to go into a trusted repo23:36
clarkbjohnsom: is the concern that you won't be able to turn those jobs off if they break? because if you define them in your repos on your project then you have full control of that23:36
johnsomRight, there are a couple of reasons behind the idea. One is to help clarify project ownership for the job. One is to make sure we can disable it should it become a problem.23:38
corvuseven if they're not defined in your repo, you're still attaching them to your check pipeline in your repo, right?23:39
johnsomWe already have a lot of "it has octavia in the name, so this is the team that owns that." kind of issues.23:39
clarkbjohnsom: isn't that strongly determined by the repo the job lives in?23:40
johnsomNo, this is a request to not included them in our check pipeline. We could do that today. This is about having a separate pipeline like third party ci23:40
clarkb(not where it is used, but where it is defined)23:40
clarkbjohnsom: right and we are saying its difficult23:40
corvusclarkb, mordred: i'm going to manually stick a cors header in etherpad okay?23:40
clarkband not possible today23:40
clarkbcorvus: ok23:40
corvusjohnsom: i'm just trying to understand -- i think you said that today you can't turn off the jobs, but i don't understand why?  aren't they attached to your project's check pipeline in your repo?  isn't it a matter of deleting one line?23:41
clarkbjohnsom: because for zuul to report to gerrit it has to report a vote value, and that would clear out existing values if it say voted +0 after check voted +123:41
johnsomRight, but there is no indication that it's owned by another team, where a separate visual/labeled section would help with that.23:41
openstackgerritMerged opendev/jeepyb master: More python3 fixes
fungii mean, technically the openstack-tox-py37 job is owned by another team too, right?23:42
mordredZOMG the jeepyb patch landed23:43
johnsomYes, but we can and are on the hook to support / fix that. For third party CI jobs we are not.23:43
johnsomNor do we want to be.23:43
fungijohnsom: it should be possible to "rename" the jobs when you include them in your project-pipeline by just creating a child job parented to it23:43
clarkbjohnsom: if I'm parsing correctly I think what you mean by support is "when this job stops successfully running reliably we fix it"23:43
clarkband today that list is all of your check jobs23:43
johnsomclarkb Correct23:44
corvusfungi: yep, should be a one-liner inheritance.  could name it "unsupported-third-party-plugin-test-thingy"23:44
corvusetherpad restarted23:44
johnsomHmm, that is actually not a half bad idea. It doesn't group them like the other third party CI jobs would, but would make it clear.23:45
mordredalso - zuul doesn't vote until all of the jobs in a pipeline are done. so I could also imagine wanting to say "jobs in check vs jobs in shrug" and the shrug jobs have no vote but have final status so you can see them - but no activity is blocked waiting on them - right?23:45
corvusjohnsom: jobs in pipelines are ordered, so you can at least sort them together within that one group23:45
corvusmordred: yes, that's the other trade-off23:45
mordredcorvus: would the existence of checks plugin in the future allow a second pipeline in opendev zuul that only left checks and didnt' leave a vote?23:45
corvusmordred: it's more or less the same issue, pipelines vote or don't23:46
johnsomRight, it would be nice to go on with normal life and if these go into endless loops (Like the OVN driver did) it wouldn't stop our progress.23:46
fungibonus for renaming them with child jobs: you can add an extra couple of lines to set your preferred timeouts23:47
johnsomlol, 10 minutes23:48
*** ysandeep is now known as ysandeep|rover23:48
clarkbcorvus: I think I figured out why the monitor is the default now too. its the always on virtual device then when I plug and unplug things I don't have to change settings23:50
clarkbexcept it is muted by default so useless by default :)23:50
clarkb(though from a privacy perspective that is a good default, justh ave to be aware of it23:50
johnsomSo, this is probably workable for us. Doing the parent/child thing. People might get worked up, but tough. However, I think the second pipeline might be the *right* longer term solution. Is there an RFE I can write up to track that?23:51
mordredclarkb, corvus, fungi I23:51
mordredclarkb, corvus, fungi I'm going to try manage-projects again23:52
clarkbjohnsom: if zuul needs updates for that (I'm not sure) then a spec there may make sense?23:52
clarkbif zuul is good to go (probably via second gerrit connection) then an infra-spec to createa a second gerrit connection with a second user would work23:52
johnsomI think designate might end up in the same situation with all of the alternate drivers if they decide to move them out of repo as well.23:52
clarkbpersonally I think that you shouldn't run those jobs if they don't provide value23:53
clarkband if they do provide value then fixing them when they break is in scope23:53
mordredclarkb, fungi, corvus: I'm in a screen on review.o.o if anybody wants to watch along23:53
clarkbmordred: I see the command prompt23:53
mordredI have already tailed the log file in the background, fwiw23:54
clarkbjohnsom: and we do a lot of cross project testing in here. Some of it fails (ansible-devel) some of it doesnt23:54
mordrednothing failed23:54
johnsomI think the other teams just don't want to use periodic jobs to test their driver against the tip of octavia.23:54
clarkbmordred: looks like it did a lot of nothing as expected?23:55
mordredclarkb: I think that means we're good to go on re-enabling manage-projects23:55
clarkbmordred: k, I'm starting to enter the time of day where I'm not in a good spot to help babysit as we're cooking potatos and those need a headstart23:55
mordredclarkb, johnsom: I think there may be a general use case here which I've thought about a few times but don't have actual thoughts on what it means yet - which is the capability for a project to "subscribe" to another project - without necessarily reporting back to them23:55
johnsomReally I guess the value is no different than the third party CI, it's just in this case they are semi-third party.23:56
clarkbjohnsom: they can test the other direction, but I guess if their change volume is low they have the problem where they don't catch it until too late23:56
clarkbmordred: ya that is kind of what silent is/was23:56
johnsomRight, our velocity is much higher than theirs23:57
mordredso - like - it's important for octavia-vmare that vmware jobs run on octavia patches, but not to the octavia team - so if the octavia-vmware team could say "we want to run these jobs to verify our driver - and have them report to XXX" - that is a pattern that isn't really possible today but is a relationship in the human world that we have with various folks23:57
clarkbmordred: we ran the job on ever change but you had to go look for them23:57
mordredyeah - but you still even have to go ask the octavia team for permission to run the job - or the opendev team23:57
clarkbextending that so that results end up where you want them and aren't completely silent would be a good thing23:57
clarkbmordred: oh thats a good poin too23:57
mordredyou can't say, in the octavia-vmware repo, I want to run this job on this input - partially because there's no good place to report23:57
mordredbut if you treated it LIKE a periodic - only reported to the dashboard23:57
mordredit could still be a nice thing to allow people to do to track things where the care is unidirectional23:58
mordred*massive hand wave here*23:58
clarkbI think silent would actually get you halfway there23:58
clarkbjohnsom: with the silent pipeline we'd report to zuul's db on every change but not to gerrit23:58
mordredclarkb, fungi, corvus: so - I think I'm going to remove review from emergency23:59
clarkbjohnsom: then the driver folks could query zuul for those results. This is similar to periodic except it runs on every change23:59
clarkbmordred: ok, you'll keep an eye on it?23:59
clarkbI guess mostly just to see that the first pulse is happy as it was manually23:59
clarkb(and all the associated ansible stuff)23:59
mordredyeah. or - actually - I'm about to dinner - so I'll remove it from emergency next time I'm ready to watch it :)23:59
fungimordred: yay! looks good23:59

Generated by 2.15.3 by Marius Gedminas - find it at!