| *** mlavalle has quit IRC | 00:03 | |
| clarkb | looks like mnaser is trying to work up a fix overin #zuul | 00:04 |
|---|---|---|
| clarkb | (it was broken in zuul-jobs looks like) | 00:04 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: Fix incorrect variable name for promote-docker-image https://review.opendev.org/724442 | 00:05 |
| mordred | clarkb: yes - i do think we should also cache the static resources | 00:05 |
| mnaser | and also on that note ^ | 00:05 |
| mordred | clarkb: they're all designed to be aggressively cached - new updates to the bundles will cause a new sha in the filename | 00:05 |
| * mnaser goes to nom | 00:06 | |
| clarkb | mordred: ok let me work up that change now | 00:06 |
| *** diablo_rojo has quit IRC | 00:08 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cache static zuul resources in apache https://review.opendev.org/724444 | 00:13 |
| clarkb | mordred: ^ | 00:13 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix incorrect variable name for promote-docker-image https://review.opendev.org/724442 | 00:15 |
| *** sean-k-mooney has quit IRC | 00:31 | |
| kevinz | fungi:clarkb:hrw: ianw: As discussed, I've enlarged the Cinder Quota to 2000gb now for os-control. | 01:05 |
| ianw | kevinz: thanks! i can add a volume for afs cache, unless fungi wants too | 01:06 |
| kevinz | iaw: :-) Please let me know if it is fine to use. | 01:07 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: infra-package-needs: drop ntp for Focal https://review.opendev.org/724450 | 01:10 |
| ianw | kevinz: how's the devstack arm64 work going? last run i saw i thought may have been related to mirror errors | 01:18 |
| kevinz | ianw: let me check | 01:32 |
| kevinz | ianw: this time the devstack cluster has setup and it is running the tempest | 01:34 |
| kevinz | ianw: Thanks! | 01:34 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: [wip] boot test of project-config elements https://review.opendev.org/724453 | 01:59 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: [wip] boot test of project-config elements https://review.opendev.org/724453 | 02:06 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: [wip] boot test of project-config elements https://review.opendev.org/724453 | 02:08 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: [wip] boot test of project-config elements https://review.opendev.org/724453 | 02:24 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: [wip] boot test of project-config elements https://review.opendev.org/724453 | 02:46 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Boot test of project-config elements https://review.opendev.org/724453 | 02:50 |
| ianw | xecuting: /tmp/apt-key-gpghome.amtrfMiWFc/gpg.1.sh --keyserver keyserver.ubuntu.com --recv 2B5DE24F0EC9F98BD2F85CA315B6CE7C018D05F5 | 02:57 |
| ianw | ^ this is a pretty common cause of image build timeouts | 02:57 |
| ianw | i wonder if there's a better way to cache that | 02:57 |
| ianw | kevinz: it looks like the cirros boot failed? https://3536015bea9563600a7f-6b9ee2b8a33ebe649008113f3dcc3069.ssl.cf5.rackcdn.com/708317/43/check-arm64/devstack-platform-arm64/61abaf9/testr_results.html | 03:08 |
| clarkb | ianw: I thinj we can just copy the public key and tben addit to the trust chain for apt | 03:08 |
| ianw | error: you need to load the kernel first. Press any key to continue... Failed to boot both default and fallback entries. | 03:09 |
| ianw | hrw: ^ long shot but maybe that triggers a "i know what that is" moment? :) | 03:10 |
| *** ykarel|away is now known as ykarel | 04:15 | |
| *** moppy has quit IRC | 05:01 | |
| *** mrunge has joined #opendev | 05:27 | |
| *** ysandeep|away is now known as ysandeep | 05:42 | |
| *** DSpider has joined #opendev | 06:55 | |
| *** iurygregory has joined #opendev | 06:58 | |
| *** ralonsoh has joined #opendev | 07:27 | |
| *** tosky has joined #opendev | 07:28 | |
| *** rpittau|afk is now known as rpittau | 07:29 | |
| AJaeger | config-core, please review https://review.opendev.org/720160 and https://review.opendev.org/#/c/723640 | 07:48 |
| *** ysandeep is now known as ysandeep|lunch | 08:08 | |
| openstackgerrit | Merged openstack/project-config master: Retire airship-in-a-bottle https://review.opendev.org/720160 | 08:09 |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Add mirroring for SIG and other misc repos https://review.opendev.org/724607 | 08:27 |
| hrw | ianw: looks like broken image | 08:28 |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Add mirroring for SIG and other misc repos https://review.opendev.org/724607 | 08:36 |
| *** ykarel is now known as ykarel|lunch | 08:39 | |
| zbr | ianw: can you please abandon https://review.opendev.org/#/c/385217/ | 08:40 |
| zbr | i wanted to do some cleanup, but i have no power to abandon them | 08:41 |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Add mirroring for SIG and other misc repos https://review.opendev.org/724607 | 08:48 |
| *** Eighth_Doctor has quit IRC | 08:56 | |
| *** ysandeep|lunch is now known as ysandeep | 09:04 | |
| *** Eighth_Doctor has joined #opendev | 09:12 | |
| openstackgerrit | Sorin Sbarnea proposed opendev/bindep master: Add OracleLinux support https://review.opendev.org/536355 | 09:29 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Only push required projects https://review.opendev.org/724626 | 09:45 |
| openstackgerrit | Sorin Sbarnea zbr proposed zuul/zuul-jobs master: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 10:04 |
| *** rpittau is now known as rpittau|bbl | 10:21 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Only push required projects https://review.opendev.org/724626 | 10:24 |
| *** ykarel|lunch is now known as ykarel | 10:25 | |
| kevinz | ianw: sorry for late reply. I manually run the tempest and it passed. It looks the test is not very stable | 10:34 |
| *** sean-k-mooney has joined #opendev | 11:07 | |
| *** ysandeep is now known as ysandeep|brb | 11:18 | |
| ttx | Hi everyone, we have a problem with tarball publication to tarballs.openstack.org. The uploaded tarballs no longer show up. Worked until 2020-04-28 14:50, failed since 2020-04-29 12:19 | 11:47 |
| ttx | Jobs report success, like for example ironic-15.0.0.tar.gz being uploaded to AFS at: https://zuul.openstack.org/build/4aeec939360d44ce8081255b8185b4ee/log/job-output.txt#4454 | 11:48 |
| ttx | And yet, it does not show up at https://tarballs.opendev.org/openstack/ironic/ | 11:49 |
| ttx | cc: fungi, corvus | 11:50 |
| *** dtantsur has joined #opendev | 11:51 | |
| *** ysandeep|brb is now known as ysandeep | 11:54 | |
| *** tkajinam has quit IRC | 12:09 | |
| *** hashar has joined #opendev | 12:10 | |
| *** roman_g has joined #opendev | 12:15 | |
| *** rpittau|bbl is now known as rpittau | 12:22 | |
| *** hrw has quit IRC | 12:33 | |
| *** hrw has joined #opendev | 12:33 | |
| *** lpetrut has joined #opendev | 12:34 | |
| frickler | infra-root: ^^ the files are in /afs/.openstack.org , but not in /afs/openstack.org , not sure why the release is failing or not happening | 12:36 |
| frickler | infra-root: also two of our hourly jobs seem broken: https://zuul.opendev.org/t/openstack/builds?pipeline=opendev-prod-hourly | 12:43 |
| * frickler is about to head into a long weekend, can't dig further | 12:43 | |
| *** dpawlik has joined #opendev | 12:46 | |
| *** dpawlik has quit IRC | 12:46 | |
| *** dpawlik has joined #opendev | 12:46 | |
| openstackgerrit | Paul Belanger proposed openstack/project-config master: Create ansible-role-boto3 project https://review.opendev.org/724640 | 12:46 |
| mordred | frickler: have a good long weekend! | 12:50 |
| AJaeger | mordred: could you review https://review.opendev.org/724607, please? That should finish the openstack git mirroring... | 12:52 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Test zuul-executor on focal https://review.opendev.org/723528 | 13:02 |
| openstackgerrit | Merged openstack/project-config master: infra-package-needs: drop ntp for Focal https://review.opendev.org/724450 | 13:07 |
| fungi | frickler: ttx: thanks, so sounds like the periodic vos release of the tarballs volume has stopped running? i'll check logs in a moment | 13:12 |
| *** ykarel is now known as ykarel|afk | 13:14 | |
| *** lpetrut has quit IRC | 13:14 | |
| AJaeger | fungi, mordred: could you review https://review.opendev.org/724607, please? That should finish the openstack git mirroring... | 13:32 |
| AJaeger | thx, fungi | 13:35 |
| fungi | you bet | 13:36 |
| fungi | still unwinding what the current state of batch vos release is, and why it stopped happening two days ago | 13:36 |
| AJaeger | yeah, that's strange | 13:37 |
| fungi | also, zuul-web is already back to being the highest memory-using process on zuul.o.o | 13:37 |
| AJaeger | :( | 13:37 |
| ttx | boo | 13:37 |
| openstackgerrit | Monty Taylor proposed opendev/ansible-role-puppet master: Add flag to control logging the rsyncs https://review.opendev.org/724418 | 13:46 |
| openstackgerrit | Merged openstack/project-config master: Add mirroring for SIG and other misc repos https://review.opendev.org/724607 | 13:46 |
| fungi | okay, so it appears /opt/afs-release/release-volumes.py runs every 5 minutes via cron on mirror-update01.opendev.org to batch release volumes like the one for the tarballs site, and its log is perpetually complaining "Unable to get lockfile!" since 2020-04-28 20:00 | 13:50 |
| fungi | according to lsof, /var/run/release-volumes.lock has an open file handle for a release-volumes.py process running since the 28th | 13:53 |
| fungi | ps says it has a child process running `vos examine docs` | 13:55 |
| fungi | and strace of that child pid shows it's perpetually waiting on a private futex | 13:55 |
| *** ysandeep is now known as ysandeep|afk | 13:56 | |
| fungi | https://wiki.openstack.org/wiki/Infrastructure_Status says afs01.dfw was rebooted around that time due to hypervisor host problems | 13:56 |
| fungi | so i'll wager this is related to that event | 13:56 |
| fungi | i'll try killing the vox examine and see if that unsticks it | 13:57 |
| fungi | sending it a hup seems to have done the trick | 13:57 |
| fungi | no remaining hold on the lockfile now | 13:58 |
| fungi | it dumped a traceback into the release-volumes log, next run should be in ~1 minute so i'm tailing the log to make sure it completes normally this time | 13:58 |
| fungi | it started at 14:00 but is still running. it starts with the docs volume, and since there's two days of churn to push out this will probably take some time | 14:05 |
| *** ykarel|afk is now known as ykarel | 14:09 | |
| *** pabelanger has joined #opendev | 14:11 | |
| pabelanger | hi! do you think I could get a +A on a new project for windmill? https://review.opendev.org/724640 | 14:12 |
| *** lpetrut has joined #opendev | 14:13 | |
| AJaeger | pabelanger: do you have any content to import or want to start from scratch? | 14:14 |
| pabelanger | AJaeger: from scratch for this one | 14:15 |
| *** tkajinam has joined #opendev | 14:16 | |
| fungi | it's gotten to the tarballs volume now | 14:17 |
| AJaeger | pabelanger: +2A | 14:20 |
| *** hashar_ has joined #opendev | 14:21 | |
| fungi | ttx: the tarballs volume was released successully as of 14:18:05 | 14:22 |
| fungi | please double-check that things look the way you expect now | 14:22 |
| *** hashar has quit IRC | 14:23 | |
| *** dtantsur has quit IRC | 14:24 | |
| pabelanger | AJaeger: thank you | 14:28 |
| *** dtantsur has joined #opendev | 14:29 | |
| openstackgerrit | Merged openstack/project-config master: Create ansible-role-boto3 project https://review.opendev.org/724640 | 14:30 |
| *** mlavalle has joined #opendev | 14:39 | |
| ttx | fungi: looks good to me | 14:41 |
| *** pabelanger has left #opendev | 14:41 | |
| ttx | fungi: it's a bit hard to detect that failure, since the upload completes successfully... hopefully it won;t happen again | 14:42 |
| corvus | i'm going to re-enqueue 723855 into promote to get a new set of zuul images | 14:44 |
| corvus | then i'll restart zuul-web with that | 14:46 |
| *** lpetrut has quit IRC | 14:46 | |
| fungi | ttx: yes, the summary is that there was some underlying problem in rackspace which hung one of our afs servers, creating a pathological condition for some afs clients. apparently one was the script performing our batch releases of content updates for the public read-only volumes backing sites like tarballs.o.o | 14:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul https://review.opendev.org/718798 | 14:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging the rsync of puppet https://review.opendev.org/724419 | 14:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use zuul checkouts of ansible roles from other repos https://review.opendev.org/724682 | 14:55 |
| fungi | ttx: so uploads to the read-write volume/filesystem were successful, they just weren't being reflected by the website itself | 14:56 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix another incorrect variable name for promote-docker-image https://review.opendev.org/724686 | 15:03 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and ensure roles https://review.opendev.org/693513 | 15:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul https://review.opendev.org/718798 | 15:10 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use zuul checkouts of ansible roles from other repos https://review.opendev.org/724682 | 15:10 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging the rsync of puppet https://review.opendev.org/724419 | 15:10 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 15:10 |
| mordred | corvus: yesterday's "let's not log the rsyncs for puppet" task led me down the rabbit hole of "let's make sure we're running our ansible roles from zuul checkouts so that depends-on works" | 15:11 |
| clarkb | fungi: re zuul-web see corvus message at 14:44. We never ended up with a newer image promotion yesterday due to job bugs | 15:17 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix another incorrect variable name for promote-docker-image https://review.opendev.org/724686 | 15:17 |
| fungi | clarkb: yep | 15:17 |
| clarkb | infra-root have a moment for https://review.opendev.org/#/c/724389/ it is epxected to fix a job failure preventing the system-config zuul.d change from merging | 15:18 |
| clarkb | fungi: no lists oom this morning either | 15:18 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and ensure roles https://review.opendev.org/693513 | 15:27 |
| clarkb | maybe semrush bot is also making zuul-web unhappy? | 15:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Test zuul-executor on focal https://review.opendev.org/723528 | 15:44 |
| fungi | mebbe | 15:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul https://review.opendev.org/718798 | 15:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use zuul checkouts of ansible roles from other repos https://review.opendev.org/724682 | 15:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging the rsync of puppet https://review.opendev.org/724419 | 15:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 15:47 |
| mordred | clarkb: don't hate me ... but I've got another change that touches a non-zero amount of .zuul.yaml - have you updated your split-zuul-yaml patch yet? | 15:48 |
| clarkb | mordred: not yet but we need job fixes anyway so I plan to redo it once job things should be happy | 15:51 |
| mordred | clarkb: cool. I thnik the thing I've got in flight should be reasonable enough to land pretty quickly | 15:51 |
| mordred | clarkb: 724682 - use ansible roles from zuul intead of direct clones | 15:52 |
| mordred | it came from a "I made a change in the ansible-role-puppet repo and I want to depends-on it in system-config to see it work but that's not a thing" moment this morning :) | 15:53 |
| clarkb | mordred: k will try to digest it shortly | 16:03 |
| mordred | clarkb: no rush | 16:04 |
| *** rpittau is now known as rpittau|afk | 16:07 | |
| *** hashar_ has quit IRC | 16:10 | |
| clarkb | mordred: left a couple notes on it. I don't think either thing is critical | 16:22 |
| mordred | k. I've also got a path thing to fix :) | 16:24 |
| clarkb | fwiw there are SemrushBot entries in zuul web's access logs, but a tiny number of them compared to lists | 16:27 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use zuul checkouts of ansible roles from other repos https://review.opendev.org/724682 | 16:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging the rsync of puppet https://review.opendev.org/724419 | 16:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 16:28 |
| *** ykarel is now known as ykarel|afk | 16:31 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Test zuul-executor on focal https://review.opendev.org/723528 | 16:31 |
| mordred | stab stab stab | 16:31 |
| mordred | clarkb, donnyd: the attempt to get zuul-jobs-test-registry-docker-multiarch to run on an ipv6-only node is failing - I'm using nested-virt-ubuntu-bionic but that seems to have vexxhost and ovh in it | 16:34 |
| *** tkajinam has quit IRC | 16:34 | |
| donnyd | mordred: I don't think I have that label in OE | 16:34 |
| donnyd | checking | 16:34 |
| clarkb | mordred: re https://review.opendev.org/#/c/724682/3..4/playbooks/roles/install-ansible-roles/tasks/main.yaml is the dest value there correct? | 16:35 |
| donnyd | oh I was looking in the wrong section | 16:35 |
| clarkb | mordred: donnyd ya this si what I was trying to point out. We don't really have a good cross section for ipv6 only labels | 16:36 |
| donnyd | multi-numa-ubuntu-bionic | 16:36 |
| mordred | clarkb: nope! | 16:36 |
| donnyd | give that one a whirl instead | 16:36 |
| donnyd | I think that is OE only | 16:36 |
| *** ykarel|afk is now known as ykarel|away | 16:36 | |
| mordred | donnyd: ++ thanks! | 16:36 |
| clarkb | ya the numa might be the only OE unique labels | 16:36 |
| donnyd | It is a good point - clarkb do you think we should have something that calls out ipv6 specifically ? | 16:37 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: DNM Run builder tests on expanded node https://review.opendev.org/724079 | 16:37 |
| clarkb | donnyd: if we get more of these things popping up then adding a limestone + OE label for ipv6 might be a good idea | 16:37 |
| donnyd | agreed | 16:37 |
| clarkb | for now I think we can get by with overloading numa, but if we see this occuring more often then a specific label will make it easier | 16:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use zuul checkouts of ansible roles from other repos https://review.opendev.org/724682 | 16:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging the rsync of puppet https://review.opendev.org/724419 | 16:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 16:38 |
| donnyd | there are 10 available slots and I can always bump that number if its being used more | 16:38 |
| mordred | well - in this case I think as long as we can get one run there we should be ok | 16:38 |
| mordred | I don't think we need to run there on every patch - it's more just a verificaiton that a specific ipv6 thing worked | 16:39 |
| clarkb | ++ | 16:40 |
| clarkb | I think for this case that is definitely good enough | 16:41 |
| openstackgerrit | Merged opendev/puppet-mailman master: Create /srv/mailman https://review.opendev.org/724389 | 17:00 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and ensure roles https://review.opendev.org/693513 | 17:06 |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: DNM Run builder tests on multi-numa-ubuntu-bionic node https://review.opendev.org/724079 | 17:13 |
| clarkb | \o/ system-config-run-lists succeeds now | 17:30 |
| clarkb | mordred: where are we on nodepool ansible docker things? That is the last outstanding issue for the system config job split | 17:31 |
| mordred | clarkb: it's ready - as soon as we're ready to roll it out, watch it - and then chown / restart things | 17:32 |
| mordred | clarkb: the other pre-split patch - amusingly enough - failed on run-lists :) | 17:33 |
| mordred | clarkb: rebasing/rechecking should clear that now I'd imagine, right? | 17:33 |
| clarkb | mordred: just a recheck (the fix was in puppet-mailman) | 17:33 |
| mordred | clarkb: cool - recheck done | 17:34 |
| mordred | otherwise the patch is green: https://review.opendev.org/#/c/724682/ and https://review.opendev.org/#/c/718798/ | 17:34 |
| clarkb | mordred: one thing I'm semi worried about as a general thing is we landed a change to puppet-mailman but nothing is running infra-prod-lists as a result | 17:34 |
| mordred | clarkb: although I'm betting it's going to conflict on the nodepool patch | 17:34 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: use-buildset-registry: add Fedora certifacts vars https://review.opendev.org/724717 | 17:34 |
| mordred | so maybe I should stack it on top of nodepool, we can land/roll out nodepool, then land it - then do the split? | 17:34 |
| clarkb | mordred: in this case we are ensuring a preexisting dir is a dir in puppet so it should noop and be totally be fine, but in general we may want to more aggreveiyl run things so they don't break hours later | 17:34 |
| clarkb | mordred: the split will conflict I'm almost sure of it. I was just going to redo it again once you landed the nodepool things | 17:35 |
| mordred | clarkb: yes - as a followup to the "run ansible roles from zuul" change I want to make a similar one for running puppet modules from zuul git - and then we can start adding the system-config run jobs to the puppet modules | 17:35 |
| clarkb | mordred: my concern with rebasing it on top is if other things land too | 17:35 |
| mordred | clarkb: yeah - I'm saying I think I want to rebase my "run ansible roles" patch on top of the nodepool patch - since I know they have a conflict - then let's land both of them, then you an re-do the split patch | 17:36 |
| clarkb | mordred: basically rebasing on the nodepool change isn't necessarily enough if the nodepool change ends up behind master too | 17:36 |
| clarkb | oh I see | 17:36 |
| clarkb | got change streams crossed | 17:36 |
| mordred | because I think we're pretty much ready to land nodepool | 17:36 |
| clarkb | mordred: it also looks like we haven't run puppet on lists in days | 17:37 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nodepool launchers with ansible and containers https://review.opendev.org/720527 | 17:37 |
| clarkb | is that expected? I thought at the very least we were running things daily | 17:37 |
| *** dpawlik has quit IRC | 17:38 | |
| clarkb | infra-prod-remote-puppet-else that is the job that should hit lists I think | 17:38 |
| clarkb | and we run that hourly | 17:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul https://review.opendev.org/718798 | 17:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use zuul checkouts of ansible roles from other repos https://review.opendev.org/724682 | 17:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging the rsync of puppet https://review.opendev.org/724419 | 17:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 17:39 |
| mordred | clarkb: I agree- I think we recently fixed an issue with remote-puppet-else - maybe there's another one? | 17:40 |
| mordred | clarkb: oh - no - infra-prod-run-lists is the lists puppet job | 17:40 |
| mordred | nope. I take that back - you were right | 17:41 |
| mordred | I don't know why puppet hasn't run on lists -we should be running it hourly | 17:41 |
| clarkb | mordred: I think I know | 17:42 |
| clarkb | mordred: we stopped logging to syslog :) | 17:42 |
| clarkb | and I've become so accustomed to check that that I assumed no syslog means no runs :) | 17:42 |
| openstackgerrit | melanie witt proposed opendev/elastic-recheck master: Update query for bug 1840159 https://review.opendev.org/724719 | 17:42 |
| openstack | bug 1840159 in OpenStack Compute (nova) "nova-grenade-live-migration intermittently fails with "Error monitoring migration: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePerform3Params)"" [Low,Confirmed] https://launchpad.net/bugs/1840159 | 17:42 |
| clarkb | mordred: I've found it in the most recent infra-prod-remote-puppet-else job so I think it is actually being puppeted hourly | 17:43 |
| clarkb | mordred: what I notice in ^ is we don't have timestamp logs like we did with syslog | 17:43 |
| openstackgerrit | melanie witt proposed opendev/elastic-recheck master: Update query for bug 1840159 https://review.opendev.org/724719 | 17:43 |
| clarkb | Notice: Compiled catalog for lists.openstack.org in environment production in 3.49 seconds <- probably is I don't know when it did that | 17:43 |
| clarkb | it was near Thursday 30 April 2020 17:39:47 +0000 I think | 17:44 |
| mordred | clarkb: yeah - also - we seem to only have old saved log files | 17:44 |
| mordred | clarkb: is our clean-old-logs logic backwards? | 17:44 |
| mordred | the most recent one we have is remote-puppet-else.yaml.log.2020-04-20T15:21:24Z - and then the most current | 17:45 |
| clarkb | mordred: remote_puppet_else.yaml.log.2020-04-30T16:26:02Z that exists too | 17:45 |
| clarkb | I think you are just sorted in reverse? | 17:45 |
| clarkb | anyway I think this is all workign largely as expected and I just need to stop looking at syslog first | 17:46 |
| clarkb | if we can work timestamps into those puppet logs in ansible that would be extra good | 17:46 |
| mordred | clarkb: root@bridge:/var/log/ansible# ls -ltra remote-puppet-else.yaml.log* | 17:46 |
| mordred | clarkb: I'm not seeing ... OH | 17:46 |
| mordred | _ | 17:46 |
| mordred | clarkb: got it :) | 17:47 |
| clarkb | underscores vs -'s | 17:47 |
| clarkb | (I guess the playbook changed its name?) | 17:48 |
| mordred | clarkb: logstash01.openstack.org : ok=24 changed=3 unreachable=0 failed=1 skipped=17 rescued=0 ignored=0 | 17:48 |
| mordred | we failed on logstash in the one I opened up | 17:48 |
| mordred | clarkb: health01.openstack.org : ok=24 changed=3 unreachable=0 failed=1 skipped=17 rescued=0 ignored=0 | 17:49 |
| mordred | in the one before that | 17:49 |
| clarkb | mordred: is there a way to make the puppte logs more useful in ansible? | 17:49 |
| clarkb | (I don't want to suggest it but honestly syslog was way more useful for debugging why and how puppet didn't work) | 17:49 |
| clarkb | mordred: Error: Execution of '/usr/bin/git fetch origin' returned 128: fatal: unable to access 'https://github.com/elasticsearch/kibana.git/': GnuTLS recv error (-9): A TLS packet with unexpected length was received. | 17:50 |
| clarkb | thats the logstash error | 17:50 |
| mordred | clarkb: I'm sure there is - or we can go back to the other thing | 17:50 |
| mordred | I found the syslog thing horrific - but I seem to be the only one who did :) | 17:51 |
| clarkb | that git repo still exists, was just a github blip I guess | 17:51 |
| clarkb | mordred: in particular I don't want split stdout and stderr especially without timestamps because the order of those lines is important to udnerstanding where puppet went wrong | 17:51 |
| clarkb | with timestamps we'd at least be able to compare them | 17:51 |
| mordred | clarkb: I don't think the puppet role is running | 17:51 |
| mordred | and I thnik it's not running silently | 17:51 |
| mordred | clarkb: look at remote_puppet_else.yaml.log.2020-04-30T15\:32\:32Z | 17:52 |
| clarkb | mordred: puppet is running the logs for logsatsh are from puppet | 17:52 |
| clarkb | it failed in puppet trying to fetch from github | 17:52 |
| mordred | clarkb: there is nothing after disable puppet agent in the log | 17:52 |
| mordred | hrm - really? | 17:52 |
| clarkb | mordred: ya look in the most recent file without a timestamp | 17:52 |
| mordred | oh - I guess so | 17:52 |
| clarkb | remote_puppet_else.yaml.log | 17:52 |
| mordred | yeah | 17:53 |
| mordred | I'm with you | 17:53 |
| clarkb | for me I think the ~3 things I would try and address with logging are timestamps with puppet output, combining stderr and stdout if possible (in order they are emitted), and better identifying what tasks the stdout and stderr belong to | 17:53 |
| clarkb | I don't think figuring that out is an emergency but something to keep in mind as we ansible more | 17:54 |
| mordred | clarkb: yeah - unfortunately combining stdout/stderr would require running hte zuul callback plugin :) | 17:54 |
| mordred | or - writing our own | 17:54 |
| clarkb | mordred: we could also have the command combine them | 17:54 |
| clarkb | puppet-apply 2>&1 for whatever | 17:55 |
| mordred | that's true - maybe we should do that in the short term | 17:55 |
| clarkb | (that might also be the way we add timestamps is via wrappign the command with something like what devstack does/did) | 17:55 |
| mordred | yeah | 17:55 |
| mordred | assuming the puppet output is incremental and not batched | 17:55 |
| fungi | zbr: you were saying in #zuul you have a google analytics cookie which was set by opendev.org? if so, i'd like to figure out how and where it came from | 17:55 |
| clarkb | mordred: I want to say it is when run interactively, I distinctly remember getting christmas tree output in scrolly fashion | 17:56 |
| mordred | clarkb: I believe you are correct | 17:59 |
| *** diablo_rojo has joined #opendev | 18:00 | |
| clarkb | fungi: zbr as a sanity check neither opendev.org nor zuul.opendev.org have a ga cookie in firefox for me. However, I block so much stuff its possible that I'm filtering that at a higher level. However, I am not aware of anywhere we set up ga for opendev.org so would be very surprised to see that we do | 18:00 |
| clarkb | I agree more details would be useful if they can be shared | 18:01 |
| zbr | interesting because i seen one in by browser, but i have no idea where it came from. | 18:01 |
| mordred | clarkb: sigh: rc, stdout, stderr = module.run_command(cmd) ... it's going to be slightly harder to combine than just adding 2>&1 | 18:01 |
| clarkb | zbr: can you double check it is for opendev.org? | 18:01 |
| clarkb | zbr: see fungi's note | 18:01 |
| clarkb | mordred: I think we'll just get an empty stderr in that case? | 18:01 |
| clarkb | mordred: we might have to switch to shell over command though | 18:01 |
| zbr | it is for .opendev.org to be precise. | 18:02 |
| clarkb | zbr: and it is present on zuul.opendev.org? | 18:02 |
| zbr | obviously, that is the meaning of using "." | 18:02 |
| clarkb | no its not obvious neither fungi nor I can reprdouce that and nowhere do we configure google analytics in our config management | 18:03 |
| fungi | zbr: using the web inspector storage tab in ff to look at cookies for opendev.org, i don't see a _ga one | 18:03 |
| fungi | there's a gitea cookie, a csrf cookie and a lang cookie | 18:03 |
| mordred | clarkb: we are not using shell or command | 18:04 |
| clarkb | and on zuul.opendev.org is a zuul_filter_string | 18:04 |
| clarkb | mordred: oh right its python module, got it | 18:04 |
| mordred | clarkb: we have a custom python module that runs puppet | 18:04 |
| mordred | yeah | 18:04 |
| mordred | clarkb: you know ... | 18:04 |
| zbr | https://sbarnea.com/ss/Screen-Shot-2020-04-30-19-04-10.98.png | 18:04 |
| fungi | zbr: if i go to zuul.opendev.org as you mentioned, i see no cookies set at all | 18:04 |
| mordred | clarkb: we COULD have the python module just return the constructed command string | 18:04 |
| mordred | and then run it with shell | 18:04 |
| mordred | oh - but then we'd have to manage return codes in ansible | 18:05 |
| fungi | zbr: wow, i'd love to know what set that. i don't think anything we're running would have | 18:05 |
| clarkb | fungi: is it possible something from docs.opendev.org set that? | 18:05 |
| zbr | https://sbarnea.com/ss/Screen-Shot-2020-04-30-19-05-08.65.png | 18:05 |
| mordred | clarkb: Imma try it anyway | 18:05 |
| clarkb | mordred: or set up the subprocess to combine sterr and stdout | 18:05 |
| zbr | i am not sure who, maybe is a browser extension. | 18:05 |
| mordred | clarkb: we do not have control over the subprocess | 18:05 |
| mordred | clarkb: that's done in the python in module.run_command | 18:06 |
| fungi | clarkb: i hope not or we're going to have to find some way to filter cookies from the docs site | 18:06 |
| clarkb | zbr: what re teh contents of the cookie? ( can that generally be shared?) | 18:06 |
| mordred | we could copy module.run_command in to the puppet module and modify it - but it's not small | 18:06 |
| clarkb | mordred: hrm ya I've not got better ideas then | 18:06 |
| clarkb | (I'm hoping the contents might better hint at its origin) | 18:07 |
| zbr | is already visible in the screenshot, afaik is an unique identifier used by google, so now you could track my ff visits :) | 18:08 |
| fungi | i have a sinking feeling google would rather you not know who's setting that cookie | 18:08 |
| fungi | but yeah, the "content" of that cookie is just what's in the value column there in the screenshot | 18:08 |
| fungi | whatever you've happened across which set that cookie, i have not managed to wind up with the same | 18:09 |
| fungi | but it's possible it was blocked by my strict no-track browser settings or one of the tracker-blocking extensions i have loaded (eff privacy badger, ddg privacy essentials...) | 18:09 |
| clarkb | http://codesearch.openstack.org/?q=GA1.2.833802541.1565283387&i=nope&files=&repos= has no results | 18:10 |
| AJaeger | zbr: which URL are you visiting? | 18:10 |
| fungi | AJaeger: zuul.opendev.org | 18:10 |
| clarkb | fungi: ya I go a step further and block things in my firewall and dns too | 18:11 |
| fungi | fwiw, the extensions i've got loaded show that they see 0 trackers on that site, so they don't seem to be blocking anything there | 18:11 |
| AJaeger | I can't reproduce that and I'm puzzled | 18:11 |
| openstackgerrit | Monty Taylor proposed opendev/ansible-role-puppet master: Run puppet in shell and combine stderr/stdout https://review.opendev.org/724724 | 18:11 |
| mordred | clarkb: ^^ | 18:11 |
| * AJaeger has only "zuul_filter_string" on zuul.opendev.org, no other cookie | 18:12 | |
| AJaeger | and I just disabled extensions etc | 18:12 |
| mordred | clarkb: I think if we like that- we should just copy the puppet role into system-config, retire ansible-role-puppet and remove a bunch of the configurability - because I thnk we're potentially well past "this is a general purpose role for running puppet" | 18:12 |
| fungi | zbr: you will definitely find a domain-wide _ga cookie being set for .openstack.org though and that's the osf's doing (and one of the reasons we would like to relocate opendev services to a domain where we control everything running there) | 18:12 |
| clarkb | thinking about where it could be coming from: could one of zuul web's js libs be setting it? | 18:14 |
| clarkb | or something in docs? | 18:14 |
| AJaeger | openstackdocstheme sets a cookie - but your search should have found that one | 18:14 |
| clarkb | AJaeger: well lets double check | 18:15 |
| fungi | i suppose it's possible anything published to docs.opendev.org with openstackdocstheme could have dynamically added a .opendev.org cookie, i suppose | 18:15 |
| mordred | clarkb, fungi, AJaeger: I see ga( being called in a few places: | 18:16 |
| mordred | https://opendev.org/openstack/openstack-manuals/src/branch/master/www/templates/google_analytics.tmpl#L8 | 18:16 |
| mordred | https://opendev.org/openstack/openstackdocstheme/src/branch/master/openstackdocstheme/theme/openstackdocs/layout.html#L30 | 18:16 |
| AJaeger | openstackdocstheme does not enable the cookie anymore | 18:16 |
| AJaeger | see I156151c3f458931f13716b4a32087afe57483794 | 18:16 |
| AJaeger | and I don't see us setting the code - nice ;) | 18:16 |
| mordred | maybe this is an old cookie that got set for zbr before it was disabled? | 18:17 |
| AJaeger | clarkb: see https://opendev.org/openstack/openstackdocstheme/src/branch/master/openstackdocstheme/theme/openstackdocs/layout.html#L22 | 18:17 |
| AJaeger | and we do not define the var | 18:17 |
| mordred | I agree | 18:18 |
| fungi | entirely possible that was set by something which no longer sets it, yeah | 18:18 |
| zbr | i am not worried much about the _ga subject | 18:18 |
| fungi | infra-root: should i restart zuul-web's container again? we're down to ~5gb ram remaining on the server | 18:19 |
| clarkb | fungi: I think corvus was waiting for new image to be available (I'm not sure where that is in the promotion pipeline) | 18:19 |
| *** sangeet has joined #opendev | 18:19 | |
| fungi | ahh, that'll probably be fine | 18:20 |
| fungi | rate of growth is not entirely linear, but looks like we'll survive at least a few more hours | 18:20 |
| sangeet | I am trying to find the lable for 32gb node for ubuntu for zuul .. please help | 18:20 |
| mordred | clarkb, fungi I just pinged you in #zuul by mistake | 18:21 |
| AJaeger | so, docs.o.o index pages do have the analytics code, but not the project content below that | 18:21 |
| *** avass has joined #opendev | 18:21 | |
| mordred | for the record here: | 18:21 |
| mordred | clarkb: I'm still getting a puppet error on lists in the gate: https://75ab7718a1ac933a54f7-6414947d36de3e18491bc440889da30b.ssl.cf2.rackcdn.com/724682/6/check/system-config-run-lists/6f39839/bridge.openstack.org/ara-report/result/513b6b1c-6952-4838-8f60-64288f83627a/ | 18:21 |
| clarkb | AJaeger: .o.o is openstack.org or opendev.org ? or maybe both? | 18:21 |
| mordred | clarkb: and it's because you fixed listpassword in a different change: https://review.opendev.org/#/c/724356/3 | 18:22 |
| mordred | fungi: ^^ would you mind reviewing that? | 18:22 |
| mordred | fungi: yay! thank you | 18:22 |
| AJaeger | clarkb: docs.openstack.org | 18:22 |
| fungi | mordred: yeah, sorry, i approved it after you pinged me in #zuul | 18:22 |
| AJaeger | clarkb: on docs.opendev.org we have no indices ;) | 18:23 |
| mordred | fungi: that's great! - no apology needed | 18:23 |
| mordred | also - booyah - https://review.opendev.org/#/c/723528/ - focal tests passing | 18:23 |
| mordred | clarkb, fungi, corvus: https://review.opendev.org/#/c/723528/ is green, should be safe to land - but it updates us to work on focal so we can start rolling out zuul-executor replacement servers using focal | 18:24 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Add semaphore to publish-tox-docs-releases job https://review.opendev.org/724727 | 18:24 |
| clarkb | AJaeger: is it a different value or the same value? | 18:25 |
| AJaeger | clarkb: the docs.o.o _ga is a different value than the one zbr showed | 18:27 |
| zbr | AJaeger: to be clear, _ga value is unique to each user! | 18:27 |
| zbr | its value will not help you identify who created in any way | 18:28 |
| clarkb | gotcha | 18:28 |
| clarkb | it must be hashed with the site identifer then | 18:28 |
| zbr | it could as well be a randomnumber | 18:28 |
| clarkb | zbr: if you delete it does it come back? | 18:28 |
| clarkb | (that might be the easiest way to try and track it down?) | 18:28 |
| zbr | nope | 18:29 |
| zbr | i removed it, it is gone | 18:30 |
| mordred | good! | 18:30 |
| clarkb | we can probably grep over our afs content to double check things there. It will be potentially slow | 18:31 |
| clarkb | the other place it could be is zuul-preview? | 18:31 |
| fungi | that's possible, yep | 18:32 |
| clarkb | the logs themselves aren't served via .opendev.org so couldn't set an .opendev.org cookie | 18:32 |
| clarkb | I'm going to pop out for an early lunch now, back in a bit. | 18:33 |
| clarkb | mordred: do I need to rereview nodepool things? | 18:33 |
| fungi | clarkb: also a very remote possibility, any of the site content we proxy could set cookies for our domain if a browser views them | 18:38 |
| clarkb | fungi: oh like pypi? | 18:38 |
| *** ralonsoh has quit IRC | 18:38 | |
| fungi | (e.g., through our ci "mirror" proxies) | 18:38 |
| clarkb | ya | 18:38 |
| fungi | i think they'd have to know they were being proxied and set the right domain in them though | 18:38 |
| fungi | apache isn't going to rewrite the domains in them on the fly | 18:39 |
| clarkb | it does in headers but not content | 18:39 |
| corvus | i'll look at restarting zuul-web now | 18:41 |
| fungi | clarkb: from what i'm reading you need to use mod_headers to get that behavior | 18:41 |
| openstackgerrit | Merged opendev/system-config master: Set up robots.txt on lists servers https://review.opendev.org/724356 | 18:42 |
| corvus | zbr: i just restarted zuul-web, and it should have included your wrap change but i don't see it in effect here: https://zuul.opendev.org/t/zuul/build/981004fed38944798ab1bca49a622d7e/console#3/0/5/ubuntu-bionic | 18:45 |
| corvus | i have confirmed that pip freeze say we're running CherryPy==18.3.0 | 18:45 |
| corvus | #status log restarted zuul-web with image built from equivalent of zuul commit d17b8e7f7f5b68c93c6d2bdba1b94df87f8ee93d (cherrypy pin 18.3.0) | 18:46 |
| openstackstatus | corvus: finished logging | 18:46 |
| corvus | it looks like we probably have enough of a history of memory growth with clarkb's change in place we can compare the memory growth over the next little bit with that | 18:48 |
| mordred | clarkb: yes please - a re-review of nodepool would be great | 18:55 |
| mordred | clarkb: I'm rechecking https://review.opendev.org/#/c/724682/ now that your lists fix is landed | 18:55 |
| donnyd | could someone point me to the repo where we build our cloud images | 19:02 |
| donnyd | pretty please | 19:02 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging the rsync of puppet https://review.opendev.org/724419 | 19:02 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 19:02 |
| clarkb | donnyd: openstack/project-config/nodepool | 19:03 |
| mordred | corvus, clarkb: the nodepool patch, https://review.opendev.org/#/c/720527/ - is a rebase from the previous patch to deal with a merge conflict | 19:03 |
| clarkb | thats the configs and elements that produce the images if that is ehat you mean | 19:03 |
| fungi | donnyd: maybe you're looking for https://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml#L131 | 19:05 |
| fungi | that's basically nodepool's entry point for definining what goes into images (which dib elements to include, what variables to set...) | 19:06 |
| donnyd | bingo | 19:06 |
| donnyd | thanks clarkb and fungi | 19:06 |
| fungi | yw! | 19:06 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Add semaphore to publish-tox-docs-releases job https://review.opendev.org/724727 | 19:12 |
| *** hashar has joined #opendev | 19:28 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: use-buildset-registry: remove unused test library from remarshal https://review.opendev.org/724735 | 19:30 |
| openstackgerrit | Merged openstack/project-config master: Add semaphore to publish-tox-docs-releases job https://review.opendev.org/724727 | 19:31 |
| mordred | clarkb: woot - https://review.opendev.org/#/c/724682 is green. it's stacked on top of nodepool | 19:38 |
| mordred | so when you get to the nodepool change - if you can review it too I think that's the last big .zuul.yaml change before re-doing the split | 19:38 |
| mordred | corvus: ^^ same to you | 19:39 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Control log archive and user preservation with vars https://review.opendev.org/701381 | 20:01 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Control log archive and user preservation with vars https://review.opendev.org/701381 | 20:05 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Control log archive and user preservation with vars https://review.opendev.org/701381 | 20:09 |
| openstackgerrit | Kendall Nelson proposed opendev/irc-meetings master: Change Meeting Room for StoryBoard Meeting https://review.opendev.org/724743 | 20:21 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add configure-podman-registry role https://review.opendev.org/724744 | 20:22 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: DNM Check to see if images from intermediate work https://review.opendev.org/724751 | 20:31 |
| clarkb | mordred: I've +2'd the nodepool change: https://review.opendev.org/#/c/720527/ but not approved it because I don't think I'm in a great spot to clsoely monitor that right now | 20:32 |
| * clarkb needs to write emails and stuff | 20:32 | |
| mordred | clarkb: ++ | 20:33 |
| mordred | clarkb: let's +A it when we're appropriately ready | 20:33 |
| fungi | clarkb: was https://review.opendev.org/718790 supposed to actually include the logo file? did you forget to git add, or is there a followup change? | 20:34 |
| fungi | https://docs.opendev.org/opendev/system-config/latest/_static/opendev.svg is a 404 for me | 20:35 |
| clarkb | fungi: I think I failed to git add | 20:35 |
| clarkb | hrm working dir is clean too | 20:36 |
| fungi | i wanted to reuse that for a new project, which is how i discovered the file is not actually present | 20:36 |
| openstackgerrit | Merged opendev/irc-meetings master: Change Meeting Room for StoryBoard Meeting https://review.opendev.org/724743 | 20:36 |
| clarkb | fungi: it is present in infra-manual | 20:36 |
| clarkb | fungi: let me fix system-config real quick | 20:36 |
| fungi | cool, i'll snag it from there. thanks!!! | 20:36 |
| fungi | and happy to review | 20:36 |
| donnyd | does anyone know if there is an element in DIB to tell it how large you want an image to be? | 20:37 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add logo file to docs https://review.opendev.org/724752 | 20:38 |
| fungi | donnyd: the rootfs size should be determined by the instance flavor in nova | 20:38 |
| clarkb | donnyd: there is a var you can set, by default it does some headroom over the size of things it built. Then you can add growroot element to fill available disk when booted | 20:39 |
| clarkb | separately you can set up a partition table but that is under documented and may be a fun journey :) | 20:39 |
| donnyd | fungi: I dont' really have that option for this case | 20:39 |
| clarkb | fungi: https://review.opendev.org/724752 | 20:39 |
| fungi | ahh | 20:39 |
| fungi | yeah, so clarkb's suggestion then | 20:39 |
| clarkb | donnyd: what are you trying to do with the sizing? | 20:39 |
| clarkb | donnyd: DIB_IMAGE_SIZE is the var | 20:39 |
| donnyd | I need to create a image that is a specific size that is large than the defaults | 20:40 |
| donnyd | where did you find that clarkb | 20:40 |
| fungi | hopefully it's in the diskimage-builder docs | 20:40 |
| clarkb | donnyd: doc/source/user_guide/building_an_image.rst | 20:40 |
| clarkb | fungi: yup | 20:40 |
| clarkb | donnyd: looks like --image-size is the comamnd line flag that maps to that env var so can set it both ways | 20:41 |
| donnyd | thank you - that is exactly what I am looking for | 20:41 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write a buildkitd config file pointing to buildset registry https://review.opendev.org/724757 | 20:46 |
| *** osmanlicilegi has quit IRC | 20:53 | |
| *** cgoncalves has quit IRC | 20:53 | |
| *** dzho has quit IRC | 20:53 | |
| *** openstackgerrit has quit IRC | 20:53 | |
| *** dzho has joined #opendev | 20:53 | |
| *** cgoncalves has joined #opendev | 20:55 | |
| *** osmanlicilegi has joined #opendev | 20:55 | |
| *** sean-k-mooney has quit IRC | 20:58 | |
| clarkb | mordred: cloud launcher change lgtm too. I did leave a couple notes on the change but meh I'm overthinking it probably | 20:58 |
| clarkb | mordred: bug on https://review.opendev.org/#/c/724419/8 | 21:02 |
| *** sean-k-mooney has joined #opendev | 21:05 | |
| *** smcginnis has quit IRC | 21:06 | |
| *** smcginnis has joined #opendev | 21:07 | |
| mordred | clarkb: thanks - how does that look? now? | 21:12 |
| clarkb | mordred: +2 looks great | 21:13 |
| mordred | woot! | 21:15 |
| clarkb | fungi (and others) http://lists.opendev.org/pipermail/service-discuss/2020-April/000015.html | 21:17 |
| clarkb | that came up on Tuesday's meeting and I promised I'd send and explicit volunteer post to the list before the deadline | 21:17 |
| fungi | you've got my vote! ;) | 21:19 |
| melwitt | hm, looks like the openstackgerritbot left all channels | 21:27 |
| *** hashar has quit IRC | 21:27 | |
| fungi | it does look that way. checking on it now | 21:31 |
| smcginnis | Poor little bot. | 21:31 |
| fungi | departed at 20:53 with no clear explanation | 21:31 |
| fungi | oh, <-- openstackgerrit (~openstack@review01.openstack.org) has quit (*.net .split) | 21:32 |
| fungi | a few other users fell off at the same time | 21:32 |
| fungi | and then returned moments later | 21:32 |
| fungi | but not so much the bot | 21:33 |
| fungi | #status log restarted gerritbot after it failed to rejoin channels following a freenode netsplit | 21:34 |
| openstackstatus | fungi: finished logging | 21:34 |
| *** openstackgerrit has joined #opendev | 21:34 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write a buildkitd config file pointing to buildset registry https://review.opendev.org/724757 | 21:34 |
| melwitt | thanks! | 21:35 |
| fungi | melwitt: thanks for the heads up! | 21:35 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Boot test of project-config elements https://review.opendev.org/724453 | 21:54 |
| ianw | ^ now i think about, even with the job it's not going to speculatively test changes to the project-config elements anyway, is it? | 22:04 |
| *** larainema has quit IRC | 22:05 | |
| clarkb | ianw: correct, project-config is a trusted repo so its all post review appliaction of updates there | 22:07 |
| ianw | yeah, i wonder if we should move the elements out of there | 22:08 |
| clarkb | infra-root: in checking on zuul-web memory use I've noticed that / on zuul01 is slowly filling up. I've found /var/log/zuul/debug.log.ianw is a big consumer of disk. ianw can I delete that? it is from last september | 22:13 |
| clarkb | that should give us plenty of headroom | 22:13 |
| ianw | clarkb: oh, yes, sorry | 22:14 |
| clarkb | that one file is about 1/3 of /var/log/zuul consumption | 22:14 |
| ianw | please delete | 22:14 |
| clarkb | ianw: thanks will do | 22:14 |
| clarkb | oh wait | 22:15 |
| clarkb | /var/log/zuul is its own device so that doesn't help / | 22:15 |
| fungi | clarkb: oh, that reminds me, by chance today i spotted that our filesystems on review.o.o are getting *very* full but didn't get a chance to investigate yet | 22:15 |
| clarkb | I've deleted the file anyway but still looking for where the big consumption is | 22:15 |
| clarkb | fungi: those git repos don't get any smaller | 22:15 |
| fungi | indeed, but the rootfs is nearly full too | 22:15 |
| clarkb | fungi: for rootfs is probably /home/gerrit2/review_site things I said I'd cleanup around december but then there weren't a lot of people around to help double check that so I got scared and left it be | 22:16 |
| clarkb | fungi: I made up a paste of the things to delete on gerrit I think. I expect if we clean that up we'll get plenty of new space | 22:16 |
| clarkb | of course finding that paste now might be difficult | 22:17 |
| fungi | i expect we'll need to do something before we can upgrade anyway | 22:17 |
| clarkb | ya we should find that paste, look it over, then delete things | 22:17 |
| clarkb | I' | 22:18 |
| clarkb | er | 22:18 |
| clarkb | I'm looking on eavesdrop now for when I talked about it | 22:18 |
| fungi | doesn't sound familiar, so i was likely out to sea by then | 22:19 |
| clarkb | fungi: (and infra-root) http://paste.openstack.org/show/BoP6WhVAe5XbXtf8gDUC/ is what I have found for gerrit cleanup | 22:20 |
| clarkb | from http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-12-18.log.html#t2019-12-18T16:33:39 | 22:20 |
| fungi | yep, that's just as i was getting myself lost in the bermuda triangle | 22:21 |
| ianw | clarkb: yeah, i feel like those backups were from project rename steps? | 22:21 |
| clarkb | ianw: ya and being many yaers old don't have much value anymore | 22:21 |
| ianw | anything in /home/gerrit2/ianw can go | 22:22 |
| clarkb | note /home/gerrit2 in its entirety is a separate device on that host | 22:22 |
| clarkb | so like zuul the cost is somewhere outside of expected | 22:22 |
| clarkb | (we should still cleanup the unnecessary stuff in /home/gerrit2 though) | 22:22 |
| ianw | i agree typing "rm" on that server is scary though :) | 22:23 |
| clarkb | ok on zuul we've got a 21GB /root/.bup | 22:24 |
| clarkb | its 13GB on review.openstack.org | 22:24 |
| fungi | 12gb in /home, 23gb in /opt, 15gb in /root... | 22:25 |
| fungi | though /opt is also a separate fs | 22:25 |
| clarkb | /opt is the ephemeral drive | 22:26 |
| fungi | so the nearly-full rootfs on review.o.o will be best addressed by cleaning up homedirs for root and anyone else who isn't gerrit2 | 22:26 |
| clarkb | fungi: yes though I'm not sure how safe it is to clean out /root/.bup | 22:26 |
| clarkb | is that used as a spool? | 22:27 |
| fungi | not sure, will require some research | 22:27 |
| clarkb | corvus: ^ review.openstack.org:/home/corvus is by far the biggest homedir other than roots on that host | 22:27 |
| clarkb | looks like it may have 11GB of repo copies. | 22:28 |
| clarkb | Starting with cleanup there might be quick and easy win though | 22:28 |
| clarkb | on zuul01 the .bup dir cost is largely in the index caches dir | 22:30 |
| clarkb | and those files were modified as recently as today | 22:30 |
| corvus | clarkb: i can't think of any reason we need old repos on gerrit, i'll clean them up | 22:42 |
| corvus | that looks pretty old | 22:42 |
| clarkb | corvus: thanks! | 22:43 |
| clarkb | I started putting notes at https://etherpad.opendev.org/p/filesystem-spring-cleaning-2020 | 22:43 |
| corvus | i also cleaned up ~gerrit2/jeblair | 22:44 |
| clarkb | corvus: in that etherpad I've noted that the zuul times db isn't tiny and I think a lot of it is stale? | 22:44 |
| clarkb | corvus: its also not huge so probably not the most urgent thing but we might be able to clean that up? | 22:44 |
| clarkb | really /root/.bup is so huge compared to everything else that figuring out if it can be trimmed might be a good next step | 22:45 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Meetpad: redirect 80 to 443 https://review.opendev.org/724199 | 22:45 |
| corvus | clarkb: i agree we should be able to delete that part of the time db; should be safe to do while running. | 22:46 |
| *** tkajinam has joined #opendev | 22:46 | |
| clarkb | the /var/lib/docker dirs aren't tiny but I'm pretty sure thats all our new shiny container image FS's and we are pruning them | 22:48 |
| clarkb | meaning thats just a new cost we have to accept | 22:48 |
| clarkb | mordred: ^ maybe you can double check that on zuul01? | 22:48 |
| clarkb | but I'm fairly certain we've done our due diligence with docker there | 22:49 |
| *** roman_g has quit IRC | 22:52 | |
| clarkb | reading about bup backups I think the way to handle tbe /root/.bup size would be to rotate that directory on the source host when we rotate the archives on the backup host | 22:55 |
| clarkb | basically its growing in concert with the archive to keep track of all the files it has backed up over time | 22:56 |
| clarkb | I bet review and zuul are hit by that more than other hosts beacuse they have lots of files that chagne name/location over time | 23:01 |
| clarkb | infra-root if you can review what I'ev got at https://etherpad.opendev.org/p/filesystem-spring-cleaning-2020 already thatwould be great. I'll try to pick that up again tomorrow when we've got more eyeballs. | 23:03 |
| clarkb | zuul01 is really close with its semi regular size jumps | 23:04 |
| clarkb | but I think we'll be ok for another day | 23:04 |
| clarkb | (and I'm running out of day today hence the pick it up tomorrow) | 23:04 |
| clarkb | corvus: can I cleanup all git.openstack.org times dirs? | 23:05 |
| corvus | clarkb: yeah, those should all be opendev.org now | 23:06 |
| clarkb | corvus: that is what I thought. thanks | 23:07 |
| corvus | clarkb: note that it's in a few different tenants | 23:07 |
| corvus | clarkb: so you can also delete times/zuul/git.openstack.org and times/kata-containers/git.openstack.org | 23:07 |
| clarkb | corvus: yup should I go ahead and do that now? | 23:07 |
| clarkb | that seems reasonably safe and quick | 23:07 |
| corvus | ++ | 23:08 |
| clarkb | k doing so now | 23:08 |
| *** tosky has quit IRC | 23:11 | |
| ianw | clarkb: not related, but i did switch that nodepool ppa key import to come from the fs with https://review.opendev.org/#/c/724462/ | 23:12 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] add plain nodes https://review.opendev.org/724776 | 23:15 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] suse python2 pip support https://review.opendev.org/724777 | 23:15 |
| clarkb | ianw: https://review.opendev.org/#/c/724344/6/Dockerfile modred did similar there but uses apt-key | 23:16 |
| clarkb | for some reason I thought using apt-key was required but your change apsses testing too | 23:16 |
| ianw | yeah the apt-key manual says "put it there to avoid having to have gpg2" :) | 23:20 |
| ianw | the trick is you have to name it ".asc", if it's .gpg it assumes it's a binary keyring ... i added a comment on that change | 23:24 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Configure htcacheclean for zuul-web https://review.opendev.org/724778 | 23:25 |
| clarkb | ianw: oh I remember this is a realtively new feature of apt, but since we are using containers that fine! | 23:26 |
| clarkb | ianw: maybe you can suggest mordred simplify his change? | 23:26 |
| clarkb | oh you did thats what you meant by "that change" | 23:26 |
| clarkb | its clearly EOD for me | 23:26 |
| clarkb | infra-root https://review.opendev.org/724778 is related to my spring fs cleaning | 23:27 |
| clarkb | I think that will help keep disk use on / down for zuul01 | 23:27 |
| clarkb | as a reminder my ISP changes ownership tomorrow. I don't expect any weirdness but seems like possibility for that is higher than usual | 23:32 |
| clarkb | I'll be able to tether if necessary but might be less useful | 23:32 |
| clarkb | and with that I think its tome to call it a day as I'm only just being confused about gpg and apt now | 23:32 |
| *** tkajinam has quit IRC | 23:43 | |
| *** tkajinam has joined #opendev | 23:43 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
