| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Put new borg hosts in cacti https://review.opendev.org/c/opendev/system-config/+/776559 | 00:00 |
|---|---|---|
| fungi | ianw: anything special i need to do to make /usr/local/bin/borg-mount on the wiki server beyond just filling the username into the borg-mount template from system-config and make it executable? where do i find the username? | 00:06 |
| ianw | fungi: umm, no ... the same details as in /usr/local/bin/borg-backup | 00:07 |
| fungi | ahh, okay | 00:07 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update upload-logs-swift and upload-logs-gcs https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 00:09 |
| clarkb | thats annoying I want to docker exec into a container to mv a file using a glob so that I can then docker cp it. But you can't docker exec if the container has stopped and docker cp can't do globs | 00:09 |
| fungi | Warning: Attempting to access a previously unknown unencrypted repository! Do you want to continue? [yN] | 00:10 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update upload-logs roles to support endpoint override https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 00:10 |
| fungi | ianw: i assume that's normal? | 00:10 |
| ianw | fungi: yep, there's some env flags you can set, i did think that the script set them | 00:10 |
| fungi | thanks, seems to have worked | 00:11 |
| ianw | # This avoids UI prompts when first accessing the remote repository | 00:11 |
| ianw | export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=1 | 00:11 |
| ianw | it's in the backup script, but not the borg-mount script | 00:11 |
| fungi | ls in the root of one of the filesystem backups takes a while | 00:12 |
| fungi | can't tab-complete paths in there, but that's not the end of the world | 00:12 |
| fungi | ianw: looks great! | 00:14 |
| fungi | thanks so much for getting that working | 00:14 |
| ianw | np, it's good to feel like it's stored redundantly | 00:14 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-openshift: remove unused role var https://review.opendev.org/c/zuul/zuul-jobs/+/775888 | 00:22 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-zookeeper: add use_tls role var https://review.opendev.org/c/zuul/zuul-jobs/+/776290 | 00:22 |
| fungi | clarkb: i've checked the wiki situation as well as reviewed 766630 for the bup cleanup, if you were waiting on me for looking that over | 00:23 |
| fungi | i'm hesitant to single-core approve it | 00:23 |
| fungi | just because, ya know, backups, precious redundancy | 00:24 |
| ianw | yeah i'd appreciate another eye, but i'm happy to babysit. i'll need to go through after and remove the old cron jobs, and then we can shutdown the servers and decide what to do with them | 00:24 |
| clarkb | ya I was hoping you'd take a look first. Let me see | 00:25 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 00:29 |
| clarkb | ianw: the content of playbooks/roles/backup/tasks/main.yaml has effectively already turned off bup backups right? | 00:31 |
| ianw | clarkb: yeah, for hosts under management https://review.opendev.org/c/opendev/system-config/+/766300 should have applied | 00:34 |
| clarkb | ya I think this change is largely to cleanup things on the config management side but hosts should largely be cleaned up already | 00:34 |
| clarkb | I +2'd given that | 00:34 |
| clarkb | but didn't approve as my day is winding down | 00:34 |
| clarkb | I smell dinner too | 00:34 |
| ianw | thanks, i can watch it in | 00:35 |
| *** mlavalle has quit IRC | 01:13 | |
| *** ysandeep|away is now known as ysandeep|ruck | 02:43 | |
| ianw | kopecmartin / clarkb : refstack held nodes documented at @ https://etherpad.opendev.org/p/refstack-docker | 03:03 |
| ianw | kopecmartin: https://github.com/kopecmartin.keys is blank so i haven't put any keys for you to login, lmn where to get a public key | 03:03 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] gerrit : add mariadb_container option https://review.opendev.org/c/opendev/system-config/+/775961 | 03:10 |
| *** dviroel has quit IRC | 03:35 | |
| *** TheJulia has quit IRC | 04:01 | |
| *** TheJulia has joined #opendev | 04:03 | |
| *** hemanth_n has joined #opendev | 04:29 | |
| *** hemanth_n has quit IRC | 04:42 | |
| *** ykarel has joined #opendev | 04:53 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Use gitea for gerrit gitweb links https://review.opendev.org/c/opendev/system-config/+/723526 | 05:05 |
| *** zoharm has joined #opendev | 05:11 | |
| openstackgerrit | Merged opendev/system-config master: borg-backup: send explicit email on backup failure https://review.opendev.org/c/opendev/system-config/+/775733 | 05:20 |
| openstackgerrit | Merged opendev/system-config master: grafana: update to 7.4.2 https://review.opendev.org/c/opendev/system-config/+/776553 | 06:03 |
| *** marios has joined #opendev | 06:10 | |
| *** iurygregory_ has joined #opendev | 06:26 | |
| *** elod has quit IRC | 06:26 | |
| *** iurygregory has quit IRC | 06:27 | |
| *** elod has joined #opendev | 06:28 | |
| *** elod has quit IRC | 06:34 | |
| *** zbr has quit IRC | 06:44 | |
| *** LowKey has quit IRC | 06:50 | |
| *** zbr has joined #opendev | 06:51 | |
| *** elod has joined #opendev | 06:54 | |
| *** Dmitrii-Sh has quit IRC | 07:04 | |
| *** Dmitrii-Sh has joined #opendev | 07:04 | |
| *** slaweq has joined #opendev | 07:16 | |
| *** sboyron has joined #opendev | 07:26 | |
| *** ralonsoh has joined #opendev | 07:35 | |
| *** eolivare has joined #opendev | 07:38 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:50 | |
| *** rpittau|afk is now known as rpittau | 08:06 | |
| *** hashar has joined #opendev | 08:11 | |
| *** mrunge has quit IRC | 08:12 | |
| *** mrunge has joined #opendev | 08:23 | |
| *** jpena|off is now known as jpena | 08:24 | |
| *** mordred has quit IRC | 08:25 | |
| *** Eighth_Doctor has quit IRC | 08:25 | |
| *** DSpider has joined #opendev | 08:31 | |
| *** andrewbonney has joined #opendev | 08:35 | |
| *** tosky has joined #opendev | 08:40 | |
| *** zbr8 has joined #opendev | 08:48 | |
| *** zbr has quit IRC | 08:49 | |
| *** zbr8 is now known as zbr | 08:49 | |
| *** hashar has quit IRC | 09:01 | |
| *** hashar has joined #opendev | 09:03 | |
| *** lpetrut has joined #opendev | 09:04 | |
| *** ysandeep|lunch is now known as ysandeep|ruck | 09:05 | |
| *** hashar has quit IRC | 09:06 | |
| *** zbr has quit IRC | 09:10 | |
| *** zbr has joined #opendev | 09:11 | |
| *** Eighth_Doctor has joined #opendev | 09:22 | |
| *** mordred has joined #opendev | 09:38 | |
| *** dtantsur|afk is now known as dtantsur | 09:42 | |
| *** zbr has quit IRC | 09:46 | |
| *** brinzhang has joined #opendev | 09:46 | |
| *** ssbarnea_ has joined #opendev | 09:52 | |
| *** zbr has joined #opendev | 09:53 | |
| *** ssbarnea_ has quit IRC | 09:54 | |
| *** hashar has joined #opendev | 10:30 | |
| *** LowKey has joined #opendev | 10:31 | |
| *** dviroel has joined #opendev | 10:50 | |
| *** dtantsur is now known as dtantsur|brb | 10:55 | |
| *** tkajinam has quit IRC | 11:25 | |
| *** marios is now known as marios|food | 11:41 | |
| *** zbr0 has joined #opendev | 11:49 | |
| *** zbr has quit IRC | 11:52 | |
| *** zbr0 is now known as zbr | 11:52 | |
| *** marios|food is now known as marios | 12:06 | |
| *** rpittau has quit IRC | 12:08 | |
| *** johnsom has quit IRC | 12:08 | |
| *** TheJulia has quit IRC | 12:08 | |
| *** LowKey has quit IRC | 12:09 | |
| *** TheJulia has joined #opendev | 12:18 | |
| *** rpittau has joined #opendev | 12:21 | |
| *** brinzhang has quit IRC | 12:22 | |
| *** johnsom has joined #opendev | 12:23 | |
| *** SWAT has quit IRC | 12:25 | |
| *** SWAT_ has joined #opendev | 12:25 | |
| *** dtantsur|brb is now known as dtantsur | 12:29 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** sshnaidm is now known as sshnaidm|off | 12:36 | |
| *** iurygregory_ is now known as iurygregory | 12:49 | |
| *** mlavalle has joined #opendev | 12:54 | |
| *** knikolla has quit IRC | 13:06 | |
| *** knikolla has joined #opendev | 13:07 | |
| kopecmartin | ianw: i've uploaded the key to my github account .. so jobs here https://review.opendev.org/c/opendev/system-config/+/776292 rebuild the page (https://refstack01.openstack.org/#/)? how do i make it to rebuild the container too? I merged a few fixes in refstack | 13:19 |
| *** ssbarnea_ has joined #opendev | 13:20 | |
| *** zbr7 has joined #opendev | 13:21 | |
| *** zbr has quit IRC | 13:22 | |
| *** zbr7 is now known as zbr | 13:22 | |
| *** zbr1 has joined #opendev | 13:31 | |
| *** zbr has quit IRC | 13:33 | |
| *** zbr1 is now known as zbr | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| *** ssbarnea_ has quit IRC | 13:34 | |
| frickler | ianw: clarkb: ^^ not sure how to best give access to those held nodes, since these aren't the usual test nodes with root login, but set up with our individual infra-root accounts | 13:35 |
| fungi | frickler: as long as root login isn't disabled you could still add the key to /root/.ssh/authorized_keys | 13:41 |
| fungi | alternatively adduser an account and put it in whatever group we're using to control full root access with sudo these days | 13:43 |
| frickler | hmm, let's stick to the root login, then, to keep it kind of consistent, I guess | 13:48 |
| openstackgerrit | Martin Chacon Piza proposed openstack/project-config master: Deprecate monasca-log-api https://review.opendev.org/c/openstack/project-config/+/776641 | 13:51 |
| frickler | except that it doesn't seem to work. | 13:53 |
| *** zbr9 has joined #opendev | 13:53 | |
| * frickler has a meeting now, will try later unless someone else takes over | 13:54 | |
| *** zbr has quit IRC | 13:56 | |
| *** zbr9 is now known as zbr | 13:56 | |
| openstackgerrit | Martin Chacon Piza proposed openstack/project-config master: Deprecate monasca-ceilometer https://review.opendev.org/c/openstack/project-config/+/776644 | 14:03 |
| fungi | kopecmartin: see if you can ssh into those held servers as kopecmartin now | 14:10 |
| *** ykarel_ has joined #opendev | 14:15 | |
| *** ykarel has quit IRC | 14:17 | |
| *** zbr7 has joined #opendev | 14:22 | |
| *** zbr has quit IRC | 14:25 | |
| *** zbr7 is now known as zbr | 14:25 | |
| *** hashar has quit IRC | 14:29 | |
| *** zbr3 has joined #opendev | 14:47 | |
| *** zbr has quit IRC | 14:49 | |
| *** zbr3 is now known as zbr | 14:49 | |
| *** ykarel_ is now known as ykarel | 14:53 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: multi-node debian: Update package lists before installing https://review.opendev.org/c/zuul/zuul-jobs/+/775866 | 14:53 |
| *** rpittau is now known as rpittau|afk | 15:07 | |
| *** hashar has joined #opendev | 15:10 | |
| *** lpetrut has quit IRC | 15:15 | |
| *** ysandeep|ruck is now known as ysandeep|away | 15:19 | |
| *** fressi has quit IRC | 15:23 | |
| kopecmartin | fungi: i can, thanks | 15:27 |
| *** icey has quit IRC | 15:32 | |
| *** icey has joined #opendev | 15:33 | |
| clarkb | frickler: fungi: I don't think we remove the zuul user either which should remain an option? | 15:40 |
| clarkb | but adding a new user is fine too | 15:40 |
| *** zbr9 has joined #opendev | 15:41 | |
| *** zbr has quit IRC | 15:43 | |
| *** zbr9 is now known as zbr | 15:43 | |
| fungi | yeah, i thought about that but was too lazy to check whether the zuul user still had sudo perms | 15:51 |
| fungi | probably did | 15:51 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 15:51 |
| *** zbr6 has joined #opendev | 16:01 | |
| *** zbr6 has quit IRC | 16:02 | |
| *** zbr9 has joined #opendev | 16:03 | |
| *** zbr has quit IRC | 16:04 | |
| *** zbr9 is now known as zbr | 16:04 | |
| openstackgerrit | Martin Chacon Piza proposed openstack/project-config master: Deprecate monasca-ceilometer https://review.opendev.org/c/openstack/project-config/+/776644 | 16:07 |
| openstackgerrit | Martin Chacon Piza proposed openstack/project-config master: Deprecate monasca-log-api https://review.opendev.org/c/openstack/project-config/+/776641 | 16:09 |
| *** mordred has quit IRC | 16:12 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Update upload-logs roles to support endpoint override https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 16:12 |
| *** mordred has joined #opendev | 16:13 | |
| openstackgerrit | Martin Chacon Piza proposed openstack/project-config master: Deprecate monasca-log-api https://review.opendev.org/c/openstack/project-config/+/776641 | 16:29 |
| openstackgerrit | Martin Chacon Piza proposed openstack/project-config master: Deprecate monasca-ceilometer https://review.opendev.org/c/openstack/project-config/+/776644 | 16:31 |
| *** eolivare has quit IRC | 16:40 | |
| *** hashar has quit IRC | 16:41 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update GCS log upload to new google auth api https://review.opendev.org/c/zuul/zuul-jobs/+/776671 | 16:44 |
| openstackgerrit | Martin Chacon Piza proposed openstack/project-config master: Deprecate monasca-ceilometer https://review.opendev.org/c/openstack/project-config/+/776644 | 16:45 |
| *** marios is now known as marios|out | 16:46 | |
| openstackgerrit | Martin Chacon Piza proposed openstack/project-config master: Deprecate monasca-log-api https://review.opendev.org/c/openstack/project-config/+/776641 | 16:47 |
| *** marios|out has quit IRC | 16:54 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: GCS logs: more robust Credential class https://review.opendev.org/c/zuul/zuul-jobs/+/776675 | 17:02 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 17:02 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Revert "Update upload-logs roles to support endpoint override" https://review.opendev.org/c/zuul/zuul-jobs/+/776676 | 17:08 |
| corvus | we will need to force-merge that; i'll do that | 17:09 |
| *** LowKey has joined #opendev | 17:10 | |
| *** ralonsoh_ has joined #opendev | 17:12 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Update upload-logs roles to support endpoint override" https://review.opendev.org/c/zuul/zuul-jobs/+/776676 | 17:12 |
| *** ysandeep|away is now known as ysandeep | 17:13 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 17:13 |
| *** ykarel has quit IRC | 17:14 | |
| clarkb | how does this look #status log All jobs are failing with POST_FAILURE due to a backward incompatible change made in the swift log upload libarary role. Working on a fix now. | 17:15 |
| clarkb | s/logs/notice/ probably | 17:15 |
| fungi | lgtm | 17:15 |
| fungi | notice yeah | 17:15 |
| clarkb | #status notice All jobs are failing with POST_FAILURE due to a backward incompatible change made in the swift log upload libarary role. Working on a fix now. | 17:15 |
| openstackstatus | clarkb: sending notice | 17:15 |
| clarkb | hrm bot not here? | 17:15 |
| -openstackstatus- NOTICE: All jobs are failing with POST_FAILURE due to a backward incompatible change made in the swift log upload libarary role. Working on a fix now. | 17:16 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Revert "Revert "Update upload-logs roles to support endpoint override"" https://review.opendev.org/c/zuul/zuul-jobs/+/776677 | 17:16 |
| clarkb | corvus: thanks | 17:18 |
| *** ralonsoh has quit IRC | 17:18 | |
| *** lourot has quit IRC | 17:18 | |
| *** auristor has quit IRC | 17:18 | |
| openstackstatus | clarkb: finished sending notice | 17:19 |
| *** ralonsoh has joined #opendev | 17:21 | |
| *** lourot has joined #opendev | 17:21 | |
| *** auristor has joined #opendev | 17:21 | |
| *** ralonsoh has quit IRC | 17:22 | |
| *** auristor has quit IRC | 17:22 | |
| *** ysandeep is now known as ysandeep|away | 17:23 | |
| *** mordred has quit IRC | 17:26 | |
| *** Eighth_Doctor has quit IRC | 17:26 | |
| *** jpena is now known as jpena|off | 17:27 | |
| *** auristor has joined #opendev | 17:28 | |
| *** mordred has joined #opendev | 17:40 | |
| *** zbr1 has joined #opendev | 17:40 | |
| *** zbr has quit IRC | 17:42 | |
| *** zbr1 is now known as zbr | 17:42 | |
| *** Eighth_Doctor has joined #opendev | 17:44 | |
| fungi | affected builds will be those started between 16:12 and 17:13 utc, looks like | 17:48 |
| clarkb | how about this now #status notice The change to the upload role has been reverted. Jobs started since the revert appear to be functioning normally. You can recheck changes that failed with POST_FAILURE safely now. | 17:49 |
| fungi | we might include the affected timeframe, just so if there are new failures people don't assume they're part of the same event | 17:49 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: GCS logs: more robust Credential class https://review.opendev.org/c/zuul/zuul-jobs/+/776675 | 17:50 |
| clarkb | fungi: my only concern with that is people will have a hard time grokking its when the job started | 17:50 |
| fungi | i usually say "builds started between..." but yes it's a bit of a nuance | 17:51 |
| clarkb | what about #status notice The change to the upload role has been reverted. Jobs started since the revert appear to be functioning normally. You can recheck changes that failed with jobs started between 16:12 and 17:13UTC reporting POST_FAILURE safely now. | 17:51 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: GCS logs: more robust Credential class https://review.opendev.org/c/zuul/zuul-jobs/+/776675 | 17:53 |
| fungi | clarkb: "failed for builds started between" might be more accurate/readable? | 17:54 |
| clarkb | like this #status notice The change to the upload role has been reverted. Jobs started since the revert appear to be functioning normally. You can recheck changes that failed for builds started between 16:12 and 17:13UTC reporting POST_FAILURE safely now. | 17:55 |
| fungi | lgtm, thanks! | 17:55 |
| clarkb | #status notice The change to the upload role has been reverted. Jobs started since the revert appear to be functioning normally. You can recheck changes that failed for builds started between 16:12 and 17:13UTC reporting POST_FAILURE safely now. | 17:56 |
| openstackstatus | clarkb: sending notice | 17:56 |
| -openstackstatus- NOTICE: The change to the upload role has been reverted. Jobs started since the revert appear to be functioning normally. You can recheck changes that failed for builds started between 16:12 and 17:13UTC reporting POST_FAILURE safely now. | 17:56 | |
| openstackstatus | clarkb: finished sending notice | 17:59 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 18:00 |
| *** andrewbonney has quit IRC | 18:04 | |
| *** zbr7 has joined #opendev | 18:05 | |
| *** zbr has quit IRC | 18:07 | |
| *** zbr7 is now known as zbr | 18:07 | |
| corvus | just noticed our node counter is at 23 million since the last time it was reset | 18:07 |
| clarkb | https://i.imgflip.com/4yox4d.jpg | 18:10 |
| *** dtantsur is now known as dtantsur|afk | 18:10 | |
| *** hashar has joined #opendev | 18:11 | |
| openstackgerrit | Merged zuul/zuul-jobs master: GCS logs: more robust Credential class https://review.opendev.org/c/zuul/zuul-jobs/+/776675 | 18:12 |
| *** zoharm has quit IRC | 18:14 | |
| *** mlavalle has quit IRC | 18:18 | |
| *** mlavalle has joined #opendev | 18:20 | |
| fungi | #infra-root: http://travaux.ovh.net/?do=details&id=48997 is marked as closed as of yesterday, and connectivity still looks fine, so i'm approving 775209 to readd ovh bhs log uploads | 18:30 |
| clarkb | fungi: ++ | 18:31 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: ensure-zookeeper: use a tmpfs https://review.opendev.org/c/zuul/zuul-jobs/+/776696 | 18:40 |
| openstackgerrit | Merged opendev/base-jobs master: Revert "Temporarily remove storage.bhs.cloud.ovh.net" https://review.opendev.org/c/opendev/base-jobs/+/775209 | 18:43 |
| *** zbr2 has joined #opendev | 18:43 | |
| *** zbr has quit IRC | 18:46 | |
| *** zbr2 is now known as zbr | 18:46 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: ensure-zookeeper: use a tmpfs https://review.opendev.org/c/zuul/zuul-jobs/+/776696 | 18:52 |
| openstackgerrit | Merged openstack/project-config master: Deprecate monasca-log-api https://review.opendev.org/c/openstack/project-config/+/776641 | 19:06 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Build some new python base images https://review.opendev.org/c/opendev/system-config/+/776699 | 19:11 |
| clarkb | fungi: corvus ^ fyi | 19:11 |
| openstackgerrit | Merged openstack/project-config master: Deprecate monasca-ceilometer https://review.opendev.org/c/openstack/project-config/+/776644 | 19:12 |
| *** gmann is now known as gmann_afk | 19:19 | |
| *** Eighth_Doctor has quit IRC | 19:36 | |
| *** auristor has quit IRC | 19:36 | |
| *** mrunge has quit IRC | 19:36 | |
| *** mordred has quit IRC | 19:36 | |
| *** ttx has quit IRC | 19:36 | |
| *** otherwiseguy has quit IRC | 19:36 | |
| *** ysandeep|away has quit IRC | 19:36 | |
| *** jhesketh has quit IRC | 19:36 | |
| *** yoctozepto has quit IRC | 19:36 | |
| *** lourot has quit IRC | 19:36 | |
| *** SWAT_ has quit IRC | 19:36 | |
| *** slaweq has quit IRC | 19:36 | |
| *** dmsimard has quit IRC | 19:36 | |
| *** cloudnull has quit IRC | 19:36 | |
| *** arxcruz has quit IRC | 19:36 | |
| *** zimmerry has quit IRC | 19:36 | |
| *** avass has quit IRC | 19:36 | |
| *** stevebaker has quit IRC | 19:36 | |
| *** gouthamr has quit IRC | 19:36 | |
| *** sshnaidm|off has quit IRC | 19:36 | |
| *** JohnnyRa1 has quit IRC | 19:36 | |
| *** mtreinish has quit IRC | 19:36 | |
| *** openstackgerrit has quit IRC | 19:36 | |
| *** snbuback has quit IRC | 19:36 | |
| *** gmann_afk has quit IRC | 19:36 | |
| *** portdirect has quit IRC | 19:36 | |
| *** rm_work has quit IRC | 19:36 | |
| *** Open10K8S has quit IRC | 19:36 | |
| *** artom has quit IRC | 19:36 | |
| *** parallax has quit IRC | 19:36 | |
| *** zaro has quit IRC | 19:36 | |
| *** aprice has quit IRC | 19:36 | |
| *** jmorgan has quit IRC | 19:36 | |
| *** mattmceuen has quit IRC | 19:36 | |
| *** Tengu has quit IRC | 19:36 | |
| *** corvus has quit IRC | 19:36 | |
| *** zigo has quit IRC | 19:36 | |
| *** stephenfin has quit IRC | 19:36 | |
| *** fdegir has quit IRC | 19:36 | |
| *** mnasiadka has quit IRC | 19:36 | |
| *** sgw has quit IRC | 19:36 | |
| *** guillaumec has quit IRC | 19:36 | |
| *** tbarron|out has quit IRC | 19:36 | |
| *** redrobot has quit IRC | 19:36 | |
| *** mhu has quit IRC | 19:36 | |
| *** noonedeadpunk has quit IRC | 19:36 | |
| *** jrosser has quit IRC | 19:36 | |
| *** hillpd has quit IRC | 19:36 | |
| *** cgoncalves has quit IRC | 19:36 | |
| *** dpawlik has quit IRC | 19:36 | |
| *** owalsh has quit IRC | 19:36 | |
| *** fbo has quit IRC | 19:36 | |
| *** slittle1 has quit IRC | 19:36 | |
| *** SotK has quit IRC | 19:36 | |
| *** clayg has quit IRC | 19:36 | |
| *** mnaser has quit IRC | 19:36 | |
| *** mugsie has quit IRC | 19:36 | |
| *** paladox has quit IRC | 19:36 | |
| *** CeeMac has quit IRC | 19:36 | |
| *** osmanlicilegi has quit IRC | 19:36 | |
| *** erbarr has quit IRC | 19:36 | |
| *** persia has quit IRC | 19:36 | |
| *** melwitt has quit IRC | 19:36 | |
| *** dhellmann has quit IRC | 19:36 | |
| *** valleedelisle has quit IRC | 19:36 | |
| *** knikolla has quit IRC | 19:36 | |
| *** johnsom has quit IRC | 19:36 | |
| *** TheJulia has quit IRC | 19:36 | |
| *** dviroel has quit IRC | 19:36 | |
| *** ildikov has quit IRC | 19:36 | |
| *** bbezak has quit IRC | 19:36 | |
| *** seongsoocho has quit IRC | 19:36 | |
| *** jroll has quit IRC | 19:36 | |
| *** mlavalle has quit IRC | 19:36 | |
| *** DSpider has quit IRC | 19:36 | |
| *** logan- has quit IRC | 19:36 | |
| *** Jeffrey4l has quit IRC | 19:36 | |
| *** ChanServ has quit IRC | 19:36 | |
| *** ralonsoh_ has quit IRC | 19:36 | |
| *** LowKey has quit IRC | 19:36 | |
| *** icey has quit IRC | 19:36 | |
| *** Dmitrii-Sh has quit IRC | 19:36 | |
| *** smcginnis has quit IRC | 19:36 | |
| *** kevinz has quit IRC | 19:36 | |
| *** hashar has quit IRC | 19:36 | |
| *** rpittau|afk has quit IRC | 19:36 | |
| *** iurygregory has quit IRC | 19:36 | |
| *** clarkb has quit IRC | 19:36 | |
| *** walshh_ has quit IRC | 19:36 | |
| *** jpena|off has quit IRC | 19:36 | |
| *** swest has quit IRC | 19:36 | |
| *** dtantsur|afk has quit IRC | 19:36 | |
| *** fungi has quit IRC | 19:36 | |
| *** zbr has quit IRC | 19:36 | |
| *** ianw has quit IRC | 19:36 | |
| *** gnuoy has quit IRC | 19:36 | |
| *** lbragstad has quit IRC | 19:36 | |
| *** calcmandan has quit IRC | 19:36 | |
| *** guilhermesp has quit IRC | 19:36 | |
| *** bodgix has quit IRC | 19:36 | |
| *** kopecmartin has quit IRC | 19:36 | |
| *** tristanC has quit IRC | 19:36 | |
| *** amotoki has quit IRC | 19:36 | |
| *** ozzzo has quit IRC | 19:36 | |
| *** dries has quit IRC | 19:36 | |
| *** odyssey4me has quit IRC | 19:36 | |
| *** janders has quit IRC | 19:36 | |
| *** sboyron has quit IRC | 19:36 | |
| *** priteau has quit IRC | 19:36 | |
| *** jaicaa has quit IRC | 19:36 | |
| *** mwhahaha has quit IRC | 19:36 | |
| *** akahat has quit IRC | 19:36 | |
| *** donnyd has quit IRC | 19:36 | |
| *** chrome0 has quit IRC | 19:36 | |
| *** frickler has quit IRC | 19:36 | |
| *** felixedel has quit IRC | 19:36 | |
| *** tosky has quit IRC | 19:36 | |
| *** jbryce has quit IRC | 19:36 | |
| *** ShadowJonathan has quit IRC | 19:36 | |
| *** artom has joined #opendev | 19:40 | |
| *** zbr has joined #opendev | 19:40 | |
| *** mlavalle has joined #opendev | 19:40 | |
| *** hashar has joined #opendev | 19:40 | |
| *** Eighth_Doctor has joined #opendev | 19:40 | |
| *** mordred has joined #opendev | 19:40 | |
| *** auristor has joined #opendev | 19:40 | |
| *** lourot has joined #opendev | 19:40 | |
| *** ralonsoh_ has joined #opendev | 19:40 | |
| *** LowKey has joined #opendev | 19:40 | |
| *** icey has joined #opendev | 19:40 | |
| *** knikolla has joined #opendev | 19:40 | |
| *** SWAT_ has joined #opendev | 19:40 | |
| *** johnsom has joined #opendev | 19:40 | |
| *** rpittau|afk has joined #opendev | 19:40 | |
| *** TheJulia has joined #opendev | 19:40 | |
| *** dviroel has joined #opendev | 19:40 | |
| *** tosky has joined #opendev | 19:40 | |
| *** DSpider has joined #opendev | 19:40 | |
| *** mrunge has joined #opendev | 19:40 | |
| *** sboyron has joined #opendev | 19:40 | |
| *** slaweq has joined #opendev | 19:40 | |
| *** Dmitrii-Sh has joined #opendev | 19:40 | |
| *** iurygregory has joined #opendev | 19:40 | |
| *** tbarron|out has joined #opendev | 19:40 | |
| *** priteau has joined #opendev | 19:40 | |
| *** gouthamr has joined #opendev | 19:40 | |
| *** ianw has joined #opendev | 19:40 | |
| *** clayg has joined #opendev | 19:40 | |
| *** dmsimard has joined #opendev | 19:40 | |
| *** jhesketh has joined #opendev | 19:40 | |
| *** cloudnull has joined #opendev | 19:40 | |
| *** sshnaidm|off has joined #opendev | 19:40 | |
| *** mnaser has joined #opendev | 19:40 | |
| *** arxcruz has joined #opendev | 19:40 | |
| *** redrobot has joined #opendev | 19:40 | |
| *** mugsie has joined #opendev | 19:40 | |
| *** JohnnyRa1 has joined #opendev | 19:40 | |
| *** mhu has joined #opendev | 19:40 | |
| *** mtreinish has joined #opendev | 19:40 | |
| *** noonedeadpunk has joined #opendev | 19:40 | |
| *** clarkb has joined #opendev | 19:40 | |
| *** zimmerry has joined #opendev | 19:40 | |
| *** openstackgerrit has joined #opendev | 19:40 | |
| *** dhellmann has joined #opendev | 19:40 | |
| *** avass has joined #opendev | 19:40 | |
| *** gnuoy has joined #opendev | 19:40 | |
| *** ttx has joined #opendev | 19:40 | |
| *** logan- has joined #opendev | 19:40 | |
| *** ysandeep|away has joined #opendev | 19:40 | |
| *** otherwiseguy has joined #opendev | 19:40 | |
| *** walshh_ has joined #opendev | 19:40 | |
| *** stevebaker has joined #opendev | 19:40 | |
| *** Open10K8S has joined #opendev | 19:40 | |
| *** rm_work has joined #opendev | 19:40 | |
| *** portdirect has joined #opendev | 19:40 | |
| *** gmann_afk has joined #opendev | 19:40 | |
| *** snbuback has joined #opendev | 19:40 | |
| *** paladox has joined #opendev | 19:40 | |
| *** lbragstad has joined #opendev | 19:40 | |
| *** ildikov has joined #opendev | 19:40 | |
| *** felixedel has joined #opendev | 19:40 | |
| *** swest has joined #opendev | 19:40 | |
| *** jaicaa has joined #opendev | 19:40 | |
| *** CeeMac has joined #opendev | 19:40 | |
| *** calcmandan has joined #opendev | 19:40 | |
| *** guilhermesp has joined #opendev | 19:40 | |
| *** parallax has joined #opendev | 19:40 | |
| *** mwhahaha has joined #opendev | 19:40 | |
| *** jrosser has joined #opendev | 19:40 | |
| *** zaro has joined #opendev | 19:40 | |
| *** aprice has joined #opendev | 19:40 | |
| *** jmorgan has joined #opendev | 19:40 | |
| *** mattmceuen has joined #opendev | 19:40 | |
| *** hillpd has joined #opendev | 19:40 | |
| *** Tengu has joined #opendev | 19:40 | |
| *** corvus has joined #opendev | 19:40 | |
| *** bodgix has joined #opendev | 19:40 | |
| *** cgoncalves has joined #opendev | 19:40 | |
| *** kopecmartin has joined #opendev | 19:40 | |
| *** zigo has joined #opendev | 19:40 | |
| *** dries has joined #opendev | 19:40 | |
| *** smcginnis has joined #opendev | 19:40 | |
| *** stephenfin has joined #opendev | 19:40 | |
| *** Jeffrey4l has joined #opendev | 19:40 | |
| *** jpena|off has joined #opendev | 19:40 | |
| *** akahat has joined #opendev | 19:40 | |
| *** dpawlik has joined #opendev | 19:40 | |
| *** fdegir has joined #opendev | 19:40 | |
| *** osmanlicilegi has joined #opendev | 19:40 | |
| *** yoctozepto has joined #opendev | 19:40 | |
| *** mnasiadka has joined #opendev | 19:40 | |
| *** bbezak has joined #opendev | 19:40 | |
| *** donnyd has joined #opendev | 19:40 | |
| *** seongsoocho has joined #opendev | 19:40 | |
| *** erbarr has joined #opendev | 19:40 | |
| *** jbryce has joined #opendev | 19:40 | |
| *** ShadowJonathan has joined #opendev | 19:40 | |
| *** sgw has joined #opendev | 19:40 | |
| *** owalsh has joined #opendev | 19:40 | |
| *** fbo has joined #opendev | 19:40 | |
| *** tristanC has joined #opendev | 19:40 | |
| *** slittle1 has joined #opendev | 19:40 | |
| *** guillaumec has joined #opendev | 19:40 | |
| *** janders has joined #opendev | 19:40 | |
| *** fungi has joined #opendev | 19:40 | |
| *** ChanServ has joined #opendev | 19:40 | |
| *** frickler has joined #opendev | 19:40 | |
| *** odyssey4me has joined #opendev | 19:40 | |
| *** dtantsur|afk has joined #opendev | 19:40 | |
| *** jroll has joined #opendev | 19:40 | |
| *** valleedelisle has joined #opendev | 19:40 | |
| *** ozzzo has joined #opendev | 19:40 | |
| *** kevinz has joined #opendev | 19:40 | |
| *** amotoki has joined #opendev | 19:40 | |
| *** persia has joined #opendev | 19:40 | |
| *** melwitt has joined #opendev | 19:40 | |
| *** SotK has joined #opendev | 19:40 | |
| *** chrome0 has joined #opendev | 19:40 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 19:40 | |
| *** zbr7 has joined #opendev | 19:44 | |
| *** zbr has quit IRC | 19:46 | |
| *** zbr7 is now known as zbr | 19:46 | |
| *** slaweq has quit IRC | 19:51 | |
| *** hamalq has joined #opendev | 19:58 | |
| *** ralonsoh_ has quit IRC | 20:06 | |
| *** zbr0 has joined #opendev | 20:08 | |
| *** zbr has quit IRC | 20:09 | |
| *** zbr0 is now known as zbr | 20:09 | |
| *** LowKey has quit IRC | 20:11 | |
| *** LowKey has joined #opendev | 20:11 | |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Test/assert Python 3.9 support https://review.opendev.org/c/opendev/git-review/+/772589 | 20:19 |
| clarkb | https://81b29ba4a4458a00ed76-e806954d4dee75658447576ad07d9921.ssl.cf5.rackcdn.com/775883/16/check/system-config-run-review-3.2/097fe24/review01.openstack.org/gatling-report/replayrecordsscenario-20210219180543714/req_push--http---lo-3452e.html finally | 20:27 |
| *** zbr7 has joined #opendev | 20:28 | |
| clarkb | https://ebfd4ec54dbc92b01f1e-dbc96e30a25abd79ad83d28f747c4e2b.ssl.cf2.rackcdn.com/775883/16/check/system-config-run-review-3.3/a59d274/review01.openstack.org/gatling-report/replayrecordsscenario-20210219180157432/req_push--http---lo-3452e.html and that is 3.3 | 20:30 |
| *** zbr has quit IRC | 20:30 | |
| *** zbr7 is now known as zbr | 20:30 | |
| clarkb | with that dataset of 1 it shows that 3.2 handles pushes more slowly than 3.3 | 20:30 |
| clarkb | finally starting to get some sort of useful info out of this | 20:30 |
| clarkb | next up is to add in ssh based interactions | 20:30 |
| fungi | so 3.3 really does seem to perform better than 3.2? excellent news if so | 20:31 |
| fungi | at least with insufficiently small sample sizes ;) | 20:31 |
| clarkb | "yes" :) I think the 3.3 uses more memory obversation continues to hold too though I haven't checked on these runs | 20:31 |
| clarkb | so we may end up under even more memory pressure but then perform slightly better when pushign code | 20:32 |
| clarkb | note the average push time is 2154ms on 3.2 and 1820ms on 3.3 in those examples with 50th percentiles of 2067 and 1744 | 20:32 |
| clarkb | this may not represent a drastic improvement but it does at least seem to point to an improvement. This is all over http though and I'd like to have ssh numbers before reading into this too much | 20:33 |
| openstackgerrit | Merged opendev/system-config master: Build some new python base images https://review.opendev.org/c/opendev/system-config/+/776699 | 20:40 |
| *** zimmerry has quit IRC | 20:40 | |
| *** zimmerry has joined #opendev | 20:40 | |
| *** zbr9 has joined #opendev | 20:41 | |
| fungi | clarkb: ^ | 20:42 |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Test/assert Python 3.9 support https://review.opendev.org/c/opendev/git-review/+/772589 | 20:42 |
| fungi | i guess we can rebuild whatever else we want on those new python base images now | 20:43 |
| *** zbr has quit IRC | 20:43 | |
| *** zbr9 is now known as zbr | 20:43 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 20:47 |
| clarkb | that attempts to add ssh operations | 20:47 |
| corvus | clarkb: apparently nasa is going to release video recordings with sound from the edl phase | 21:13 |
| clarkb | nice did you see https://twitter.com/NASAPersevere/status/1362839907824136193/photo/1 ? | 21:14 |
| clarkb | they caught it landing from space too | 21:14 |
| clarkb | also https://twitter.com/NASAPersevere/status/1362825545227018240/photo/1 | 21:15 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-zookeeper: use a tmpfs https://review.opendev.org/c/zuul/zuul-jobs/+/776696 | 21:15 |
| corvus | ++ | 21:16 |
| *** diablo_rojo_phon has joined #opendev | 21:22 | |
| *** zbr3 has joined #opendev | 21:27 | |
| *** zbr has quit IRC | 21:29 | |
| *** zbr3 is now known as zbr | 21:29 | |
| *** zbr6 has joined #opendev | 21:35 | |
| *** zbr has quit IRC | 21:38 | |
| *** zbr6 is now known as zbr | 21:38 | |
| *** hashar has quit IRC | 22:05 | |
| *** zbr7 has joined #opendev | 22:10 | |
| *** zbr has quit IRC | 22:13 | |
| *** zbr7 is now known as zbr | 22:13 | |
| *** zbr1 has joined #opendev | 22:24 | |
| *** zbr has quit IRC | 22:27 | |
| *** zbr1 is now known as zbr | 22:27 | |
| *** artom has quit IRC | 22:30 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Use gitea for gerrit gitweb links https://review.opendev.org/c/opendev/system-config/+/723526 | 22:38 |
| *** sboyron has quit IRC | 22:41 | |
| *** zbr7 has joined #opendev | 22:49 | |
| *** zbr has quit IRC | 22:51 | |
| *** zbr7 is now known as zbr | 22:51 | |
| mordred | ooh | 22:51 |
| mordred | ianw: that's exciting | 22:52 |
| *** zbr1 has joined #opendev | 22:54 | |
| *** zbr has quit IRC | 22:55 | |
| *** zbr1 is now known as zbr | 22:55 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Try to make gatling-git work with our test gerrit https://review.opendev.org/c/opendev/system-config/+/775883 | 23:11 |
| *** zbr5 has joined #opendev | 23:19 | |
| *** gmann_afk is now known as gmann | 23:21 | |
| *** zbr has quit IRC | 23:21 | |
| *** zbr5 is now known as zbr | 23:21 | |
| *** zbr2 has joined #opendev | 23:23 | |
| *** zbr has quit IRC | 23:25 | |
| *** zbr2 is now known as zbr | 23:25 | |
| corvus | clarkb, fungi: hows about we restart the entire zuul/nodepool system? | 23:31 |
| clarkb | ++ | 23:31 |
| clarkb | we could do a representative subset first if we're worried about any regressions in the tls requirement changes | 23:31 |
| corvus | i'm not worried | 23:31 |
| clarkb | but also its firday and it is tested so just going for it seems fine | 23:32 |
| corvus | where do our image pulls happen? | 23:33 |
| corvus | are they only periodic, or do we put them in the start script? | 23:33 |
| corvus | i think they are only periodic.... | 23:33 |
| clarkb | corvus: they are periodic but hourly not daily | 23:33 |
| clarkb | the opendev-prod-hourly pipeline has jobs that do that for zuul and nodepool iirc | 23:34 |
| corvus | so we either need to validate that we have all the latest images, or incorporate pulls into the restart | 23:34 |
| clarkb | looking at nl01 I don't think it has the new image | 23:35 |
| clarkb | ze01 however does seem to have it | 23:35 |
| clarkb | (problems with nodepool updates maybe?) | 23:35 |
| corvus | or just happened later? | 23:35 |
| clarkb | oh ya could be that we didn't trigger at 2300 because jobs were still running from 2200 | 23:36 |
| clarkb | or something like that | 23:36 |
| clarkb | incorporating a quick docker-compose pull first seems like a reasonable thing to do | 23:36 |
| corvus | right. s/quick// | 23:36 |
| corvus | this is spread out across like 6 roles | 23:36 |
| corvus | i'll work on a change | 23:37 |
| clarkb | ah ya and the dirs are all different names | 23:37 |
| fungi | sorry, stepped away for a moment | 23:38 |
| fungi | back now and caught up | 23:39 |
| fungi | seems like a fine time for restarts | 23:39 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add pull tasks for nodepool/zuul https://review.opendev.org/c/opendev/system-config/+/776720 | 23:43 |
| *** zbr1 has joined #opendev | 23:43 | |
| corvus | clarkb, fungi: ^ how's that look? | 23:44 |
| *** zbr has quit IRC | 23:45 | |
| *** zbr1 is now known as zbr | 23:45 | |
| clarkb | that looks about right | 23:45 |
| corvus | okay i have staged that on my working dir on bridge | 23:50 |
| *** hamalq has quit IRC | 23:50 | |
| corvus | i'll start the sequence now | 23:51 |
| *** zbr8 has joined #opendev | 23:52 | |
| corvus | all stopped | 23:53 |
| clarkb | hrm nl01 still reports a 3 day old image | 23:54 |
| clarkb | but I would've expected a brand new one from the tls required | 23:54 |
| corvus | i'm only just now running the pull | 23:54 |
| *** zbr has quit IRC | 23:54 | |
| *** zbr8 is now known as zbr | 23:54 | |
| clarkb | oh I see | 23:54 |
| clarkb | you stopped first | 23:54 |
| corvus | try again? | 23:54 |
| clarkb | yup its updated now | 23:54 |
| corvus | yeah, i didn't choose the most efficient sequence :/ | 23:54 |
| corvus | pulls done | 23:54 |
| corvus | starts done, zuul is starting up | 23:55 |
| fungi | pull extraction lgtm | 23:56 |
| clarkb | I'm watching the debug log on nl01, mostly out of curioustiy but that should indicate if any of the changes were unhappy | 23:56 |
| clarkb | (currently it is just saying there are no active requests, that should change when zuul is up again) | 23:57 |
| corvus | if it got that far it should be good | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
