| *** gothicserpent_ has joined #opendev | 00:17 | |
| *** gothicserpent has quit IRC | 00:18 | |
| *** gothicserpent_ has quit IRC | 00:19 | |
| *** gothicserpent has joined #opendev | 00:20 | |
| *** tosky has quit IRC | 00:42 | |
| *** ykarel has joined #opendev | 03:17 | |
| *** prometheanfire has quit IRC | 04:07 | |
| *** prometheanfire has joined #opendev | 04:07 | |
| *** marios has joined #opendev | 05:55 | |
| *** ykarel_ has joined #opendev | 06:36 | |
| *** ykarel has quit IRC | 06:38 | |
| openstackgerrit | xinliang proposed openstack/diskimage-builder master: Add aarch64 support for rhel https://review.opendev.org/c/openstack/diskimage-builder/+/770703 | 07:01 |
|---|---|---|
| *** slaweq has joined #opendev | 07:10 | |
| *** toomer has joined #opendev | 07:20 | |
| *** ralonsoh has joined #opendev | 07:23 | |
| *** lpetrut has joined #opendev | 07:24 | |
| *** sshnaidm has quit IRC | 07:41 | |
| *** fressi has joined #opendev | 07:49 | |
| *** fressi has joined #opendev | 07:52 | |
| openstackgerrit | Steve Baker proposed openstack/diskimage-builder master: Don't run grub2-install for efi block devices https://review.opendev.org/c/openstack/diskimage-builder/+/779107 | 07:52 |
| *** eolivare has joined #opendev | 07:53 | |
| *** ysandeep is now known as ysandeep|lunch | 08:00 | |
| *** sshnaidm has joined #opendev | 08:01 | |
| *** ykarel_ is now known as ykarel | 08:04 | |
| *** sboyron has joined #opendev | 08:12 | |
| *** andrewbonney has joined #opendev | 08:15 | |
| openstackgerrit | daniel.pawlik proposed openstack/diskimage-builder master: Changed get-pip url https://review.opendev.org/c/openstack/diskimage-builder/+/779173 | 08:17 |
| openstackgerrit | daniel.pawlik proposed openstack/diskimage-builder master: Changed get-pip url https://review.opendev.org/c/openstack/diskimage-builder/+/779173 | 08:18 |
| openstackgerrit | daniel.pawlik proposed openstack/diskimage-builder master: Change get-pip url https://review.opendev.org/c/openstack/diskimage-builder/+/779173 | 08:20 |
| *** rpittau|afk is now known as rpittau | 08:23 | |
| *** whoami-rajat_ has joined #opendev | 08:26 | |
| dpawlik | fungi, ianw, clarkb: hey, pls take a look on https://review.opendev.org/c/openstack/diskimage-builder/+/779173 . The pypa community has changed the url... | 08:30 |
| *** hashar has joined #opendev | 08:32 | |
| *** tosky has joined #opendev | 08:34 | |
| *** brinzhang has joined #opendev | 08:34 | |
| *** brinzhang0 has quit IRC | 08:36 | |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Fixes all tasks should be named rule https://review.opendev.org/c/zuul/zuul-jobs/+/779177 | 08:41 |
| openstackgerrit | Lee Yarwood proposed openstack/project-config master: Revert "Add custom cirros image with ahci module enabled to cache" https://review.opendev.org/c/openstack/project-config/+/779140 | 08:44 |
| openstackgerrit | Lee Yarwood proposed openstack/project-config master: Add Cirros 0.5.2 to cache https://review.opendev.org/c/openstack/project-config/+/779178 | 08:49 |
| *** klonn has joined #opendev | 08:56 | |
| *** jpena|off is now known as jpena | 08:58 | |
| *** ysandeep|lunch is now known as ysandeep | 09:30 | |
| *** mugsie__ is now known as mugsie | 10:11 | |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Fixes all tasks should be named rule https://review.opendev.org/c/zuul/zuul-jobs/+/779177 | 11:24 |
| *** klonn has quit IRC | 11:25 | |
| *** hashar has quit IRC | 11:37 | |
| *** ykarel has quit IRC | 11:52 | |
| *** ykarel has joined #opendev | 11:53 | |
| *** ysandeep is now known as ysandeep|afk | 12:00 | |
| *** ykarel_ has joined #opendev | 12:17 | |
| *** ykarel has quit IRC | 12:20 | |
| *** marios is now known as marios|biab | 12:27 | |
| *** ysandeep|afk is now known as ysandeep | 12:30 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** lpetrut has quit IRC | 12:44 | |
| *** artom has joined #opendev | 12:45 | |
| *** iurygregory has quit IRC | 12:46 | |
| *** marios|biab is now known as marios | 12:58 | |
| *** lpetrut has joined #opendev | 13:09 | |
| *** hashar has joined #opendev | 13:13 | |
| *** ykarel_ is now known as ykarel | 13:23 | |
| *** jpena|lunch is now known as jpena | 13:24 | |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Fixes all tasks should be named rule https://review.opendev.org/c/zuul/zuul-jobs/+/779177 | 13:30 |
| fungi | dpawlik: thanks, i emergency approved that | 13:32 |
| *** zoharm has joined #opendev | 13:33 | |
| fungi | codesearch says the now broken url is also used in master branches of openstack/openstack-helm-images, opendev/system-config, openstack/mistral, openstack/openstack-ansible-ops, opendev/puppet-pip | 13:35 |
| fungi | i'll work on fixes for opendev/system-config and opendev/puppet-pip in a moment | 13:35 |
| dpawlik | thanks fungi | 13:48 |
| fungi | dpawlik: i have a feeling there are also lots of stable branches which will need updates for that (at least devstack) | 13:53 |
| dpawlik | fungi: they can even create an redirection in such update, but not only set message with "We ware sorry" | 13:56 |
| dpawlik | fungi: checking... | 13:57 |
| fungi | dpawlik: yeah, when i first saw the pr, i thought maybe they were creating a wrapper around get-pip.py to emit a warning on stderr or something before invoking the original script | 13:59 |
| fungi | but yeah, a redirect seems like it would have solved the problem too | 13:59 |
| fungi | replacing a script at a well-known url with a different script which just spits out an error, while certainly visible, is not so swell for automated systems people may not be watching nonstop | 14:01 |
| dpawlik | fungi: exactly | 14:02 |
| *** mnaser has quit IRC | 14:03 | |
| *** mnaser has joined #opendev | 14:03 | |
| *** mnaser has quit IRC | 14:03 | |
| *** mnaser has joined #opendev | 14:03 | |
| openstackgerrit | daniel.pawlik proposed opendev/system-config master: Change get-pip url https://review.opendev.org/c/opendev/system-config/+/779268 | 14:04 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fixes all tasks should be named rule https://review.opendev.org/c/zuul/zuul-jobs/+/779177 | 14:08 |
| dpawlik | fungi: ups, I also proposed a patch for system-config with just simple find + sed. Feel free to abandon if its bad | 14:10 |
| fungi | oh, thanks! you beat me to it | 14:18 |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-pip master: Change get-pip url https://review.opendev.org/c/opendev/puppet-pip/+/779274 | 14:28 |
| fungi | dpawlik: and that's ^ the equivalent for our puppet-pip module | 14:28 |
| fungi | looks like we've got arm64 node problems again | 14:29 |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Bits to keep ansible-lint happy https://review.opendev.org/c/zuul/zuul-jobs/+/779276 | 14:47 |
| zbr | Do we have any progress on gerrit ability to auto complete reviewers? It becomes impossible to pick someone. | 14:50 |
| zbr | 26s to get an autocomplete for "Kevin", often result are returning in random order (not sure if this impacts the display of results) | 14:53 |
| zbr | https://sbarnea.com/ss/Screen-Shot-2021-03-08-14-53-51.33.png | 14:54 |
| openstackgerrit | Merged openstack/diskimage-builder master: Change get-pip url https://review.opendev.org/c/openstack/diskimage-builder/+/779173 | 15:10 |
| *** hashar has quit IRC | 15:14 | |
| fungi | zbr: what do you mean progress? | 15:26 |
| fungi | i expect that 26s depends on when you try and how much load the server is under | 15:27 |
| fungi | sounds like you may be describing a symptom of high load rather than an actual inefficiency with user typeahead autocompletion | 15:28 |
| fungi | have you tried to time it when the system is mostly idle? | 15:28 |
| fungi | or on a separate idle server like review-test? | 15:28 |
| zbr | yeah, since the v3 upgrade, it was very slow most of the time during normal working hours (the fact that is fact when nobody is using is not of much practical use) | 15:29 |
| *** fressi has quit IRC | 15:29 | |
| zbr | during the weekend it is easy to see how performance goes to normal values | 15:30 |
| zbr | my impression is that there was not a sudden explosion of number of patches being created/updated since the upgrade, and do not remember observing this kind of delays with gerrit v2 ever. I hope clarkb does not take this as a complaints about the upgrade ;) | 15:32 |
| fungi | zbr: sure, but my point was that gerrit itself has been very slow/loaded in general since the upgrade, i'm not sure whu user typeahead is necessarily a separate problem | 15:32 |
| *** lpetrut has quit IRC | 15:32 | |
| fungi | so are you asking about problems with typeahead, or are you asking about general gerrit slowness? | 15:32 |
| zbr | the typeahead feature is a very special one as user cannot really freeze while trying to select a reviewer. | 15:33 |
| fungi | i haven't seen gerrit freeze while i'm typing usernames | 15:33 |
| zbr | i have no problem waiting 10-30s more seconds when uploading a change using git-review, but when adding reviewers inside the browser my expectations are very different. | 15:33 |
| zbr | nothing happens, if you switch window to do something else while the server is processing a text search, when you go back the popup disapears. | 15:34 |
| zbr | the current implementation forces you to have to wait inside the browser for a magic answer. | 15:35 |
| fungi | sometimes it doesn't provide suggestions right away, but it doesn't stop me from continuing to type into the field. maybe i'm misunderstanding what you're saying | 15:35 |
| fungi | judging from http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=26&rra_id=all if you tried in the past hour, yes i expect gerrit's api would have been quite slow to respond | 15:36 |
| zbr | "right away" is for me <5s, not 36s. I do not happen to know the exact usernames of all reviewers i need to pick. | 15:36 |
| zbr | Should I build a local copy paste list of usernames and paste them? | 15:37 |
| fungi | i think if you want faster response on typeahead while under heavy load, you'll likely need to work upstream to implement query optimizations. we're probably going to continue trying to find ways to reduce the load on our gerrit deployment so that everything is generally faster rather than focusing on typeahead autocompletion responsiveness | 15:38 |
| zbr | i wonder if we can do something particularly for these lookups. | 15:38 |
| fungi | yep, the source code of gerrit is freely available, and its maintainers are reasonably resoinsive folks | 15:38 |
| fungi | er, responsive | 15:39 |
| fungi | and if you can work with them on improvements to typeahead completion then i expect we can backport whatever improvement they approve | 15:39 |
| *** Guest58358 is now known as JayF | 15:39 | |
| *** JayF has quit IRC | 15:39 | |
| *** JayF has joined #opendev | 15:39 | |
| *** ysandeep is now known as ysandeep|dinner | 15:40 | |
| fungi | i expect with the variety different identity types those fields match on, even the cache lookups for them are very expensive | 15:41 |
| fungi | but maybe something like a separate trigram cache could help | 15:42 |
| clarkb | zbr: my frustration is that some of us (like me) are activiely trying to continue to profile and improve these problems. We know the performance isn't great, but I don't just go upstream and yell at them about it | 15:45 |
| clarkb | zbr: if you'd like to helpw ith that that would be excellent but the semi periodic complaints really don't help imo | 15:45 |
| clarkb | the most recent things I have been working on include dstat system metric recording for gerrit (and other services) in CI as well as a gerrit specific gatling git job which does load testing | 15:46 |
| clarkb | the gatling-git stuff could definitely use some work as its very naive right now and doesn't even succeed reliably | 15:46 |
| clarkb | none of this work requires special access or privileges as it relies on CI to generate the data | 15:47 |
| clarkb | specifically, I'm hoping we can measure some of this stuff between gerrit 3.2 and 3.3 to see if 3.3 improves things before we upgrade. Preliminary data indicates that gerrit 3.3 may consume a bit more memory but perform slightly better for rtt on pushing changes and the like | 15:49 |
| zbr | based on https://bugs.chromium.org/p/gerrit/issues/detail?id=13781&q=suggest&can=1&sort=-id I am afraid it could be much worse. | 15:49 |
| clarkb | it is also worth noting that interacting with upstream's gerrit is also often slow | 15:50 |
| clarkb | I expect there is likely more we can do to improve things, but that we should expect some sort of floor considering upstream seems to struggle similarly | 15:51 |
| zbr | did we reach the upper hw limits for the machine on which we can host gerrit? can't we throw more ram/cores/ssd? | 15:52 |
| clarkb | the disk is already an ssd (and doesn't seem to be the bottleneck, we see higher disk activity when doing other system acitivities like backups and git gc), we are running on a 60GB of memory instance and using far too much of that and it has 16 cpu cores | 15:53 |
| clarkb | it is theoretically possible to use a bigger instance, but we're pretty much maxing out what is there | 15:53 |
| fungi | for the most part. it's a 60gb ram/16 vcpu flavor, and we put the data on ssd-backed cinder, yeah | 15:53 |
| zbr | baremetal? | 15:54 |
| clarkb | no, we don't have access to any baremetal | 15:54 |
| fungi | there also aren't great options for scaling gerrit "horizontally" (you can do some offload via replication, but we more or less already do that anyway by using a separate farm of git servers) | 15:55 |
| clarkb | it appears that the zuul graceful shutdown resulted in zuul-executors restarting for some reason | 15:55 |
| clarkb | and they cannot connect to zk so are idling | 15:55 |
| clarkb | (on the old servers I mean) | 15:55 |
| clarkb | I'll down zuul on them once morning meetings are done | 15:56 |
| fungi | well, at least they're not doing anything | 15:56 |
| fungi | but yeah i wonder why they started back up | 15:56 |
| zbr | i don't know if that helps, but I can share my scaling experience I had with corporate Citrix Jira instance (JVM) too, which was slow too (same patterns slow response times). | 15:57 |
| clarkb | zbr: as mentioned, feel free to poke at it in CI and leverage things like dstat and gatling git to help measure the results | 15:57 |
| zbr | it was ~6-7 years ago and I got two Dell servers 48cores, 96GB RAM. It run well on them but when I decided to ditch the hypervisor and run baremetal, I got a 3.5x increase in performance (measures with a full database reindex, something that used to take hours). Because I had two identical machines, I was able to compare differences. | 15:59 |
| *** gmann is now known as gmann_afk | 16:01 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: trigger image upload https://review.opendev.org/c/opendev/system-config/+/779289 | 16:11 |
| *** ysandeep|dinner is now known as ysandeep | 16:13 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Add custom cirros image with ahci module enabled to cache" https://review.opendev.org/c/openstack/project-config/+/779140 | 16:20 |
| *** zoharm has quit IRC | 16:24 | |
| *** ysandeep is now known as ysandeep|away | 16:26 | |
| clarkb | zuul-executor has been stopped on ze05-08.openstack.org | 16:32 |
| clarkb | I'm going to eat some breakfast then will delete those servers and start spinning up 09-12 | 16:32 |
| *** marios is now known as marios|out | 17:01 | |
| clarkb | #status log Deleted ze05-z08.openstack.org as new ze05-ze08.opendev.org servers have taken over | 17:01 |
| openstackstatus | clarkb: finished logging | 17:01 |
| *** ykarel has quit IRC | 17:02 | |
| clarkb | working on 09-12 now to finish this set up | 17:03 |
| *** eolivare has quit IRC | 17:03 | |
| *** jpena is now known as jpena|brb | 17:15 | |
| fungi | looks like neutron and nova added py38 and py39 unit test jobs to the check-arm64 pipeline at some point. interesting, though that's creating quite the backlog in the lenovo cloud | 17:15 |
| clarkb | linaro you eman? | 17:15 |
| fungi | linaro, yes sorry | 17:15 |
| fungi | several of the same letters | 17:16 |
| *** iurygregory has joined #opendev | 17:17 | |
| *** marios|out has quit IRC | 17:18 | |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Adding ze09-12.opendev.org to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/779307 | 17:20 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Replace ze09-12.openstack.org with ze09-12.opendev.org https://review.opendev.org/c/opendev/system-config/+/779309 | 17:24 |
| clarkb | infra-root if we can get both of those approved I'll happily monitor the results today and get servers swapped out | 17:24 |
| corvus | clarkb: ++ and nice tach on removing ze12 from testing | 17:26 |
| clarkb | I'm hoping I'll be able to observe the results of the graceful stop in real time this time around to see if I can tell why the service was restarted | 17:27 |
| *** hamalq has joined #opendev | 17:29 | |
| clarkb | fungi: did you want to review those too ^ or should I go ahead and approve them? | 17:35 |
| *** jpena|brb is now known as jpena | 17:37 | |
| clarkb | I've gone ahead and approved them. I'll watch them | 17:43 |
| fungi | yep, sorry, got sidetracked dealing with other tasks, but i'm good with it | 17:45 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Adding ze09-12.opendev.org to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/779307 | 17:49 |
| *** iurygregory has quit IRC | 17:49 | |
| *** andrewbonney has quit IRC | 18:05 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Replace ze09-12.openstack.org with ze09-12.opendev.org https://review.opendev.org/c/opendev/system-config/+/779309 | 18:08 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Fix get-pip.py url for python3.5 compat https://review.opendev.org/c/opendev/system-config/+/779336 | 18:08 |
| clarkb | fungi: corvus ^ that probably deserves a bit more careful review now. I'm also going to quickly check if we use the old url anywhere | 18:08 |
| fungi | clarkb: the same fix has already been approved | 18:09 |
| fungi | and https://review.opendev.org/779274 does the same for our puppet-pip module | 18:09 |
| fungi | https://review.opendev.org/779268 is the system-config one | 18:10 |
| *** toomer has quit IRC | 18:10 | |
| openstackgerrit | Clark Boylan proposed opendev/puppet-pip master: Update the get-pip.py url for python3.5 https://review.opendev.org/c/opendev/puppet-pip/+/779337 | 18:11 |
| fungi | clarkb: see above | 18:11 |
| clarkb | oh heh I should look first I guess | 18:11 |
| clarkb | I'll abandon and rebase accordingly | 18:12 |
| fungi | system-config fix is ~40 minutes eta on merging | 18:12 |
| fungi | i single-core approved it a little while ago | 18:13 |
| fungi | approval of my puppet-pip version would be appreciated though | 18:13 |
| fungi | and i see you just did--thanks! | 18:13 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Replace ze09-12.openstack.org with ze09-12.opendev.org https://review.opendev.org/c/opendev/system-config/+/779309 | 18:13 |
| clarkb | fungi: ^ reabsed on top of the existing chagne already in the gate. I have abandoned the changes i created. | 18:14 |
| fungi | clarkb: also, before you propose any others, the similar fix for dib already merged earlier today | 18:14 |
| fungi | it was 779173 | 18:15 |
| clarkb | fungi: codeserach only showed system-config and puppet-pip remaining, so I wasn't palnning on doing more. THough I suppose some stable branches may have problems? | 18:15 |
| fungi | yeah, there's fixes going in for devstack already too | 18:15 |
| fungi | dpawlik was on the ball | 18:15 |
| fungi | many thanks to him | 18:15 |
| clarkb | thank you dpawlik ! | 18:16 |
| *** rpittau is now known as rpittau|afk | 18:16 | |
| fungi | and yeah, i enumerated some additional openstack projects in here which may see issues at least in master: mistral, openstack-ansible-ops, openstack-helm-images (seemed to be in some dockerfiles mostly) | 18:17 |
| *** dtantsur is now known as dtantsur|afk | 18:17 | |
| fungi | various pins for old python versions. thankfully most of the bootstrap.pypa.io urls were for unversioned get-pip (and a few for easy_install) | 18:19 |
| *** mlavalle has joined #opendev | 18:23 | |
| clarkb | fungi: were these old urls deprecated for more than a week? its hard to tell based on that commit | 18:25 |
| clarkb | (I can see why they appeared to do it, they wanted to namespace the script paths which makes sense | 18:25 |
| fungi | at first i thought they were saying they'd add a wrapper at the old urls to emit a warning and call the original script, but i didn't actually find evidence of it | 18:27 |
| fungi | first we noticed was when they switched it to a script which just prints an error and exits 1 | 18:28 |
| *** jpena is now known as jpena|off | 18:30 | |
| openstackgerrit | Merged opendev/system-config master: Change get-pip url https://review.opendev.org/c/opendev/system-config/+/779268 | 18:44 |
| openstackgerrit | Merged opendev/puppet-pip master: Change get-pip url https://review.opendev.org/c/opendev/puppet-pip/+/779274 | 18:44 |
| fungi | clarkb: ^ those are in now | 18:45 |
| clarkb | yup, and my chagne is based on it so it should +1 now | 18:45 |
| *** iurygregory has joined #opendev | 18:46 | |
| clarkb | and now my change is in the gate so that was all that was holding it up | 19:02 |
| clarkb | thanks again! | 19:02 |
| fungi | righteous | 19:10 |
| fungi | righteous | 19:11 |
| * fungi is clearly having problems with his input history today | 19:12 | |
| *** stand has quit IRC | 19:15 | |
| *** stand has joined #opendev | 19:22 | |
| *** gmann_afk is now known as gmann | 19:32 | |
| openstackgerrit | Merged opendev/system-config master: Replace ze09-12.openstack.org with ze09-12.opendev.org https://review.opendev.org/c/opendev/system-config/+/779309 | 19:38 |
| fungi | i guess that means the pipes are lubricated and thinks are moving again | 19:39 |
| fungi | er, things | 19:39 |
| *** klonn has joined #opendev | 19:41 | |
| clarkb | indeed, now we wait a bit for zuul to catch up with managing itself | 19:41 |
| *** whoami-rajat_ is now known as whoami-rajat | 19:44 | |
| *** hashar has joined #opendev | 20:03 | |
| openstackgerrit | Emma Foley proposed openstack/project-config master: Add infrawatch/{collectd,qdr}-config to available repos https://review.opendev.org/c/openstack/project-config/+/779361 | 20:34 |
| openstackgerrit | Merged openstack/project-config master: Add infrawatch/{collectd,qdr}-config to available repos https://review.opendev.org/c/openstack/project-config/+/779361 | 20:53 |
| *** arxcruz is now known as arxcruz|off | 21:04 | |
| *** auristor has quit IRC | 21:08 | |
| *** auristor has joined #opendev | 21:10 | |
| *** klonn has quit IRC | 21:12 | |
| ianw | forgot to mention yesterday was a public holiday here. back and catching up now | 21:57 |
| fungi | ianw: no worries, hope it was refreshing! | 21:58 |
| fungi | there was not too much excitement | 21:58 |
| fungi | pypa decided to reorganize bootstrap.pypa.io a bit but there weren't too many fixes needed | 21:58 |
| fungi | clarkb is almost through replacing executors | 21:59 |
| ianw | excellent. i hope to knock off our kerberos servers in ansible today, there's hopefully not much to it | 22:00 |
| fungi | just waiting for 779309 to get through infra-prod-service-zuul in the deploy pipeline, it's up next | 22:01 |
| fungi | then we should be able to bring up the new executor services and gracefully stop their old counterparts | 22:02 |
| fungi | now the job is running | 22:02 |
| fungi | i need to do a few chores but will try to keep an eye on it and take a break to start/graceful-stop services once it's passed | 22:03 |
| *** whoami-rajat has quit IRC | 22:06 | |
| *** slaweq has quit IRC | 22:15 | |
| mordred | ianw: for some reason my eyes read your line about public holiday as having been something fungi said. so then I thought "yesterday was a public holiday? shoot - I missed it. wait ... yesterday was sunday ... I'm confused! OH - that was ianw" :) | 22:18 |
| fungi | mordred: there's nothing stopping you from observing australian holidays | 22:21 |
| fungi | or maybe there is, but you shouldn't... let that stop you | 22:22 |
| fungi | infra-prod-service-zuul: success | 22:24 |
| fungi | i'll check the new servers to make sure they're configured now | 22:24 |
| fungi | yep, looks like it deployed | 22:27 |
| ianw | mordred: it was the Queen's birthday public holiday. It was not actually the Queen's birthday. It is not celebrated in all states. but if you want to raise a gin and tonic to Her Majesty's theoretical birthday, feel free :) | 22:27 |
| *** hashar has quit IRC | 22:29 | |
| * fungi hums a few bars of the sex pistols' "god save the queen" | 22:30 | |
| fungi | okay, executor containers have been upped on the new ze09-12 | 22:30 |
| fungi | i've now issued a `zuul-executor graceful` in the containers on the old ze09-12 | 22:34 |
| mordred | ianw: I feel like raising a gin and tonic to HM's theoretical birthday is a great idea | 22:41 |
| fungi | it's true that boston only dumped tea in the harbor, they held onto the gin | 22:44 |
| openstackgerrit | Merged openstack/project-config master: Add Cirros 0.5.2 to cache https://review.opendev.org/c/openstack/project-config/+/779178 | 22:54 |
| clarkb | fungi: thanks for doing that, I'm back now and will continue to monitor the replacement process. | 23:19 |
| clarkb | on my bike ride I started thinking about the launcher replacements, can I basically set max-servers: to 0 on nl01.openstack.org's config and max-servers: real value on nl01.opendev.org? | 23:20 |
| clarkb | I think that may work | 23:20 |
| clarkb | corvus: ^ if you're still around does that raise any alarms for you? I think I may also need to switch over the min servers value in the process | 23:23 |
| corvus | clarkb: if they have the same provider name, they may think they have leaked each others nodes; better to stop/start processes | 23:25 |
| clarkb | gotcha | 23:27 |
| clarkb | in that case I think the process looks more like removing the old server from inventory when we add the new one but also stop the launcher on the old one before we start the launcher on the new one | 23:28 |
| clarkb | ze10 is waiting on 2 jobs to complete before it gracefully stops | 23:32 |
| clarkb | I wonder if we are exiting non zero and thati s why the restart: on-failure is tripping | 23:32 |
| clarkb | anyone know if docker records the exit status of historical containers somewhere | 23:33 |
| ianw | clarkb: docker ps -a might ... | 23:42 |
| clarkb | I'll try that once at least one of them flips over | 23:43 |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix installation of proliant tools https://review.opendev.org/c/openstack/diskimage-builder/+/778524 | 23:47 |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix hooks order for CentOS/Fedora when mirror used https://review.opendev.org/c/openstack/diskimage-builder/+/772350 | 23:51 |
| fungi | clarkb: actually what you suggested will probably work... you're changing domain names and we fixed nodepool some time back to include use the fqdn instead of the short hostname for its identifier | 23:51 |
| fungi | corvus: ^ yeah? | 23:52 |
| clarkb | fungi: ya, but they all use the same provider name which I think means they will fight over ownership? | 23:52 |
| fungi | oh, provider name not launcher id. okay got it | 23:52 |
| clarkb | I think both come into play | 23:52 |
| clarkb | basically if launcher foo is operating on provider bar if it sees nodes from launcher baz in the same provider in zk it will see them as leaked? | 23:53 |
| clarkb | something like that | 23:53 |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove fedora-31 testing https://review.opendev.org/c/openstack/diskimage-builder/+/776503 | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
