Tuesday, 2021-03-09

corvusclarkb: oh, you know what, it may be okay00:01
corvusi was worried about leaked instances, but as long as they both have access to the same zk data, should be fine.00:01
openstackgerritMerged openstack/diskimage-builder master: Don't use hardcode while override base image file  https://review.opendev.org/c/openstack/diskimage-builder/+/77197800:03
clarkbcorvus: oh interesting00:05
clarkbcorvus: how does that work for rejecting node requests?00:05
clarkbisn't that done by provider? so if wev'e got multiple providers in different launchers taht collide would we maybe break there?00:05
clarkbze12 is down to a single job. I expect it will finish up first00:07
openstackgerritMerged opendev/system-config master: refstack: trigger image upload  https://review.opendev.org/c/opendev/system-config/+/77928900:11
openstackgerritMerged opendev/system-config master: Fix sshfp record printing  https://review.opendev.org/c/opendev/system-config/+/77904400:11
clarkbze10 just restarted a few seconds ago00:14
fungii guess graceful means graceful restart not graceful stop00:14
clarkbfungi: yes, that was a known problem ebfore when we had docker-compose config that said restart: always, but we chagned it to restart: on-failure00:15
clarkbhowever, maybe it has to be started after the restart: on-failure config change was made to take effect?00:15
clarkbdocker ps -a only shows the current container, I think docker compose is good at cleaing up after itself00:15
clarkbya I wonder if that is it, the config at startup is what matters00:20
fungiahh, interesting00:22
*** lbragstad has quit IRC00:23
clarkbze10.openstack.org has been deleted now. Still waiting on the others00:23
*** tosky has quit IRC00:28
clarkbze11 has been deleted now (it just restarted zuul and failed to reconnect)00:29
clarkbjust two more to go then this set of rotations is done00:29
clarkb09 is gone now too. We wait on ze1200:36
clarkbhttps://zuul.opendev.org/t/openstack/stream/fe6fd4182ff7467a8326b142f08b43be?logfile=console.log is the last job we are waiting on. I wonder if the lack of output in the last almost 2 hours indicates it is stuck00:42
*** lbragstad has joined #opendev00:52
funginot paused?00:53
clarkbcorrect, it is executing normally if I parse the tail of the executor log properly00:53
clarkbwith a paused job the last thing the executor log spits out is "I'm pausing this job" but that isn't present in this case. I think it is a buggy job that is on its way to timing out00:54
clarkbI'll wait a few more minutes but then I need to go help with dinner00:55
clarkbcan clean it up in the morning. If it restarts it will just try to reconnect to zk and fail and doesn't cause problems00:55
clarkbyup its still going. Not a big deal I'll finish this up in the morning. Thank you for the help fungi01:00
fungiany time!01:03
*** mlavalle has quit IRC01:21
*** hamalq has quit IRC01:24
openstackgerritMerged openstack/diskimage-builder master: Add aarch64 support for rhel  https://review.opendev.org/c/openstack/diskimage-builder/+/77070301:52
openstackgerritMerged openstack/diskimage-builder master: Use the same bootloader pkg-map for all redhat family  https://review.opendev.org/c/openstack/diskimage-builder/+/77910502:00
openstackgerritMerged openstack/diskimage-builder master: Add efibootmgr utility for UEFI boot menu management  https://review.opendev.org/c/openstack/diskimage-builder/+/77910602:11
openstackgerritMerged opendev/system-config master: install-ansible: ensure stevedore  https://review.opendev.org/c/opendev/system-config/+/77835402:52
openstackgerritSteve Baker proposed openstack/diskimage-builder master: Don't run grub2-install for efi block devices  https://review.opendev.org/c/openstack/diskimage-builder/+/77910703:07
*** lbragstad has quit IRC03:43
*** dpawlik6 has joined #opendev03:51
*** dhellmann_ has joined #opendev03:54
*** raukadah has joined #opendev03:54
*** avass_ has joined #opendev03:55
*** icey_ has joined #opendev03:56
*** owalsh_ has joined #opendev03:58
*** openstackgerrit has quit IRC03:59
*** icey has quit IRC03:59
*** dhellmann has quit IRC03:59
*** dmellado has quit IRC03:59
*** avass has quit IRC03:59
*** chandankumar has quit IRC03:59
*** tbarron has quit IRC03:59
*** mhu has quit IRC03:59
*** cgoncalves has quit IRC03:59
*** dpawlik has quit IRC03:59
*** owalsh has quit IRC03:59
*** fbo has quit IRC03:59
*** dhellmann_ is now known as dhellmann03:59
*** dpawlik6 is now known as dpawlik03:59
*** fbo has joined #opendev04:02
*** dmellado has joined #opendev04:02
*** ykarel has joined #opendev04:23
*** whoami-rajat_ has joined #opendev04:41
*** whoami-rajat_ is now known as whoami-rajat04:46
*** raukadah is now known as chandankumar04:51
*** ysandeep|away is now known as ysandeep05:16
*** iurygregory has quit IRC05:26
*** marios has joined #opendev06:07
*** slaweq has joined #opendev06:50
*** cgoncalves has joined #opendev07:09
*** hashar has joined #opendev07:09
dpawlikclarkb: hehe, no problem :)07:12
*** eolivare has joined #opendev07:39
*** fressi has joined #opendev07:47
*** DSpider has joined #opendev08:04
*** lpetrut has joined #opendev08:04
*** DSpider has quit IRC08:05
*** hashar has quit IRC08:19
*** andrewbonney has joined #opendev08:19
*** rpittau|afk is now known as rpittau08:24
*** toomer has joined #opendev08:45
*** jpena|off is now known as jpena08:55
*** tosky has joined #opendev09:02
*** hashar has joined #opendev09:25
*** klonn has joined #opendev09:30
*** iurygregory_ has joined #opendev10:45
*** iurygregory_ is now known as iurygregory10:46
*** marios has quit IRC10:56
*** icey_ is now known as icey11:02
*** marios has joined #opendev11:03
*** hashar has quit IRC11:08
*** ysandeep is now known as ysandeep|afk11:11
*** klonn has quit IRC11:13
*** avass_ is now known as avass11:17
*** dtantsur|afk is now known as dtantsur11:25
*** marios has quit IRC11:31
*** marios has joined #opendev11:37
*** artom has quit IRC11:51
*** slaweq_ has joined #opendev11:52
*** brinzhang has quit IRC11:56
*** lpetrut has quit IRC11:56
*** lpetrut has joined #opendev11:58
*** slaweq has quit IRC11:59
*** ysandeep|afk is now known as ysandeep12:01
*** ykarel_ has joined #opendev12:17
*** ykarel has quit IRC12:20
*** ykarel_ is now known as ykarel12:21
*** artom has joined #opendev12:25
*** klonn has joined #opendev12:29
*** jpena is now known as jpena|lunch12:34
*** slaweq_ is now known as slaweq12:38
*** hashar has joined #opendev13:04
*** lbragstad has joined #opendev13:24
*** jpena|lunch is now known as jpena13:36
*** tkajinam has quit IRC13:53
*** artom has quit IRC14:49
*** ysandeep is now known as ysandeep|afk14:50
*** klonn has quit IRC15:13
clarkb#status log Replaced ze09-12.openstack.org with ze09-12.opendev.org focal servers. This concludes the rolling replacement of the zuul executors.15:13
openstackstatusclarkb: finished logging15:13
*** fressi has quit IRC15:28
*** hashar has quit IRC15:28
*** lpetrut has quit IRC15:29
*** ysandeep|afk is now known as ysandeep|away15:34
*** ykarel has quit IRC15:43
*** stephenfin has quit IRC15:53
*** stephenfin has joined #opendev15:54
*** hashar has joined #opendev15:57
toomerWhat is the best way to change username in Gerrit after migration to NoteDb ?16:10
toomerShould I even think about changing username for the account in Gerrit ?16:10
clarkbtoomer: you cannot change gerrit usernames16:12
clarkb(this was true pre notedb too)16:12
toomerSQL update helped from what I remember16:12
clarkbyes you could update the db but it had side effects that caused gerrit to strongly recommend against it iirc16:14
clarkbwe never really offered username changes to people as a result16:14
clarkb(its a value you set in your .gitconfig then never have to think about again)16:14
*** klonn has joined #opendev16:20
clarkbtoomer: https://groups.google.com/g/repo-discuss/c/y3Kq0YiKC7Q/m/1HT6w6QA7tQJ16:24
clarkbthats the really old thread explaining why it is a bad idea16:25
clarkb(we don't take advantage of that feature, but they have still made it very difficult to change anyway)16:25
toomerThanks, I will read that. I was only doing that for new account and not the one that had patches published.16:35
*** marios is now known as marios|out16:56
*** rpittau is now known as rpittau|afk17:05
*** hashar has quit IRC17:07
*** lpetrut has joined #opendev17:18
*** eolivare has quit IRC17:22
*** openstackgerrit has joined #opendev17:39
openstackgerritElod Illes proposed opendev/system-config master: Add stable branch delete rights to release managers  https://review.opendev.org/c/opendev/system-config/+/77954617:39
*** lpetrut has quit IRC18:00
*** artom has joined #opendev18:01
*** jpena is now known as jpena|off18:02
*** dtantsur is now known as dtantsur|afk18:13
*** klonn has quit IRC18:18
*** marios|out has quit IRC18:26
*** ralonsoh has quit IRC18:28
*** toomer has quit IRC18:29
*** hamalq has joined #opendev18:30
*** SWAT has quit IRC18:45
*** SWAT has joined #opendev18:53
*** hashar has joined #opendev18:53
*** andrewbonney has quit IRC19:09
*** hamalq has quit IRC19:09
*** hamalq has joined #opendev19:09
*** lbragstad has quit IRC20:01
clarkbinfra-root I've put the proposed external id cleanups input file at review:~clarkb/gerrit_user_cleanups/external_id_cleanups.txt.20210309.proposed if you can look that over and compare it to the classifications files in that dir that would be helpful20:03
clarkbI can run that today if we're happy with that list or sometime tomorrow, just want to keep this moving along20:04
clarkband now lunch20:04
*** lbragstad has joined #opendev20:15
*** irclogbot_2 has quit IRC20:31
*** irclogbot_0 has joined #opendev20:32
openstackgerritClark Boylan proposed opendev/system-config master: Use upstream jitsi-meet web image  https://review.opendev.org/c/opendev/system-config/+/77830820:36
clarkbcorvus: ^ that changes the parent job of the system-config-run-meetpad job to drop the containers which seems to assume that it will hae built images in intermediate registry but since we pull upstream that isn't the case.20:37
*** sboyron has quit IRC20:38
*** whoami-rajat has quit IRC20:42
openstackgerritClark Boylan proposed opendev/system-config master: Update nodepool launchers to focal in testing  https://review.opendev.org/c/opendev/system-config/+/77698120:45
openstackgerritClark Boylan proposed opendev/system-config master: Update zookeeper servers in testing to focal  https://review.opendev.org/c/opendev/system-config/+/77698220:45
openstackgerritClark Boylan proposed opendev/system-config master: Update zuul01 to focal in testing  https://review.opendev.org/c/opendev/system-config/+/77698320:45
clarkbinfra-root in addition to reviewing the contents of that list of cleanups do you think you can reivew https://review.opendev.org/c/opendev/system-config/+/777846 and possibly land it? That way we get the current set of tooling in the tree which will simplify when I switch branches around for other work :)20:47
*** klonn has joined #opendev20:56
*** klonn has quit IRC20:57
clarkb#status log Cleaned up old DNS records for old openstack.org zuul mergers and executors21:00
openstackstatusclarkb: finished logging21:00
funginice to see the records in that zone shrinking as we go21:00
clarkb self.launcher_id = "%s-%s-%s" % (socket.gethostname(), os.getpid(), self.name) <- I think that will make my plan for easy mode launcher replacement tricky21:22
clarkbmaybe we can manipulate the docker setup to select a high pid?21:22
clarkband then that would work I think21:22
clarkbcould do nl05-08, or maybe update nodepool to use the full fqdn if that is a safe transition21:25
clarkbreading the code I think it may be ok to have two launchers with the same provider name running at the same time as long as the launcher_id above is unique. They will fight over deleting leaked instances but they should only delete nodes unknown to nodepool21:32
clarkb(that is nodepool as a whole as it looks up the node id in zk first)21:32
fungiahh, yeah i guess os.getpid() inside containers will be consistent21:39
fungithough hard to confirm since our images don't include procps21:41
fungioh, right, i can `ls /proc`21:41
clarkbyou can ps within the container I think?21:42
clarkbbut the nodepool logs seem to confirm it looking at the values. It isn't 100% overlap but its close enough to be dangerous :)21:42
fungiyeah, pid 7 seems like21:43
fungiat least on both nl02 and nl0221:44
fungier, 01 and 0221:44
clarkband 6 on 03 I think21:44
fungiyep, confirmed with `cat /proc/6/cmdline` there21:47
clarkbdiscussing potential changes to nodepool in #zuul. I think I've convinced myself a change would be safe, just need to figure out what the most appropriate change would be21:47
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation.  https://review.opendev.org/c/zuul/zuul-jobs/+/76480821:57
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation.  https://review.opendev.org/c/zuul/zuul-jobs/+/76480822:18
openstackgerritMartin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI  https://review.opendev.org/c/opendev/system-config/+/77629222:22
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation.  https://review.opendev.org/c/zuul/zuul-jobs/+/76480822:24
corvusclarkb: there may be some grafana dashboards that need updating with hostname changes (there might be some openstack.org names in the metrics)22:49
corvus(i'm thinking like executor hostname rollups)22:50
clarkboh ya let me look at grafyaml22:50
clarkbhrm the grafyaml configs don't explicitly do anything with those names22:51
clarkbit must just be matching a * in the graphite stats paths?22:51
openstackgerritMartin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI  https://review.opendev.org/c/opendev/system-config/+/77629222:52
clarkbI don't know that we should merge those together since they are distinct, maybe carry the old stats for a while then delete them?22:52
*** hashar has quit IRC22:53
clarkbor maybe drop the grafana zuul dashboard and haev it recreate? its not entirely clear to me how much of that is remembered by grafana22:53
clarkbI do see where we laias the openstack names to their hostnames22:54
corvusyeah, grep for openstack.org (don't escape the dot)22:55
corvusi think that's what's causing issues on https://grafana.opendev.org/d/5Imot6EMk/zuul-status?orgId=1 right now22:56
corvussee "running builds"22:56
clarkbya I'm working on a change22:56
clarkbI think I see how to make this better22:56
corvusfwiw, i'd recommend just cutting over to the new names at once; the old guages are stuck at their old values (will need a statsd restart to kill them)22:57
openstackgerritClark Boylan proposed openstack/project-config master: Update grafana for opendev zuul executors  https://review.opendev.org/c/openstack/project-config/+/77962623:00
*** tkajinam has joined #opendev23:00
clarkbcorvus: ^ yup that does a cut over or attempts to anyway. I expect the glob I've done on the metric path is valid like that?23:00
corvusclarkb: https://graphite.opendev.org/?showTarget=stats.gauges.zuul.executor.*_opendev_org.running_builds&width=586&height=308&target=stats.gauges.zuul.executor.*_opendev_org.running_builds&hideLegend=false looks legit23:02
clarkbyup graphite docs seem to say my glob is valid23:02
clarkbhttps://graphite-api.readthedocs.io/en/latest/api.html#paths-and-wildcards was what I found23:02
clarkbthey have examples with multiple * and context between23:03
clarkbcorvus: ianw has +2'd the chagne did you want to approve it or shoudl I go ahead and do that23:15
*** mlavalle has joined #opendev23:22
openstackgerritMerged openstack/project-config master: Update grafana for opendev zuul executors  https://review.opendev.org/c/openstack/project-config/+/77962623:26
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884023:42
openstackgerritIan Wienand proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI  https://review.opendev.org/c/opendev/system-config/+/77629223:54

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!