Saturday, 2021-11-13

corvusi'm going to restart zuul14:34
corvusfor good measure, i'm going to go ahead and clear the zk state, so it'll take a bit to restart14:34
corvusstarting up now14:39
fricklerstill getting a 503 on the web service, is that also expected to take so much time?14:56
fricklercorvus: ^^15:00
corvusfrickler: probably so with the new changes -- the web service uses zk directly now, so it's going to behave differently (and poorly at startup)15:01
corvusbut it's up now15:01
corvusand i'm re-enqueuing15:01
corvusthe jobs list doesn't seem to work as expected15:03
corvusi think it's because i started everything with an empty cache; i'm going to restart zuul-web.  it will 5xx again for a while, but will hopefully get a complete configuration this time.15:12
corvusjobs page looks good now; so that's just a startup sequencing issue15:23
corvusstarting up zuul01 now15:24
corvusand at this point, unlike the previous version, we should not observe any issues with the status page during the second scheduler startup (it should be stable) since it's being handled by zuul-web itself15:25
fungithanks, just got back from a walk and will try to keep an eye on it15:31
corvus01 is up and running15:31
fungiso is the solution for the empty jobs list to have zuul-web refresh its copy of the config periodically?15:32
corvusfungi: it does do that, it's just that it was unable to do so on the first startup because the branch cache was empty, and it's not permitted to fill it (because we don't want it talking to gerrit, etc).15:33
corvusit would have eventually worked once some tenant reconfiguration happened, but that would take a long time on a weekend :)15:33
fungiif the branch cache changes though, it will be able to reflect that?15:33
corvusyep, it reconfigures just like a scheduler15:34
fungioh, branch cache updates trigger reconfig, got it15:34
corvusyep, they happen first15:34
corvus(it effectively is a scheduler at this point, just one that doesn't actually process pipelines)15:34
corvus(a read-only scheduler)15:35
fungiso we could force a reconfig in such situations rather than having a specific start order, i guess15:35
corvusit's probably going to use more memory than before.  that's one of the things i'm interested in seeing15:35
corvusfungi: yes, or have it smart enough to wait until the branch cache is populated15:35
fungioh, yep15:35
corvusbut really, just restarting zuul-web has the same effect on zuul-web as a full reconfig, but only affects the one process, so i still think that's preferable15:36
fungimakes sense15:45
fungiand it's only relevant when "bootstrapping" from empty zk anyway15:45
fungiso not a concern for normal restarts/upgrades unless we clear zk content15:46
fungirackspace had an outage which impacted the ethercalc server, so it was rebooted roughly 8 hours ago. seems to be up and working currently though18:28
*** mgoddard- is now known as mgoddard19:32

Generated by 2.17.2 by Marius Gedminas - find it at!