| corvus | i'm going to restart zuul | 14:34 |
|---|---|---|
| corvus | for good measure, i'm going to go ahead and clear the zk state, so it'll take a bit to restart | 14:34 |
| corvus | starting up now | 14:39 |
| frickler | still getting a 503 on the web service, is that also expected to take so much time? | 14:56 |
| frickler | corvus: ^^ | 15:00 |
| corvus | frickler: probably so with the new changes -- the web service uses zk directly now, so it's going to behave differently (and poorly at startup) | 15:01 |
| corvus | but it's up now | 15:01 |
| corvus | and i'm re-enqueuing | 15:01 |
| corvus | the jobs list doesn't seem to work as expected | 15:03 |
| corvus | i think it's because i started everything with an empty cache; i'm going to restart zuul-web. it will 5xx again for a while, but will hopefully get a complete configuration this time. | 15:12 |
| corvus | jobs page looks good now; so that's just a startup sequencing issue | 15:23 |
| corvus | starting up zuul01 now | 15:24 |
| corvus | and at this point, unlike the previous version, we should not observe any issues with the status page during the second scheduler startup (it should be stable) since it's being handled by zuul-web itself | 15:25 |
| fungi | thanks, just got back from a walk and will try to keep an eye on it | 15:31 |
| corvus | 01 is up and running | 15:31 |
| fungi | so is the solution for the empty jobs list to have zuul-web refresh its copy of the config periodically? | 15:32 |
| corvus | fungi: it does do that, it's just that it was unable to do so on the first startup because the branch cache was empty, and it's not permitted to fill it (because we don't want it talking to gerrit, etc). | 15:33 |
| corvus | it would have eventually worked once some tenant reconfiguration happened, but that would take a long time on a weekend :) | 15:33 |
| fungi | if the branch cache changes though, it will be able to reflect that? | 15:33 |
| corvus | yep, it reconfigures just like a scheduler | 15:34 |
| fungi | oh, branch cache updates trigger reconfig, got it | 15:34 |
| corvus | yep, they happen first | 15:34 |
| corvus | (it effectively is a scheduler at this point, just one that doesn't actually process pipelines) | 15:34 |
| corvus | (a read-only scheduler) | 15:35 |
| fungi | so we could force a reconfig in such situations rather than having a specific start order, i guess | 15:35 |
| corvus | it's probably going to use more memory than before. that's one of the things i'm interested in seeing | 15:35 |
| corvus | fungi: yes, or have it smart enough to wait until the branch cache is populated | 15:35 |
| fungi | oh, yep | 15:35 |
| corvus | but really, just restarting zuul-web has the same effect on zuul-web as a full reconfig, but only affects the one process, so i still think that's preferable | 15:36 |
| fungi | makes sense | 15:45 |
| fungi | and it's only relevant when "bootstrapping" from empty zk anyway | 15:45 |
| fungi | so not a concern for normal restarts/upgrades unless we clear zk content | 15:46 |
| fungi | rackspace had an outage which impacted the ethercalc server, so it was rebooted roughly 8 hours ago. seems to be up and working currently though | 18:28 |
| *** mgoddard- is now known as mgoddard | 19:32 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!