| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815175: Add reno about zk disconnect thread fix https://review.opendev.org/c/zuul/zuul/+/815175 | 02:45 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Simon Westphahl: [zuul/zuul] 815111: Store builds in Zookeeper https://review.opendev.org/c/zuul/zuul/+/815111 | 15:50 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 15:52 | |
| - [zuul/zuul] 815077: Run dstat in tox jobs https://review.opendev.org/c/zuul/zuul/+/815077 | ||
| - [zuul/zuul] 814684: DNM: Increase unit test job timeout to 2h https://review.opendev.org/c/zuul/zuul/+/814684 | ||
| - [zuul/zuul] 814685: DNM: Test unit tests on larger nodes https://review.opendev.org/c/zuul/zuul/+/814685 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815195: DNM: remove stestr concurrency argument https://review.opendev.org/c/zuul/zuul/+/815195 | 16:55 | |
| @fungicide:matrix.org | > <@jim:acmegating.com> Clark, fungi: ^ could we go ahead and approve that? so that if the opendev restart goes well, we can just tag and release that on monday? | 16:56 |
|---|---|---|
| yeah, retroactive +2, sorry had stepped away from the machine where i've got a matrix client up | ||
| @jim:acmegating.com | fungi: thanks! | 16:57 |
| @fungicide:matrix.org | things seem to be looking fine so far since yesterday's restart, but it's also a very quiet saturday in opendev | 17:00 |
| @fungicide:matrix.org | we bottomed out at 101k znodes a few hours ago | 17:01 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815072: Don't use --verbose with yarn https://review.opendev.org/c/zuul/zuul/+/815072 | 17:04 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815196: Use an election for scheduler periodic stats https://review.opendev.org/c/zuul/zuul/+/815196 | 17:19 | |
| @jim:acmegating.com | here's what i've learned about test runtime: | 18:30 |
| * on current nodes: ranges from 1:00 to 2:00, average: 87 | ||
| * on large nodes: same range, but closer to low end, average: 68 | ||
| * on large nodes with uncapped concurrency: range 0:45-1:15, average: 48 | ||
| @jim:acmegating.com | i think it's going to be worth it to switch to large nodes and uncap the concurrency setting. | 18:31 |
| @jim:acmegating.com | the dstat cpu graph looks pretty nice on the large uncapped nodes. maybe a little bit of room for improvement still. | 18:35 |
| @jim:acmegating.com | and, i apologize for what i'm about to do. but it's necessary since there are now some conflicts between master and the stack. | 18:37 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 18:38 | |
| - [zuul/zuul] 815077: Use larger test nodes and uncap concurrency https://review.opendev.org/c/zuul/zuul/+/815077 | ||
| - [zuul/zuul] 809293: Add an API for ZK-backed objects https://review.opendev.org/c/zuul/zuul/+/809293 | ||
| - [zuul/zuul] 810328: Temporarily enqueue cycle changes for reporting https://review.opendev.org/c/zuul/zuul/+/810328 | ||
| - [zuul/zuul] 811244: Dequeue items after we're done with them https://review.opendev.org/c/zuul/zuul/+/811244 | ||
| - [zuul/zuul] 812750: Add LocalZKContext for job freezing https://review.opendev.org/c/zuul/zuul/+/812750 | ||
| - [zuul/zuul] 812760: Add RepoState object https://review.opendev.org/c/zuul/zuul/+/812760 | ||
| - [zuul/zuul] 813552: Remove Worker class https://review.opendev.org/c/zuul/zuul/+/813552 | ||
| - [zuul/zuul] 813895: Move job_graph attribute to BuildSet https://review.opendev.org/c/zuul/zuul/+/813895 | ||
| - [zuul/zuul] 813913: Serialize JobGraph objects to ZK https://review.opendev.org/c/zuul/zuul/+/813913 | ||
| - [zuul/zuul] 814065: Serialize ProjectMetadata on JobGraph https://review.opendev.org/c/zuul/zuul/+/814065 | ||
| - [zuul/zuul] 814071: Add test_freeze_noop_job https://review.opendev.org/c/zuul/zuul/+/814071 | ||
| - [zuul/zuul] 814069: Remove setBase from job freeze API https://review.opendev.org/c/zuul/zuul/+/814069 | ||
| - [zuul/zuul] 814070: Create Abstract and FrozenJob classes https://review.opendev.org/c/zuul/zuul/+/814070 | ||
| - [zuul/zuul] 814242: Make FrozenJob.updateParentData a static method https://review.opendev.org/c/zuul/zuul/+/814242 | ||
| - [zuul/zuul] 814281: Remove toDict from FrozenJob https://review.opendev.org/c/zuul/zuul/+/814281 | ||
| - [zuul/zuul] 814243: Make FrozenJob a ZKObject https://review.opendev.org/c/zuul/zuul/+/814243 | ||
| - [zuul/zuul] 814329: Implement frozen job serialization/deserialization https://review.opendev.org/c/zuul/zuul/+/814329 | ||
| - [zuul/zuul] 814679: Store FrozenJob data in separate znodes https://review.opendev.org/c/zuul/zuul/+/814679 | ||
| - [zuul/zuul] 815154: Update test_inventory to be ZK-friendly https://review.opendev.org/c/zuul/zuul/+/815154 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Simon Westphahl: | 18:38 | |
| - [zuul/zuul] 809532: Simplified attribute API for ZKObjects https://review.opendev.org/c/zuul/zuul/+/809532 | ||
| - [zuul/zuul] 809414: Make QueueItem a Zookeeper object https://review.opendev.org/c/zuul/zuul/+/809414 | ||
| - [zuul/zuul] 810658: Store pipeline state in Zookeeper https://review.opendev.org/c/zuul/zuul/+/810658 | ||
| - [zuul/zuul] 810920: Store change queues in Zookeeper https://review.opendev.org/c/zuul/zuul/+/810920 | ||
| - [zuul/zuul] 811422: Save and restore bundle with item in Zookeeper https://review.opendev.org/c/zuul/zuul/+/811422 | ||
| - [zuul/zuul] 811955: Pass ZK context to deserialize method of ZKObjects https://review.opendev.org/c/zuul/zuul/+/811955 | ||
| - [zuul/zuul] 812450: Move ZuulMark from configloader to model https://review.opendev.org/c/zuul/zuul/+/812450 | ||
| - [zuul/zuul] 812451: Recursively delete all sub-nodes of ZKObjects https://review.opendev.org/c/zuul/zuul/+/812451 | ||
| - [zuul/zuul] 812466: Only retry ZK operations for Kazoo exceptions https://review.opendev.org/c/zuul/zuul/+/812466 | ||
| - [zuul/zuul] 812452: Store build sets in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812452 | ||
| - [zuul/zuul] 812467: Add support for sharded ZKObjects https://review.opendev.org/c/zuul/zuul/+/812467 | ||
| - [zuul/zuul] 812673: Store RepoFiles for a build set in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812673 | ||
| - [zuul/zuul] 813805: Remove project pipeline config from queue item https://review.opendev.org/c/zuul/zuul/+/813805 | ||
| - [zuul/zuul] 813809: Lookup event class names from global symbol table https://review.opendev.org/c/zuul/zuul/+/813809 | ||
| - [zuul/zuul] 813826: Store and resolve queue item's ahead/behind refs https://review.opendev.org/c/zuul/zuul/+/813826 | ||
| - [zuul/zuul] 814544: Cleanup stale items after refreshing a pipeline https://review.opendev.org/c/zuul/zuul/+/814544 | ||
| - [zuul/zuul] 814570: Reference active change queues in pipeline state https://review.opendev.org/c/zuul/zuul/+/814570 | ||
| - [zuul/zuul] 814571: Update pipeline state when modifying attributes https://review.opendev.org/c/zuul/zuul/+/814571 | ||
| - [zuul/zuul] 814772: Allow passing extra attributes to ZKObject.fromZK https://review.opendev.org/c/zuul/zuul/+/814772 | ||
| - [zuul/zuul] 814862: Bail out when a project moves between connections https://review.opendev.org/c/zuul/zuul/+/814862 | ||
| - [zuul/zuul] 814773: Move re-enqueue to pipeline processing https://review.opendev.org/c/zuul/zuul/+/814773 | ||
| - [zuul/zuul] 814899: Delete old build sets immediately https://review.opendev.org/c/zuul/zuul/+/814899 | ||
| - [zuul/zuul] 815111: Store builds in Zookeeper https://review.opendev.org/c/zuul/zuul/+/815111 | ||
| @jim:acmegating.com | fungi: opendev may be slightly busier now. :) | 18:38 |
| @clarkb:matrix.org | corvus: Looking at graphs memory doesn't seem to be the bottleneck. Seems more CPU limited | 20:38 |
| @jim:acmegating.com | Clark: yep, i think the real benefit of the larger nodes is consistently getting 8 cpus | 20:39 |
| @clarkb:matrix.org | Well I think all of our other nodes are also 8cpus. But the oversubscription for those larger flavors may be better giving us better 8 cpus | 20:39 |
| @jim:acmegating.com | really? i thought they were some 2/4 in there | 20:40 |
| @clarkb:matrix.org | The last cloud we had <8 cpus in was osic | 20:40 |
| @clarkb:matrix.org | And we got away with that because 4 osic cpus were better than 8 cpus elsewhere | 20:41 |
| @jim:acmegating.com | oh. well i can throw a 'remove concurrency' change at the current setup and see what kind of numbers we get | 20:41 |
| @jim:acmegating.com | er, remove concurrency limit i mean | 20:42 |
| @jim:acmegating.com | at least, with concurrency limited, there seems to be a difference on average, but the range does overlap. so i suspect you're right; maybe it's mostly about specific cloud performance and oversubscription. | 20:43 |
| @clarkb:matrix.org | Ya it wouldn't surprise me as I think that can have a major impact on real performance. You get told you have 8 cpus but what really matters is your timeslice on those cpus | 20:43 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815205: DNM: Test 8GB test nodes with no concurrency limit https://review.opendev.org/c/zuul/zuul/+/815205 | 20:44 | |
| @jim:acmegating.com | we'll have all the boxes in the truth table with that | 20:45 |
| @jim:acmegating.com | oof, i forgot to include the https://review.opendev.org/815163 as a depends-on in that stack so it's still failing tests :/ | 20:47 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 20:49 | |
| - [zuul/zuul] 815077: Use larger test nodes and uncap concurrency https://review.opendev.org/c/zuul/zuul/+/815077 | ||
| - [zuul/zuul] 809293: Add an API for ZK-backed objects https://review.opendev.org/c/zuul/zuul/+/809293 | ||
| - [zuul/zuul] 810328: Temporarily enqueue cycle changes for reporting https://review.opendev.org/c/zuul/zuul/+/810328 | ||
| - [zuul/zuul] 811244: Dequeue items after we're done with them https://review.opendev.org/c/zuul/zuul/+/811244 | ||
| - [zuul/zuul] 812750: Add LocalZKContext for job freezing https://review.opendev.org/c/zuul/zuul/+/812750 | ||
| - [zuul/zuul] 812760: Add RepoState object https://review.opendev.org/c/zuul/zuul/+/812760 | ||
| - [zuul/zuul] 813552: Remove Worker class https://review.opendev.org/c/zuul/zuul/+/813552 | ||
| - [zuul/zuul] 813895: Move job_graph attribute to BuildSet https://review.opendev.org/c/zuul/zuul/+/813895 | ||
| - [zuul/zuul] 813913: Serialize JobGraph objects to ZK https://review.opendev.org/c/zuul/zuul/+/813913 | ||
| - [zuul/zuul] 814065: Serialize ProjectMetadata on JobGraph https://review.opendev.org/c/zuul/zuul/+/814065 | ||
| - [zuul/zuul] 814071: Add test_freeze_noop_job https://review.opendev.org/c/zuul/zuul/+/814071 | ||
| - [zuul/zuul] 814069: Remove setBase from job freeze API https://review.opendev.org/c/zuul/zuul/+/814069 | ||
| - [zuul/zuul] 814070: Create Abstract and FrozenJob classes https://review.opendev.org/c/zuul/zuul/+/814070 | ||
| - [zuul/zuul] 814242: Make FrozenJob.updateParentData a static method https://review.opendev.org/c/zuul/zuul/+/814242 | ||
| - [zuul/zuul] 814281: Remove toDict from FrozenJob https://review.opendev.org/c/zuul/zuul/+/814281 | ||
| - [zuul/zuul] 814243: Make FrozenJob a ZKObject https://review.opendev.org/c/zuul/zuul/+/814243 | ||
| - [zuul/zuul] 814329: Implement frozen job serialization/deserialization https://review.opendev.org/c/zuul/zuul/+/814329 | ||
| - [zuul/zuul] 814679: Store FrozenJob data in separate znodes https://review.opendev.org/c/zuul/zuul/+/814679 | ||
| - [zuul/zuul] 815154: Update test_inventory to be ZK-friendly https://review.opendev.org/c/zuul/zuul/+/815154 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Simon Westphahl: | 20:49 | |
| - [zuul/zuul] 809532: Simplified attribute API for ZKObjects https://review.opendev.org/c/zuul/zuul/+/809532 | ||
| - [zuul/zuul] 809414: Make QueueItem a Zookeeper object https://review.opendev.org/c/zuul/zuul/+/809414 | ||
| - [zuul/zuul] 810658: Store pipeline state in Zookeeper https://review.opendev.org/c/zuul/zuul/+/810658 | ||
| - [zuul/zuul] 810920: Store change queues in Zookeeper https://review.opendev.org/c/zuul/zuul/+/810920 | ||
| - [zuul/zuul] 811422: Save and restore bundle with item in Zookeeper https://review.opendev.org/c/zuul/zuul/+/811422 | ||
| - [zuul/zuul] 811955: Pass ZK context to deserialize method of ZKObjects https://review.opendev.org/c/zuul/zuul/+/811955 | ||
| - [zuul/zuul] 812450: Move ZuulMark from configloader to model https://review.opendev.org/c/zuul/zuul/+/812450 | ||
| - [zuul/zuul] 812451: Recursively delete all sub-nodes of ZKObjects https://review.opendev.org/c/zuul/zuul/+/812451 | ||
| - [zuul/zuul] 812466: Only retry ZK operations for Kazoo exceptions https://review.opendev.org/c/zuul/zuul/+/812466 | ||
| - [zuul/zuul] 812452: Store build sets in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812452 | ||
| - [zuul/zuul] 812467: Add support for sharded ZKObjects https://review.opendev.org/c/zuul/zuul/+/812467 | ||
| - [zuul/zuul] 812673: Store RepoFiles for a build set in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812673 | ||
| - [zuul/zuul] 813805: Remove project pipeline config from queue item https://review.opendev.org/c/zuul/zuul/+/813805 | ||
| - [zuul/zuul] 813809: Lookup event class names from global symbol table https://review.opendev.org/c/zuul/zuul/+/813809 | ||
| - [zuul/zuul] 813826: Store and resolve queue item's ahead/behind refs https://review.opendev.org/c/zuul/zuul/+/813826 | ||
| - [zuul/zuul] 814544: Cleanup stale items after refreshing a pipeline https://review.opendev.org/c/zuul/zuul/+/814544 | ||
| - [zuul/zuul] 814570: Reference active change queues in pipeline state https://review.opendev.org/c/zuul/zuul/+/814570 | ||
| - [zuul/zuul] 814571: Update pipeline state when modifying attributes https://review.opendev.org/c/zuul/zuul/+/814571 | ||
| - [zuul/zuul] 814772: Allow passing extra attributes to ZKObject.fromZK https://review.opendev.org/c/zuul/zuul/+/814772 | ||
| - [zuul/zuul] 814862: Bail out when a project moves between connections https://review.opendev.org/c/zuul/zuul/+/814862 | ||
| - [zuul/zuul] 814773: Move re-enqueue to pipeline processing https://review.opendev.org/c/zuul/zuul/+/814773 | ||
| - [zuul/zuul] 814899: Delete old build sets immediately https://review.opendev.org/c/zuul/zuul/+/814899 | ||
| - [zuul/zuul] 815111: Store builds in Zookeeper https://review.opendev.org/c/zuul/zuul/+/815111 | ||
| @jim:acmegating.com | dockerhub is still going to fail a lot of those, but wcyd | 20:49 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815207: Document (and warn about) tag job role checkouts https://review.opendev.org/c/zuul/zuul/+/815207 | 21:45 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!