| *** dave-mccowan has joined #openstack-infra | 00:01 | |
| *** slaweq has quit IRC | 00:03 | |
| *** dave-mcc_ has quit IRC | 00:03 | |
| *** baoli has quit IRC | 00:03 | |
| *** sflanigan has quit IRC | 00:04 | |
| *** pahuang has quit IRC | 00:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: On reconfiguration, re-enqueue items at the same position https://review.openstack.org/516799 | 00:07 |
|---|---|---|
| jeblair | clarkb, mwhahaha: ^ there's a fix | 00:07 |
| *** slaweq has joined #openstack-infra | 00:12 | |
| jeblair | mwhahaha: that bug may have had a significant impact on dwell times in the gate pipeline. basically, any time a change at the head of the queue was failing, the queue behind it was not doing useful work; so we wouldn't effectively start testing the next passing change until the one at the head finished all its jobs. | 00:12 |
| jeblair | mwhahaha: thanks for finding that | 00:12 |
| clarkb | jeblair: I'm reviewing the change and curious if items_to_remove is valid when we check against it in scheduler.py | 00:14 |
| clarkb | jeblair: are we iterating across the list in such a way that we always process item ahead first so it will be in items_to_remove from the last pass through? | 00:14 |
| clarkb | for item in shared_queue.queue: <- is what determines that I think | 00:14 |
| jeblair | clarkb: yeah... it's a little weird -- it's a queue that's also a tree. :| the queue itself is always strictly linear, but your "item ahead" may not be the item immediately preceding you in the queue, it may be one farther up (if the item preceding you is failing). | 00:16 |
| *** Apoorva has quit IRC | 00:16 | |
| jeblair | clarkb: but those two things mean that your "item ahead" does always precede you in the linear queue | 00:16 |
| clarkb | got it so it will be in the items_to_remove list if appropriate | 00:17 |
| jeblair | clarkb: which means that since we iterate over the linear queue, we will have always processed the item ahead before we get to that item. and it will either be in the remove list or enqueued. | 00:17 |
| jeblair | clarkb: at least, that's my reasoning. :) | 00:17 |
| jeblair | (thinking about it graphically -- the order of the items on the status page is the order in the linear queue. but the little lines don't always point to the one ahead, sometimes they skip items to point further up) | 00:18 |
| jeblair | (but the re-enqueue algorithm goes strictly in order down the page) | 00:18 |
| *** links has joined #openstack-infra | 00:20 | |
| clarkb | ya ist an in order traversal of the tree | 00:20 |
| clarkb | just making sure that is the case | 00:21 |
| *** dave-mccowan has quit IRC | 00:21 | |
| *** pahuang has joined #openstack-infra | 00:21 | |
| clarkb | and I've just confirmed assertHistory checks that all the job match and not just a subset so the test lgtm | 00:22 |
| clarkb | actually | 00:22 |
| clarkb | jeblair: does the test need to release a job after the failure to get it started, then reconfigure and make sure it doesn't abort at that point? | 00:23 |
| jeblair | clarkb: i think that's covered by other tests... lemme check | 00:23 |
| jeblair | clarkb: well actually hrm | 00:24 |
| clarkb | my concern is an abort on top of an abort may just be a noop | 00:24 |
| clarkb | maybe a wait for settled before the reconfigure is sufficient | 00:24 |
| clarkb | and just double check that the current state is no longer abort but queued | 00:24 |
| openstackgerrit | Merged openstack-infra/puppet-bup master: Ignore postgres working directory https://review.openstack.org/516798 | 00:24 |
| jeblair | clarkb: job2 for change A crosses the restart | 00:24 |
| jeblair | clarkb: as do both of the jobs for change B | 00:25 |
| jeblair | (both of the re-launched jobs, that is) | 00:25 |
| clarkb | oh I see we do wait until settled afterwards then it is just state asserts | 00:25 |
| clarkb | so if reconfigure did abort anything else they would show up as new aborts | 00:26 |
| clarkb | I've +2'd the change. This was a fun one | 00:26 |
| jeblair | clarkb: yep. since we verify we have exactly 2 aborts -- from the initial pre-reconfigure failure -- i think we're covered. | 00:26 |
| *** edmondsw has joined #openstack-infra | 00:26 | |
| jeblair | clarkb: did you want to look at the override-branch change, https://review.openstack.org/516451 or should i push it through (it has 2x+2) | 00:28 |
| clarkb | jeblair: I think its probably fine and I need to go get my trick or treaters ready | 00:28 |
| clarkb | (the general plan seemed fine) | 00:28 |
| jeblair | clarkb: kk. i'll +W it then | 00:28 |
| *** edmondsw has quit IRC | 00:31 | |
| *** thorst has joined #openstack-infra | 00:31 | |
| *** ansiwen[q] has joined #openstack-infra | 00:33 | |
| fungi | regarding 516799, nice find! | 00:34 |
| *** ansiwen[q] is now known as ansiwen_q | 00:34 | |
| *** ansiwen_q is now known as ansiwen[q] | 00:35 | |
| *** thorst has quit IRC | 00:36 | |
| *** sflanigan has joined #openstack-infra | 00:37 | |
| *** sflanigan has joined #openstack-infra | 00:37 | |
| mtreinish | fungi: I'm back online down under now, and looking at o-h it looks like you got the subunit worker stuff solved (although with some dropped data over the weekend) | 00:38 |
| mtreinish | did it solve the mqtt errors too? | 00:38 |
| *** ansiwen[q] has quit IRC | 00:40 | |
| *** pvaneck has quit IRC | 00:40 | |
| *** xingchao has joined #openstack-infra | 00:42 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add support for override-checkout, deprecate override-branch https://review.openstack.org/516451 | 00:43 |
| mwhahaha | jeblair: cool so maybe that'll help queue times in the future as well? | 00:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: add more zuul executor/merger info https://review.openstack.org/516800 | 00:44 |
| jeblair | mwhahaha: yeah i think so | 00:45 |
| *** edmondsw has joined #openstack-infra | 00:45 | |
| mwhahaha | Nifty | 00:45 |
| *** slaweq has quit IRC | 00:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: fix gerrit event graph https://review.openstack.org/516801 | 00:46 |
| *** hashar has quit IRC | 00:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: drop geard workers gauge https://review.openstack.org/516802 | 00:47 |
| *** hongbin has joined #openstack-infra | 00:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: add more zuul executor/merger info https://review.openstack.org/516800 | 00:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: fix gerrit event graph https://review.openstack.org/516801 | 00:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: drop geard workers gauge https://review.openstack.org/516802 | 00:50 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: On reconfiguration, re-enqueue items at the same position https://review.openstack.org/516799 | 00:50 |
| *** slaweq has joined #openstack-infra | 00:50 | |
| jeblair | i'll restart the scheduler after dinner; it should be installed by then, and it should be less disruptive than waiting till tomorrow | 00:51 |
| *** LindaWang has joined #openstack-infra | 00:51 | |
| *** markvoelker has joined #openstack-infra | 00:52 | |
| *** aeng has quit IRC | 00:54 | |
| fungi | mtreinish: well, puppet-subunit2sql is still not deployable with ensure=>latest, but we can discuss remaining solutions at another time. real load issue was addressed with a performance fix to the gear library | 00:57 |
| fungi | it was also impacting the logstash workers in a similar manner | 00:58 |
| mtreinish | ah, ok | 00:59 |
| *** baoli has joined #openstack-infra | 01:02 | |
| *** huanxie has joined #openstack-infra | 01:02 | |
| *** baoli has quit IRC | 01:06 | |
| ianw | clarkb: do you remember playing with lxcfs on translate.o.o ? | 01:07 |
| *** salv-orlando has quit IRC | 01:08 | |
| clarkb | No | 01:08 |
| *** salv-orlando has joined #openstack-infra | 01:09 | |
| *** xingchao has quit IRC | 01:09 | |
| *** xingchao has joined #openstack-infra | 01:11 | |
| *** bobh has joined #openstack-infra | 01:11 | |
| *** salv-orlando has quit IRC | 01:13 | |
| *** xingchao has quit IRC | 01:16 | |
| *** xingchao has joined #openstack-infra | 01:18 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add new backup host to puppet https://review.openstack.org/516148 | 01:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Backup zuulv3 with bup https://review.openstack.org/516157 | 01:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove ci-backup-rs-ord.openstack.org https://review.openstack.org/516159 | 01:19 |
| jeblair | config-core: https://review.openstack.org/516800 and 2 children will make zuul grafana dashboard shinier and more useful | 01:19 |
| *** xingchao has quit IRC | 01:20 | |
| ianw | infra-root: would everyone agree that the follow host is unused and can be removed | 01:20 |
| ianw | fd76ccd1-3483-4345-9683-9b23afa0b571 | ci-backup-rs-ord.openstack.org | ACTIVE | public=198.101.148.81, 2001:4801:7806:45:4240:b6ff:fe14:19b8; private=10.181.100.2 | | 15 GB Classic v1 | | 01:20 |
| ianw | the real ci-backup-rs-ord.openstack.org being 23.253.166.59 | 01:21 |
| fungi | ianw: yes, that was the older one which got replaced | 01:22 |
| ianw | ok, i'll delete it then, as it has no volumes attached and isn't doing anything | 01:22 |
| *** slaweq has quit IRC | 01:23 | |
| *** slaweq has joined #openstack-infra | 01:25 | |
| *** markvoelker has quit IRC | 01:26 | |
| dmsimard | jeblair: commented on https://review.openstack.org/#/c/516800/ | 01:26 |
| dmsimard | jeblair: hrm, looking again, I'm not sure how it works if you specify a span of >12, maybe it's relative or something | 01:28 |
| *** edmondsw has quit IRC | 01:29 | |
| dmsimard | yup, nevermind that -- looking at places where we had a span of >12, grafana folded the graphs automatically .. for example in the first panel the spans are 2 each and there's 6 graph per row | 01:30 |
| *** xingchao has joined #openstack-infra | 01:30 | |
| *** xingchao has quit IRC | 01:31 | |
| *** bobh has quit IRC | 01:31 | |
| *** bobh has joined #openstack-infra | 01:32 | |
| *** zhurong has joined #openstack-infra | 01:33 | |
| *** bobh has quit IRC | 01:33 | |
| *** smatzek has joined #openstack-infra | 01:33 | |
| *** xinliang has quit IRC | 01:37 | |
| *** xingchao has joined #openstack-infra | 01:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Grafana: add more zuul executor/merger info https://review.openstack.org/516800 | 01:37 |
| *** xingchao has quit IRC | 01:38 | |
| dmsimard | ianw: re: iptables within the bridge subnet | 01:38 |
| dmsimard | ianw: context was from earlier in the day with inc0 | 01:39 |
| *** dingyichen has joined #openstack-infra | 01:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Grafana: fix gerrit event graph https://review.openstack.org/516801 | 01:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Grafana: drop geard workers gauge https://review.openstack.org/516802 | 01:39 |
| dmsimard | he is working on a multi node set up and apparently those rules were necessary -- see https://review.openstack.org/#/c/512779/ | 01:39 |
| *** sflanigan has quit IRC | 01:40 | |
| *** pahuang has quit IRC | 01:40 | |
| dmsimard | the exact details escape me, but we had a discussion around it earlier today.. let me get some links | 01:41 |
| *** liujiong_lj has joined #openstack-infra | 01:41 | |
| ianw | dmsimard: it's ok ... just sometimes (depending on kernel & distro) you have to ensure bridges actually use iptables with a proc flag. but this is ovs so i'm sure it's fine | 01:42 |
| *** xingchao has joined #openstack-infra | 01:42 | |
| dmsimard | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-10-31.log.html#t2017-10-31T16:53:43 | 01:42 |
| *** psachin has joined #openstack-infra | 01:43 | |
| dmsimard | so https://review.openstack.org/#/c/516757/ has +4, just need +W :) | 01:43 |
| dmsimard | and then need +3 on iptables persistence https://review.openstack.org/#/c/513943/ with the matching integration tests https://review.openstack.org/#/c/513934/ | 01:44 |
| *** baoli has joined #openstack-infra | 01:44 | |
| *** cuongnv has joined #openstack-infra | 01:46 | |
| *** xingchao has quit IRC | 01:46 | |
| ianw | dmsimard: you can +w it if you want ... i just thought it was something we should watch a bit to make sure no unintended consequences | 01:47 |
| *** smatzek has quit IRC | 01:48 | |
| *** felipemonteiro_ has joined #openstack-infra | 01:48 | |
| *** smatzek has joined #openstack-infra | 01:48 | |
| *** xinliang has joined #openstack-infra | 01:49 | |
| *** xinliang has quit IRC | 01:49 | |
| *** xinliang has joined #openstack-infra | 01:49 | |
| dmsimard | ianw: pretty safe, inc0 also had his patch use Depends-On to properly set up the rules | 01:50 |
| *** xingchao has joined #openstack-infra | 01:51 | |
| * ianw notes dmsimard's famous last words :) | 01:51 | |
| dmsimard | I mean, we've done worse | 01:51 |
| dmsimard | like merging a _projects typo | 01:51 |
| * dmsimard cough | 01:51 | |
| ianw | indeed :) | 01:51 |
| *** smatzek has quit IRC | 01:52 | |
| *** namnh has joined #openstack-infra | 01:53 | |
| *** kiennt26 has joined #openstack-infra | 01:53 | |
| *** sflanigan has joined #openstack-infra | 01:53 | |
| *** sflanigan has joined #openstack-infra | 01:53 | |
| *** pahuang has joined #openstack-infra | 01:54 | |
| *** xingchao has quit IRC | 01:55 | |
| *** yamahata has quit IRC | 01:56 | |
| *** slaweq has quit IRC | 01:59 | |
| dmsimard | I don't know how you people run your log pruning, our first one just finished in ~7 days and only have a 2TB volume (was at 1.5TB used) | 02:01 |
| dmsimard | I guess our volume I/O kind of sucks, though. | 02:01 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Authorize the multi-node-bridge network in iptables if there's one https://review.openstack.org/516757 | 02:04 |
| *** kiennt26 has quit IRC | 02:06 | |
| *** slaweq has joined #openstack-infra | 02:06 | |
| pabelanger | dmsimard: we'd discussed hashing by date, to make it easier to delete things in bulk, could do the same in RDO | 02:06 |
| *** kiennt26 has joined #openstack-infra | 02:06 | |
| *** baoli has quit IRC | 02:06 | |
| dmsimard | pabelanger: the main problem is that you first crawl all files, delete old ones -- and then delete directories that are empty | 02:06 |
| dmsimard | it's a very safe approach | 02:06 |
| kfox1111 | so, our multinode jobs seem to be running differently in v3. | 02:07 |
| dmsimard | but ideally you wouldn't need to crawl all files -- but I get that it's otherwise kind of hard to base deletion purely on directory mtime | 02:07 |
| kfox1111 | in v2, only the master got the script started on it. in v3, itseems like the slaves also run the main script? | 02:07 |
| dmsimard | kfox1111: what's "main script" ? | 02:07 |
| kfox1111 | is there a way to revert to the old behavior? | 02:07 |
| kfox1111 | we have a script in kolla-kubernetes/tools/setup_gate.sh | 02:07 |
| dmsimard | kfox1111: the jobs run playbooks, you have a host definition in that playbook (ex: "all", "primary", etc), the playbook will run against the hosts matching that definition. | 02:08 |
| kfox1111 | so we need to tweak the new job to run just primary? | 02:08 |
| dmsimard | kfox1111: it depends what you're expecting -- what's the job in question ? | 02:08 |
| kfox1111 | legacy-kolla-kubernetes-deploy-centos-binary-2-ceph-multi | 02:09 |
| kfox1111 | I'm expecting the previous behavior of the script starting only on the master, and it setting the slaves up. | 02:09 |
| dmsimard | kfox1111: this is what runs: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/kolla-kubernetes-deploy-centos-binary-2-ceph-multi/run.yaml | 02:09 |
| dmsimard | kfox1111: I guess the part you don't expect to run on all nodes is "tools/setup_gate.sh deploy centos binary ceph-multi centos-7-2-node shell 2 gate" | 02:10 |
| kfox1111 | ok. cool. should I just tweak hosts: there to be primary? | 02:10 |
| kfox1111 | right. | 02:10 |
| dmsimard | kfox1111: ok, this is what your inventory looks like: http://logs.openstack.org/27/516727/3/check/legacy-kolla-kubernetes-deploy-centos-binary-2-ceph-multi/536da46/zuul-info/inventory.yaml | 02:11 |
| dmsimard | kfox1111: if everything else looks fine, you'd probably just want to split that last task into another play and have it target only "primary" | 02:12 |
| dmsimard | makes sense ? | 02:12 |
| kfox1111 | yeah. I think so. | 02:12 |
| kfox1111 | if I split the task into a different file, do I need to include it somewhere else to get it to work? | 02:12 |
| dmsimard | kfox1111: you don't need to split it into another file, it can be in the same file, just different plays -- like this for example is two plays: https://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base/pre.yaml | 02:13 |
| kfox1111 | ah. ok. thanks. :) | 02:14 |
| dmsimard | kfox1111: we can merge a fix for you inside the legacy openstack-zuul-jobs, but I'd recommend looking into migrating those in-tree inside kolla-k8s so you're able to be independant in playing with your things | 02:14 |
| kfox1111 | yeah. gotta get it fixed quick. then can work on migrating more perminantly. | 02:15 |
| kfox1111 | so, something like this? | 02:18 |
| openstackgerrit | Kevin Fox proposed openstack-infra/openstack-zuul-jobs master: kolla-kubernetes multinode gate fix https://review.openstack.org/516814 | 02:18 |
| *** iyamahat has quit IRC | 02:20 | |
| *** gildub has quit IRC | 02:20 | |
| *** markvoelker has joined #openstack-infra | 02:23 | |
| dmsimard | kfox1111: looks sane to me, added small comment | 02:24 |
| openstackgerrit | Kevin Fox proposed openstack-infra/openstack-zuul-jobs master: kolla-kubernetes multinode gate fix https://review.openstack.org/516814 | 02:27 |
| kfox1111 | k. tweaked. | 02:27 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Re-order Zuul graphs https://review.openstack.org/516816 | 02:28 |
| dmsimard | jeblair: ^ a small nit | 02:29 |
| dmsimard | kfox1111: +2 | 02:29 |
| kfox1111 | cool. thanks for the help. | 02:30 |
| jeblair | dmsimard: ++ | 02:32 |
| jeblair | i love that we have 18.000 mergers | 02:32 |
| dmsimard | that's a lot of mergers | 02:32 |
| dmsimard | Do we really need all those ? | 02:32 |
| *** thorst has joined #openstack-infra | 02:32 | |
| jeblair | dmsimard: it includes the executors too | 02:33 |
| dmsimard | oh | 02:33 |
| jeblair | so we have 10 executors and 8 dedicated mergers | 02:33 |
| jeblair | (the dedicated mergers are small hosts) | 02:33 |
| jeblair | but to answer your question -- i'm not sure -- that's one of the things i want to use these new stats to find out | 02:33 |
| dmsimard | it looks like the executors are pretty loaded at times, it'll be interesting to keep an eye on that | 02:34 |
| dmsimard | are the executors their own zuul merger or something somehow ? or do they feed from just any merger ? | 02:34 |
| jeblair | dmsimard: they are their own. | 02:35 |
| dmsimard | what's the other 8 do ? :) | 02:35 |
| jeblair | dmsimard: speculative configuration loading and verifying that changes can merge | 02:35 |
| dmsimard | oh, makes sense | 02:35 |
| dmsimard | purpose built software written from scratch, it rocks | 02:36 |
| jeblair | i'm going to restart the scheduler now | 02:36 |
| *** thorst has quit IRC | 02:37 | |
| *** slaweq has quit IRC | 02:38 | |
| jeblair | (it looks like the top tripleo change in the gate is dead anyway, so this is probably a net benefit) | 02:38 |
| *** salv-orlando has joined #openstack-infra | 02:39 | |
| *** dhinesh has quit IRC | 02:40 | |
| *** dhinesh has joined #openstack-infra | 02:40 | |
| jeblair | for fun, watch the merge queue graph now | 02:42 |
| jeblair | (zoomed in to 'last 5m') | 02:43 |
| jeblair | that's all the cat jobs needed to get the initial configuration | 02:43 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Make python changes more reliable https://review.openstack.org/516819 | 02:44 |
| jeblair | re-enqueueing changes now | 02:44 |
| *** dhinesh has quit IRC | 02:45 | |
| *** salv-orlando has quit IRC | 02:45 | |
| SamYaple | ive *almost* convinced the team to do zuulv3 for our internal openstack deployment testing (currently there is some syntax checks..... but thats it) | 02:45 |
| dmsimard | jeblair: 3k? wtf? | 02:45 |
| SamYaple | so i might have some hands on operation experincewith it at a bit of scale soon | 02:45 |
| dmsimard | jeblair: is that right ? | 02:46 |
| jeblair | dmsimard: no, more like 4k actually. | 02:46 |
| dmsimard | jeblair: seems like a lot, is there a lot of implicit "jobs" in there or are all those in zuulv3.o.o status page ? | 02:46 |
| pabelanger | jeblair: question, why didn't infracloud-chocolate release ready nodes? | 02:47 |
| dmsimard | SamYaple: keep in mind that zuul v3 is not "released" yet so while it works there might be short deprecations or things like that. | 02:47 |
| jeblair | SamYaple: neat! there are still definitely rough edges for new deployments... and, well, basically no deployment documentation, but we're working on it and i'm sure we'd love your help with both of those. :) | 02:47 |
| dmsimard | SamYaple: what do you plan on deploying it with ? | 02:48 |
| pabelanger | infracloud-vanilla did, but chocolate seems to still have an issue | 02:48 |
| jeblair | dmsimard: those are 'cat' jobs, which are basically, "check out this branch of this repo and give me the zuul.yaml files you find" we have a lot of project*branch combinations, and zuul has to check every one of them to builds its configuration. | 02:48 |
| jeblair | dmsimard: the 18 mergers do that in about 3 minutes. | 02:48 |
| dmsimard | jeblair: oh, I didn't realize what you meant by cat jobs | 02:49 |
| jeblair | the name of that job in gearman is 'merger:cat' | 02:49 |
| *** slaweq has joined #openstack-infra | 02:49 | |
| *** ijw has quit IRC | 02:50 | |
| dmsimard | meow | 02:50 |
| dmsimard | I'm off, see ya tomorrow :) | 02:50 |
| *** ijw has joined #openstack-infra | 02:50 | |
| *** stakeda has joined #openstack-infra | 02:50 | |
| *** ijw has quit IRC | 02:51 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Make preinstall.d more deterministic https://review.openstack.org/516821 | 02:53 |
| jeblair | pabelanger: i don't know. it looks complicated. maybe let's look into it with shrews tomorrow? | 02:53 |
| *** zhurong has quit IRC | 02:54 | |
| pabelanger | sure | 02:54 |
| *** yamahata has joined #openstack-infra | 02:55 | |
| jeblair | pabelanger, Shrews: one thing i see that's wrong is this: | 02:55 |
| *** huanxie has quit IRC | 02:55 | |
| jeblair | | 0000632928 | infracloud-chocolate | nova | centos-7 | None | failed | 00:06:05:31 | locked | None | None | None | None | 22 | nl02.openstack.org-29406-PoolWorker.infracloud-chocolate-main | 100-0000802736 | None | None | | 02:55 |
| jeblair | that's a node just sitting there in failed state, not being deleted. so it's holding the request open. the request is for 2 nodes, one of which is ready. | 02:56 |
| *** markvoelker has quit IRC | 02:57 | |
| jeblair | that node entry should be deleted. it's not actionable. | 02:57 |
| jeblair | nodepool.task_manager.ManagerStoppedException: Manager infracloud-chocolate is no longer running | 02:58 |
| jeblair | perhaps that's why it's not being deleted | 02:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Re-order Zuul graphs https://review.openstack.org/516816 | 02:59 |
| *** dtantsur|afk has quit IRC | 03:01 | |
| *** bobh has joined #openstack-infra | 03:01 | |
| *** gildub has joined #openstack-infra | 03:01 | |
| SamYaple | dmsimard: i plan on deploying it with sheer power of will | 03:02 |
| SamYaple | jeblair: mrhillsman has apparently deployed zuulv3 to some some sucess already | 03:02 |
| *** armax has quit IRC | 03:03 | |
| pabelanger | jeblair: Hmm, I rolled up to 60 max-servers then back to 56 by manually changing nodepool.yaml, possible that done something with our manager for infracloud-chocolate | 03:06 |
| pabelanger | hopefully not | 03:06 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Enable support for Gentoo overlays https://review.openstack.org/516823 | 03:07 |
| *** yamamoto has joined #openstack-infra | 03:08 | |
| *** felipemonteiro_ has quit IRC | 03:10 | |
| *** bobh has quit IRC | 03:13 | |
| *** edmondsw has joined #openstack-infra | 03:14 | |
| *** edmondsw has quit IRC | 03:19 | |
| *** hongbin has quit IRC | 03:19 | |
| *** kiennt26 has quit IRC | 03:20 | |
| *** kiennt26 has joined #openstack-infra | 03:21 | |
| *** slaweq has quit IRC | 03:23 | |
| *** ijw has joined #openstack-infra | 03:24 | |
| *** zhurong has joined #openstack-infra | 03:27 | |
| *** rosmaita has quit IRC | 03:28 | |
| *** slaweq has joined #openstack-infra | 03:30 | |
| *** salv-orlando has joined #openstack-infra | 03:31 | |
| *** thorst has joined #openstack-infra | 03:33 | |
| *** salv-orl_ has joined #openstack-infra | 03:36 | |
| *** thorst has quit IRC | 03:38 | |
| *** salv-orlando has quit IRC | 03:40 | |
| ianw | clarkb: i don't think #pypa channel is logged, but i got some very good info on over-installing system packages just now from dstufft. i'm going to write it all up in that github issue | 03:41 |
| *** salv-orl_ has quit IRC | 03:43 | |
| *** markvoelker has joined #openstack-infra | 03:54 | |
| *** slaweq has quit IRC | 04:02 | |
| *** gcb_ has quit IRC | 04:04 | |
| *** slaweq has joined #openstack-infra | 04:05 | |
| *** gcb_ has joined #openstack-infra | 04:05 | |
| *** coolsvap has joined #openstack-infra | 04:16 | |
| openstackgerrit | gongysh proposed openstack-infra/project-config master: Remove tacker legacy tripleo job https://review.openstack.org/516838 | 04:23 |
| *** gongysh has joined #openstack-infra | 04:23 | |
| *** ijw has quit IRC | 04:25 | |
| *** ykarel has joined #openstack-infra | 04:26 | |
| *** jascott1 has quit IRC | 04:27 | |
| *** jascott1 has joined #openstack-infra | 04:28 | |
| *** markvoelker has quit IRC | 04:28 | |
| *** jascott1 has quit IRC | 04:32 | |
| *** janki has joined #openstack-infra | 04:33 | |
| *** jascott1 has joined #openstack-infra | 04:33 | |
| *** ramishra has joined #openstack-infra | 04:37 | |
| *** gildub has quit IRC | 04:38 | |
| *** slaweq has quit IRC | 04:38 | |
| *** udesale has joined #openstack-infra | 04:39 | |
| *** slaweq has joined #openstack-infra | 04:45 | |
| *** bswartz has quit IRC | 04:48 | |
| *** robled has quit IRC | 04:53 | |
| *** dhinesh has joined #openstack-infra | 04:57 | |
| *** salv-orlando has joined #openstack-infra | 05:03 | |
| *** edmondsw has joined #openstack-infra | 05:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 05:04 |
| *** luzC has quit IRC | 05:05 | |
| *** luzC has joined #openstack-infra | 05:06 | |
| *** edmondsw has quit IRC | 05:07 | |
| *** salv-orlando has quit IRC | 05:07 | |
| *** huanxie has joined #openstack-infra | 05:10 | |
| *** armaan has joined #openstack-infra | 05:12 | |
| *** ijw has joined #openstack-infra | 05:14 | |
| *** ijw has quit IRC | 05:16 | |
| *** dsariel__ has joined #openstack-infra | 05:16 | |
| *** slaweq has quit IRC | 05:18 | |
| *** slaweq has joined #openstack-infra | 05:19 | |
| *** ijw has joined #openstack-infra | 05:22 | |
| ramishra | it seems zuul is voting -1 at times inspite of all jobs finishing successfully. https://review.openstack.org/#/c/516710/, something wrong? | 05:24 |
| *** markvoelker has joined #openstack-infra | 05:24 | |
| *** ijw has quit IRC | 05:27 | |
| *** armaan has quit IRC | 05:29 | |
| *** janki has quit IRC | 05:30 | |
| *** gildub has joined #openstack-infra | 05:33 | |
| AJaeger | ramishra: press the "Toggle CI" button at the bottom to see full information | 05:33 |
| AJaeger | ramishra: I see some errors like MERGER_FAILURE that came from a full disk, this is fixed now. So, please add a "recheck" comment. | 05:34 |
| ramishra | AJaeger: Ok, thanks! | 05:34 |
| *** thorst has joined #openstack-infra | 05:34 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: kolla-kubernetes multinode gate fix https://review.openstack.org/516814 | 05:35 |
| AJaeger | config-core, please review https://review.openstack.org/516397 to fix tarball generation | 05:35 |
| *** sridhar_ram has joined #openstack-infra | 05:36 | |
| *** aeng has joined #openstack-infra | 05:36 | |
| *** ijw has joined #openstack-infra | 05:36 | |
| *** armaan has joined #openstack-infra | 05:37 | |
| *** thorst has quit IRC | 05:39 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Enable support for Gentoo overlays https://review.openstack.org/516823 | 05:43 |
| *** janki has joined #openstack-infra | 05:48 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Add the groundwork for musl profile support https://review.openstack.org/516844 | 05:50 |
| *** cshastri has joined #openstack-infra | 05:50 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Enable support for Gentoo overlays https://review.openstack.org/516823 | 05:53 |
| *** markvoelker has quit IRC | 05:58 | |
| *** markvoelker has joined #openstack-infra | 06:01 | |
| *** huanxie has quit IRC | 06:01 | |
| AJaeger | config-core, please review https://review.openstack.org/516397 to fix tarball generation and https://review.openstack.org/516610 for the requirements propose job | 06:03 |
| *** ijw has quit IRC | 06:04 | |
| *** salv-orlando has joined #openstack-infra | 06:04 | |
| *** markvoelker has quit IRC | 06:06 | |
| *** salv-orlando has quit IRC | 06:08 | |
| ianw | AJaeger: 516397 ... i feel like we discussed that in the meeting? | 06:10 |
| ianw | over removing zuul-cloner from the base job all together | 06:10 |
| AJaeger | ianw: yes, that is related - but we need this now to not break jobs. Once mordred's changes are in, we might be able to get rid of this | 06:12 |
| *** udesale__ has joined #openstack-infra | 06:12 | |
| AJaeger | thanks for reviews, ianw | 06:14 |
| *** udesale has quit IRC | 06:15 | |
| *** udesale__ has quit IRC | 06:18 | |
| *** gongysh has quit IRC | 06:19 | |
| *** udesale__ has joined #openstack-infra | 06:19 | |
| ykarel | Is there some way(apart from parsing status.json) to get the status of a pipeline(For ex: post/release/release-post etc) run in zuulv3 for a particular commit. | 06:20 |
| ykarel | AJaeger, ianw ^^ | 06:20 |
| *** witek has quit IRC | 06:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Upload artifacts only on success https://review.openstack.org/516851 | 06:20 |
| AJaeger | ykarel: not that I'm aware of | 06:21 |
| ianw | yeah i think REST-y apis are on the way | 06:21 |
| *** udesale has joined #openstack-infra | 06:21 | |
| *** threestrands has quit IRC | 06:22 | |
| ykarel | AJaeger, Ok ianw: any link? | 06:22 |
| AJaeger | ianw: what do you think about 516851 ? | 06:23 |
| *** witek has joined #openstack-infra | 06:23 | |
| *** slaweq has quit IRC | 06:23 | |
| *** udesale__ has quit IRC | 06:23 | |
| AJaeger | ykarel: check whether the openstack-infra/zuul repo has any changes for that. Might not be there yet... | 06:23 |
| ykarel | AJaeger, Ok | 06:24 |
| *** slaweq has joined #openstack-infra | 06:25 | |
| *** aeng has quit IRC | 06:25 | |
| ianw | ykarel: i'm thinking of the dashboard stuff mentioned in https://etherpad.openstack.org/p/zuulv3-roadmap | 06:31 |
| *** ihrachys_ has joined #openstack-infra | 06:32 | |
| *** ihrachys has quit IRC | 06:32 | |
| ykarel | ianw, Ok will check, Thanks | 06:33 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Congress legacy jobs https://review.openstack.org/510440 | 06:34 |
| *** kukacz has quit IRC | 06:36 | |
| *** annp has joined #openstack-infra | 06:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Fix periodic neutron jobs https://review.openstack.org/516856 | 06:38 |
| *** gongysh has joined #openstack-infra | 06:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix networking-odl/-bagpipe periodic jobs https://review.openstack.org/516858 | 06:40 |
| AJaeger | ianw: and two more changes for neutron that might be temporary ^ | 06:40 |
| ianw | AJaeger: nit on the whitespace | 06:42 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Fix periodic neutron jobs https://review.openstack.org/516856 | 06:43 |
| AJaeger | ianw: thanks - fixed | 06:43 |
| *** kukacz has joined #openstack-infra | 06:43 | |
| *** dhinesh has quit IRC | 06:44 | |
| *** d0ugal_ has quit IRC | 06:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Increase timeout for requirements propose job https://review.openstack.org/516610 | 06:49 |
| *** edmondsw has joined #openstack-infra | 06:51 | |
| *** links has quit IRC | 06:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Fix periodic neutron jobs https://review.openstack.org/516856 | 06:54 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Rework periodic stable docs jobs https://review.openstack.org/516866 | 06:54 |
| *** d0ugal_ has joined #openstack-infra | 06:55 | |
| *** edmondsw has quit IRC | 06:56 | |
| *** jtomasek has joined #openstack-infra | 06:56 | |
| *** slaweq has quit IRC | 06:59 | |
| *** markvoelker has joined #openstack-infra | 07:02 | |
| *** slaweq has joined #openstack-infra | 07:03 | |
| *** salv-orlando has joined #openstack-infra | 07:05 | |
| *** pgadiya has joined #openstack-infra | 07:07 | |
| *** dhajare has joined #openstack-infra | 07:08 | |
| *** salv-orlando has quit IRC | 07:09 | |
| *** spectr-RH has joined #openstack-infra | 07:12 | |
| *** spectr has quit IRC | 07:13 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Create build-openstack-puppet-tarball https://review.openstack.org/515980 | 07:17 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove publish-openstack-puppet-branch-tarball from post pipeline https://review.openstack.org/515982 | 07:17 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove project-team-guide-publish https://review.openstack.org/514054 | 07:17 |
| *** jascott1 has quit IRC | 07:18 | |
| *** jascott1 has joined #openstack-infra | 07:19 | |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** links has joined #openstack-infra | 07:20 | |
| *** robled has joined #openstack-infra | 07:21 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Update proliant-tools to support Gen10 Proliant servers https://review.openstack.org/509359 | 07:22 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Cleanup ironic-inspector legacy jobs https://review.openstack.org/515738 | 07:22 |
| *** Dinesh_Bhor has joined #openstack-infra | 07:23 | |
| *** jascott1 has quit IRC | 07:23 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /tenants route https://review.openstack.org/503268 | 07:24 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/status route https://review.openstack.org/503269 | 07:25 |
| *** salv-orlando has joined #openstack-infra | 07:25 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs route https://review.openstack.org/503270 | 07:27 |
| *** zhurong has quit IRC | 07:29 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route https://review.openstack.org/466561 | 07:30 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 07:30 |
| *** huanxie has joined #openstack-infra | 07:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{source}/{project}.pub route https://review.openstack.org/502530 | 07:32 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove publish-openstack-puppet-branch-tarball https://review.openstack.org/515984 | 07:34 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/devstack-gate master: Revert "Work around cffi install error" https://review.openstack.org/516872 | 07:35 |
| *** markvoelker has quit IRC | 07:35 | |
| *** thorst has joined #openstack-infra | 07:35 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add os_congress role to openstack-ansible https://review.openstack.org/516874 | 07:39 |
| *** zhurong has joined #openstack-infra | 07:39 | |
| *** thorst has quit IRC | 07:40 | |
| *** jaosorior has joined #openstack-infra | 07:42 | |
| *** rcernin has quit IRC | 07:42 | |
| *** slaweq has quit IRC | 07:43 | |
| *** kiennt26 has quit IRC | 07:44 | |
| *** kiennt26 has joined #openstack-infra | 07:45 | |
| *** sridhar_ram has quit IRC | 07:45 | |
| *** shardy_afk is now known as shardy | 07:47 | |
| *** slaweq has joined #openstack-infra | 07:47 | |
| *** hashar has joined #openstack-infra | 07:48 | |
| evrardjp | it looks like I have to do something with a Zuul administrator: https://review.openstack.org/#/c/516874 ? What is the procedure? | 07:48 |
| *** tesseract has joined #openstack-infra | 07:48 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Move publish-openstack-puppet-branch-tarball into ozj https://review.openstack.org/515981 | 07:50 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Revert "Remove publish-openstack-puppet-branch-tarball from post pipeline" https://review.openstack.org/515983 | 07:51 |
| *** slaweq_ has joined #openstack-infra | 07:51 | |
| AJaeger | evrardjp: read the complete creators guide at http://docs.openstack.org/infra/manual/creators.html - you need to split the change in two | 07:52 |
| evrardjp | thanks | 07:52 |
| *** jbadiapa has quit IRC | 08:02 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add os_congress role to openstack-ansible https://review.openstack.org/516874 | 08:06 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add os_congress role to openstack-ansible https://review.openstack.org/516884 | 08:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix networking periodic jobs https://review.openstack.org/516858 | 08:06 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add os_congress role to openstack-ansible https://review.openstack.org/516874 | 08:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix networking periodic jobs https://review.openstack.org/516858 | 08:07 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add os_congress role to openstack-ansible https://review.openstack.org/516884 | 08:08 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add os_congress role to openstack-ansible https://review.openstack.org/516874 | 08:09 |
| *** martinkopec has joined #openstack-infra | 08:10 | |
| frickler | ianw: AJaeger: is there any reason not to merge https://review.openstack.org/516711 and https://review.openstack.org/516694 afterwards? makes me suspicous to see double +2 but no +W | 08:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix openstack-infra publishing https://review.openstack.org/516010 | 08:12 |
| AJaeger | argh, merge conflict. ianw, clarkb do you want to review again, please? ^ | 08:12 |
| AJaeger | frickler: that's a neutron job - and we wait for neutron liaison to +1 first beforew we +A | 08:12 |
| AJaeger | see https://wiki.openstack.org/wiki/CrossProjectLiaisons#Infra for liaisons we wait for | 08:13 |
| frickler | AJaeger: ah, that makes sense, thanks | 08:13 |
| AJaeger | so, approved the first - and we can approve https://review.openstack.org/#/c/516694/ after recheck | 08:14 |
| AJaeger | frickler: do you want to review https://review.openstack.org/#/c/516010/ ? Had already two +2s and then a merge conflict ;8 | 08:14 |
| * AJaeger will now be offline, back later... | 08:14 | |
| *** armaan has quit IRC | 08:15 | |
| *** e0ne has joined #openstack-infra | 08:17 | |
| *** pcaruana has joined #openstack-infra | 08:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-grenade-dsvm-neutron-nova-next https://review.openstack.org/516711 | 08:17 |
| *** Goneri has joined #openstack-infra | 08:18 | |
| *** slaweq has quit IRC | 08:19 | |
| *** armaan has joined #openstack-infra | 08:19 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Do not run legacy-grenade-dsvm-designate-pdns4 on stable/ocata https://review.openstack.org/516885 | 08:19 |
| *** kjackal_ has quit IRC | 08:20 | |
| *** slaweq has joined #openstack-infra | 08:22 | |
| *** dtantsur has joined #openstack-infra | 08:23 | |
| *** xarses has joined #openstack-infra | 08:23 | |
| *** xarses_ has joined #openstack-infra | 08:26 | |
| *** xarses has quit IRC | 08:28 | |
| *** spectr-RH has quit IRC | 08:32 | |
| *** markvoelker has joined #openstack-infra | 08:32 | |
| *** spectr has joined #openstack-infra | 08:33 | |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add nodejs4-jobs to heat-dashboard project https://review.openstack.org/516889 | 08:33 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add nodejs4-jobs to heat-dashboard project https://review.openstack.org/516889 | 08:34 |
| *** salv-orlando has quit IRC | 08:35 | |
| *** salv-orl_ has joined #openstack-infra | 08:36 | |
| *** kjackal_ has joined #openstack-infra | 08:38 | |
| *** edmondsw has joined #openstack-infra | 08:39 | |
| *** salv-orl_ has quit IRC | 08:40 | |
| *** rcernin has joined #openstack-infra | 08:42 | |
| *** d0ugal_ has quit IRC | 08:42 | |
| *** edmondsw has quit IRC | 08:44 | |
| *** d0ugal has joined #openstack-infra | 08:44 | |
| *** d0ugal has quit IRC | 08:44 | |
| *** d0ugal has joined #openstack-infra | 08:44 | |
| *** dingyichen has quit IRC | 08:49 | |
| *** csomerville has joined #openstack-infra | 08:58 | |
| *** cody-somerville has quit IRC | 08:58 | |
| *** slaweq has quit IRC | 08:59 | |
| *** ccamacho has quit IRC | 09:00 | |
| *** xarses_ has quit IRC | 09:00 | |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** priteau has joined #openstack-infra | 09:03 | |
| *** markvoelker has quit IRC | 09:06 | |
| *** jistr_ is now known as jistr | 09:07 | |
| *** gouthamr has quit IRC | 09:07 | |
| *** panda|ruck is now known as panda|ruck|bbl | 09:08 | |
| yamamoto | docs.o.o doesn't seem to have doc changes in https://review.openstack.org/#/c/515535/ . is it a known issue? | 09:11 |
| *** esberglu_ has joined #openstack-infra | 09:11 | |
| *** esberglu has quit IRC | 09:11 | |
| *** zhurong has quit IRC | 09:16 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/infra-manual master: zuul v3: Include file extension in playbook path https://review.openstack.org/516896 | 09:18 |
| *** derekh has joined #openstack-infra | 09:18 | |
| *** yamamoto has quit IRC | 09:20 | |
| *** ralonsoh has joined #openstack-infra | 09:20 | |
| *** ralonsoh has quit IRC | 09:20 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/status route https://review.openstack.org/503269 | 09:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs route https://review.openstack.org/503270 | 09:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route https://review.openstack.org/466561 | 09:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 09:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{source}/{project}.pub route https://review.openstack.org/502530 | 09:28 |
| *** lucas-afk is now known as lucasagomes | 09:31 | |
| *** gmann is now known as gmann_afk | 09:31 | |
| *** lifeless has quit IRC | 09:32 | |
| *** yamamoto has joined #openstack-infra | 09:34 | |
| *** thorst has joined #openstack-infra | 09:36 | |
| *** nicolasbock has joined #openstack-infra | 09:37 | |
| *** salv-orlando has joined #openstack-infra | 09:37 | |
| *** ccamacho has joined #openstack-infra | 09:37 | |
| *** ralonsoh has joined #openstack-infra | 09:39 | |
| *** thorst has quit IRC | 09:41 | |
| *** yamahata has quit IRC | 09:45 | |
| *** gongysh has quit IRC | 09:47 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 09:52 |
| *** jkilpatr has joined #openstack-infra | 09:56 | |
| *** mat128 has joined #openstack-infra | 09:56 | |
| *** links has quit IRC | 09:57 | |
| *** electrofelix has joined #openstack-infra | 09:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Refactor provider config to driver module https://review.openstack.org/488384 | 09:58 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 09:58 |
| *** kiennt26 has quit IRC | 09:58 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add username to build and upload information https://review.openstack.org/453968 | 09:59 |
| *** alexchadin has joined #openstack-infra | 10:00 | |
| *** LindaWang has quit IRC | 10:00 | |
| *** markvoelker has joined #openstack-infra | 10:03 | |
| *** ykarel is now known as ykarel|lunch | 10:04 | |
| dtantsur | hey folks! is it possible to somehow remove the ironic release from pypi? https://pypi.python.org/pypi/ironic/6.3.0 | 10:06 |
| *** salv-orlando has quit IRC | 10:07 | |
| dtantsur | we were not supposed to publish there, it's probably an artifact of the zuul v3 transition | 10:07 |
| *** salv-orlando has joined #openstack-infra | 10:07 | |
| *** cuongnv has quit IRC | 10:08 | |
| *** jkilpatr has quit IRC | 10:11 | |
| *** liujiong_lj has quit IRC | 10:11 | |
| *** salv-orlando has quit IRC | 10:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 10:12 |
| *** pgadiya has quit IRC | 10:12 | |
| *** namnh has quit IRC | 10:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: builder: do not cleanup image for driver not managing image https://review.openstack.org/516920 | 10:14 |
| *** gridinv has quit IRC | 10:16 | |
| *** LindaWang has joined #openstack-infra | 10:17 | |
| ianw | dtantsur: in a word, i think no. missing releases or things going backwards always ends badly | 10:25 |
| *** pgadiya has joined #openstack-infra | 10:26 | |
| dtantsur | ianw: hmm, then what should we do about it? I don't want people to think that's the only ironic release we ever published.. | 10:26 |
| *** gridinv has joined #openstack-infra | 10:27 | |
| ianw | dtantsur: oh, ok, so it's the only thing ever released ... | 10:29 |
| ianw | and you definitely don't want it there? | 10:30 |
| *** boden has joined #openstack-infra | 10:31 | |
| dtantsur | ianw: I think we don't release servers to pypi, do we? | 10:32 |
| dtantsur | I don't like it, but it seems a (semi-)offical policy | 10:32 |
| ianw | i guess not. i'll delete it via the ui | 10:33 |
| ianw | dtantsur: ok, i think that's back the way it was | 10:34 |
| *** ldnunes has joined #openstack-infra | 10:34 | |
| *** pgadiya has quit IRC | 10:36 | |
| *** markvoelker has quit IRC | 10:37 | |
| *** Goneri has quit IRC | 10:38 | |
| dtantsur | thanks ianw | 10:41 |
| ianw | dtantsur: thanks for pointing it out, i wonder if any others escaped? you're right, having one release in pypi is going to be quite confusing | 10:43 |
| *** pgadiya has joined #openstack-infra | 10:48 | |
| *** smatzek has joined #openstack-infra | 10:50 | |
| *** udesale has quit IRC | 10:50 | |
| *** kjackal_ has quit IRC | 10:52 | |
| *** armaan has quit IRC | 10:52 | |
| *** armaan has joined #openstack-infra | 10:53 | |
| *** Goneri has joined #openstack-infra | 10:53 | |
| *** rfolco_ has joined #openstack-infra | 10:53 | |
| *** makowals has quit IRC | 10:54 | |
| *** pbourke has quit IRC | 10:54 | |
| *** yamamoto has quit IRC | 10:56 | |
| *** pbourke has joined #openstack-infra | 10:56 | |
| *** rcernin has quit IRC | 10:57 | |
| *** hashar is now known as hasharAway | 10:58 | |
| *** gildub has quit IRC | 10:58 | |
| *** sambetts|afk is now known as sambetts | 10:59 | |
| *** yamamoto has joined #openstack-infra | 11:00 | |
| *** lifeless has joined #openstack-infra | 11:02 | |
| *** boden has quit IRC | 11:03 | |
| *** kjackal_ has joined #openstack-infra | 11:04 | |
| odyssey4me | it appears that gerrit has slowed down to a crawl | 11:05 |
| ianw | per mail to infra, i disabled the duplicate account 24821 and disabled it | 11:06 |
| ianw | odyssey4me : give it a minute, i had to flush the caches after deleting a duplicate account | 11:07 |
| *** stakeda has quit IRC | 11:07 | |
| odyssey4me | ah ok, thanks :) | 11:07 |
| *** sshnaidm|afk is now known as sshnaidm | 11:10 | |
| *** tpsilva has joined #openstack-infra | 11:14 | |
| *** huanxie has quit IRC | 11:17 | |
| *** edmondsw has joined #openstack-infra | 11:21 | |
| *** dave-mccowan has joined #openstack-infra | 11:22 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 11:28 |
| *** gildub has joined #openstack-infra | 11:30 | |
| *** e0ne has quit IRC | 11:32 | |
| *** markvoelker has joined #openstack-infra | 11:34 | |
| *** jpich has quit IRC | 11:34 | |
| *** e0ne has joined #openstack-infra | 11:35 | |
| *** jpich has joined #openstack-infra | 11:35 | |
| *** thorst has joined #openstack-infra | 11:37 | |
| fungi | ianw: dtantsur: we can delete them as long as we're reasonably sure nobody is installing it from pypi. what ianw mentions is more of a concern around git tag deletion | 11:38 |
| fungi | or deleting broken releases from pypi in hopes that uses will "roll back" to a previous release | 11:39 |
| *** alexchadin has quit IRC | 11:39 | |
| dtantsur | ack | 11:39 |
| * fungi is not really around for the net couple of days. starting the long journey to sydney shortly | 11:40 | |
| fungi | s/net/next/ | 11:40 |
| dtantsur | safe travel! | 11:40 |
| *** wolverineav has joined #openstack-infra | 11:40 | |
| *** thorst has quit IRC | 11:42 | |
| *** alexchadin has joined #openstack-infra | 11:43 | |
| *** ccamacho has quit IRC | 11:46 | |
| *** jcoufal has joined #openstack-infra | 11:47 | |
| *** ykarel|lunch is now known as ykarel | 11:47 | |
| *** wolverineav has quit IRC | 11:47 | |
| fungi | thanks! | 11:48 |
| *** alexchadin has quit IRC | 11:48 | |
| *** dtantsur is now known as dtantsur|bbl | 11:48 | |
| *** jcoufal_ has joined #openstack-infra | 11:50 | |
| *** dprince has joined #openstack-infra | 11:52 | |
| *** jcoufal has quit IRC | 11:54 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Don't run legacy-grenade-dsvm-neutron* jobs in newton or ocata https://review.openstack.org/516694 | 11:58 |
| *** rhallisey has joined #openstack-infra | 11:59 | |
| *** armaan has quit IRC | 11:59 | |
| *** rhallisey has quit IRC | 11:59 | |
| *** armaan has joined #openstack-infra | 11:59 | |
| *** thorst has joined #openstack-infra | 12:00 | |
| *** rhallisey has joined #openstack-infra | 12:01 | |
| *** lucasagomes is now known as lucas-hungry | 12:04 | |
| frickler | config-core: another grenade on newton-eol fix https://review.openstack.org/516885 | 12:04 |
| *** markvoelker has quit IRC | 12:07 | |
| *** salv-orlando has joined #openstack-infra | 12:09 | |
| *** weshay|PTO is now known as weshay | 12:09 | |
| *** bobh has joined #openstack-infra | 12:13 | |
| *** pgadiya has quit IRC | 12:14 | |
| *** salv-orlando has quit IRC | 12:14 | |
| *** coolsvap has quit IRC | 12:15 | |
| Shrews | jeblair: pabelanger: well, the failed node is not getting deleted because there is no server ID | 12:16 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Rework periodic stable jobs https://review.openstack.org/516866 | 12:17 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove stable periodic docs/py27 jobs https://review.openstack.org/516972 | 12:17 |
| AJaeger | frickler: commented - I prefer to have this on the job definition, like the other grenade jobs have. | 12:18 |
| *** udesale has joined #openstack-infra | 12:18 | |
| AJaeger | fungi, safe travels! | 12:18 |
| Shrews | jeblair: pabelanger: that node (0000632928) was attempting to be launched during the whole ManagerStoppedException episode (beginning around 2017-10-31 20:49) | 12:20 |
| *** jcoufal_ has quit IRC | 12:21 | |
| *** panda|ruck|bbl is now known as panda|ruck | 12:22 | |
| *** jcoufal has joined #openstack-infra | 12:23 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 12:25 |
| *** trown|outtypewww is now known as trown | 12:25 | |
| Shrews | pabelanger: Do these "Creating new ProviderManager" events correspond to you manually editing the nodepool.yaml file? http://paste.openstack.org/show/625202/ | 12:26 |
| Shrews | I don't understand why the task manager thread would not start on reconfiguration though | 12:27 |
| *** catintheroof has joined #openstack-infra | 12:28 | |
| *** salv-orlando has joined #openstack-infra | 12:29 | |
| kazsh | Hi, could you someone infra root check this minor change for heat-dashboard ? https://review.openstack.org/#/c/516889/2 AJaeger: thanks for your review :) | 12:30 |
| *** armaan has quit IRC | 12:31 | |
| *** armaan has joined #openstack-infra | 12:31 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Remove legacy tempest-tox-plugin-sanity-check jobs https://review.openstack.org/516977 | 12:32 |
| *** rlandy has joined #openstack-infra | 12:33 | |
| *** armaan has quit IRC | 12:33 | |
| *** armaan has joined #openstack-infra | 12:33 | |
| *** hemna has joined #openstack-infra | 12:34 | |
| *** bhavik1 has joined #openstack-infra | 12:35 | |
| *** markvoelker has joined #openstack-infra | 12:36 | |
| Shrews | pabelanger: jeblair: I restarted nl02. Doing so released the locks on the infracloud-chocolate "ready" nodes, so it wasn't zuul holding those locks. I'm confused as to what went wrong there. | 12:38 |
| Shrews | infra-root: ^^^^ | 12:38 |
| Shrews | nodepool-launcher on nl02, that is | 12:38 |
| openstackgerrit | Sam Betts proposed openstack-infra/openstack-zuul-jobs master: Update timeout on openstack-tox-compare-cover job https://review.openstack.org/516980 | 12:39 |
| *** bhavik1 has quit IRC | 12:39 | |
| *** gildub has quit IRC | 12:41 | |
| Shrews | pabelanger: i'm seeing quota errors for citycloud, too | 12:42 |
| Shrews | pabelanger: also, "No valid host was found" during server creation on ovh-gra1. Odd | 12:46 |
| *** yamamoto has quit IRC | 12:46 | |
| *** bobh has quit IRC | 12:46 | |
| *** markvoelker has quit IRC | 12:48 | |
| *** markvoelker has joined #openstack-infra | 12:54 | |
| *** gcb_ has quit IRC | 12:57 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Do not run legacy-grenade-dsvm-designate-pdns4 on stable/ocata https://review.openstack.org/516885 | 12:57 |
| *** gcb_ has joined #openstack-infra | 13:00 | |
| *** yamamoto has joined #openstack-infra | 13:02 | |
| *** dhajare has quit IRC | 13:05 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Use >= in instance count comparison to max-servers https://review.openstack.org/516988 | 13:05 |
| Shrews | pabelanger: jeblair: I think that ^^^ was at least one problem with chocolate | 13:06 |
| Shrews | pabelanger: jeblair: i manually deleted some chocolate nodes to get the number of instances below max-servers, and i'm seeing them being used now | 13:06 |
| Shrews | pabelanger: jeblair: i now have to get ready for an eye exam. i may not be looking at a screen for a few hours. | 13:07 |
| *** lucas-hungry is now known as lucasagomes | 13:08 | |
| *** kgiusti has joined #openstack-infra | 13:10 | |
| *** rloo has joined #openstack-infra | 13:10 | |
| *** rosmaita has joined #openstack-infra | 13:12 | |
| *** salv-orlando has quit IRC | 13:14 | |
| *** salv-orlando has joined #openstack-infra | 13:15 | |
| *** baoli has joined #openstack-infra | 13:16 | |
| *** lbragstad has joined #openstack-infra | 13:18 | |
| *** ramishra has quit IRC | 13:18 | |
| *** salv-orlando has quit IRC | 13:19 | |
| *** bobh has joined #openstack-infra | 13:25 | |
| *** smatzek has quit IRC | 13:27 | |
| *** bswartz has joined #openstack-infra | 13:27 | |
| *** lbragstad has quit IRC | 13:30 | |
| *** mat128 has quit IRC | 13:34 | |
| mpeterson | hi, question, if I were to start to define a custom job that's a sibling of devstack or devstack-tempest, where would I start looking to understand what I need to parametize and what extra steps I need to add? | 13:34 |
| *** hasharAway is now known as hashar | 13:35 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add nodejs4-jobs to heat-dashboard project https://review.openstack.org/516889 | 13:38 |
| *** lbragstad has joined #openstack-infra | 13:39 | |
| *** ykarel is now known as ykarel|afk | 13:40 | |
| frickler | mpeterson: shade might give a good example: https://review.openstack.org/500365 | 13:40 |
| mpeterson | frickler: right, the link in the migration manual. is there another reference (ie: manual, where these jobs are defined, etc)? | 13:42 |
| *** mriedem has joined #openstack-infra | 13:47 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add ansible-role-k8s-tripleo https://review.openstack.org/515316 | 13:48 |
| *** armax has joined #openstack-infra | 13:49 | |
| frickler | mpeterson: the job itself is defined in the devstack repo and you can find e.g. the readmes for the roles it uses there. I don't know of any other documentation | 13:50 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-tripleo to zuul.d https://review.openstack.org/515317 | 13:50 |
| *** ykarel|afk has quit IRC | 13:50 | |
| *** esberglu_ has quit IRC | 13:51 | |
| *** Goneri has quit IRC | 13:51 | |
| *** esberglu has joined #openstack-infra | 13:51 | |
| mpeterson | frickler: okey, thanks | 13:51 |
| *** Goneri has joined #openstack-infra | 13:53 | |
| *** alexchadin has joined #openstack-infra | 13:54 | |
| *** eharney has joined #openstack-infra | 13:55 | |
| *** kiennt26 has joined #openstack-infra | 13:55 | |
| *** esberglu has quit IRC | 13:56 | |
| *** kiennt26 has quit IRC | 13:57 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 13:57 |
| *** kiennt26 has joined #openstack-infra | 13:57 | |
| *** hongbin has joined #openstack-infra | 13:58 | |
| *** ralonsoh has quit IRC | 13:59 | |
| *** tesseract has quit IRC | 14:06 | |
| *** mat128 has joined #openstack-infra | 14:06 | |
| *** mat128 has quit IRC | 14:08 | |
| *** esberglu has joined #openstack-infra | 14:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use periodic-stable-jobs https://review.openstack.org/516996 | 14:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Use generic stable periodic jobs https://review.openstack.org/516997 | 14:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove now obsolete periodic-stable templates https://review.openstack.org/516998 | 14:12 |
| *** armaan has quit IRC | 14:13 | |
| *** janki has quit IRC | 14:13 | |
| *** janki has joined #openstack-infra | 14:13 | |
| *** armaan has joined #openstack-infra | 14:13 | |
| mwhahaha | so i think i found a new issue with zuul. it seems that i managed to bypass the check queue | 14:13 |
| AJaeger | mwhahaha: how did you do that? | 14:14 |
| mwhahaha | i have no idea | 14:14 |
| mwhahaha | but 515884,1 failed check but is now currently showing up as failed in gate | 14:14 |
| mwhahaha | or rather it seems that the failure in check was ignored | 14:15 |
| jeblair | Shrews: right, but a failed node with no server id should just be deleted from the zk database. there's no reason to keep it around. | 14:15 |
| jeblair | Shrews: i imagine it would be except that it was locked. so there's a path where a node launch can fail, and the node can stay locked, preventing it from being deleted. | 14:16 |
| jeblair | Shrews: maybe because of the manager being stopped or restarted | 14:16 |
| AJaeger | infra-root, http://git.openstack.org/cgit/openstack/ansible-role-k8s-tripleo exists but it's not listed in http://git.openstack.org/cgit and https://review.openstack.org/#/c/515317/3 gives a " The project "openstack/ansible-role-k8s-tripleo" was not found." Could somebody check what's going on, please? | 14:17 |
| jeblair | AJaeger: it needs to be listed in project-config/zuul/main.yaml | 14:17 |
| *** armaan has quit IRC | 14:18 | |
| AJaeger | jeblair: argh ;( I would have expected it to be there, missed that in project creation review ;( | 14:18 |
| AJaeger | thanks | 14:18 |
| AJaeger | jeblair: please check https://review.openstack.org/#/c/515884/ - has -1 from Zuul but enters gate | 14:18 |
| *** ykarel|afk has joined #openstack-infra | 14:19 | |
| mwhahaha | AJaeger: jeblair: I added to ether pad under 'check job ignored so patch went to the gate even when check job failed' | 14:19 |
| AJaeger | jeblair: it's listed there - but flaper87 is too quick, it just merged 30 mins ago.. So, all good - we need to wait ;9 | 14:22 |
| *** rbrndt has joined #openstack-infra | 14:22 | |
| AJaeger | jeblair: is https://review.openstack.org/516997 the way you suggest to do periodic jobs? | 14:24 |
| *** dhinesh has joined #openstack-infra | 14:26 | |
| flaper87 | AJaeger: danke :D | 14:26 |
| * flaper87 is impatient | 14:27 | |
| AJaeger | mwhahaha: I think in this case it's fine - it had a +1 from Zuul, got approved and just entered gate directly - but also restarted check. | 14:27 |
| *** rfolco has quit IRC | 14:27 | |
| AJaeger | mwhahaha: no, it's not - checking timing ;( | 14:27 |
| AJaeger | mwhahaha: ignore my comments above, please | 14:28 |
| mwhahaha | AJaeger: what comments? ;) | 14:28 |
| AJaeger | mwhahaha: exactly | 14:28 |
| AJaeger | flaper87: you need to rebase again... | 14:29 |
| jeblair | mwhahaha, AJaeger: it entered the gate pipelite at 21:50:02; at that time it had a +1 from zuul, a code-review +2 and a workflow +1 | 14:29 |
| mwhahaha | jeblair: but if it's abanonded and restored shouldn't it clear all and reset? | 14:29 |
| jeblair | the comment from zuul from the check-tripleo pipeline is the triggering event (despite check-tripleo not voting, gerrit emitted the event with a verified+1 vote since that was still zuul's current vote from the last check run) | 14:30 |
| AJaeger | jeblair: this morning zuul commented "Verified Starting gate jobs." - so that was far later | 14:30 |
| rloo | hi, has anyone noticed that the build-openstack-releasenotes job doesn't always fire? Missing: https://review.openstack.org/#/c/516060/ | 14:30 |
| jeblair | mwhahaha: apparently not? | 14:30 |
| rloo | done: https://review.openstack.org/#/c/495385/ | 14:30 |
| *** ykarel|afk is now known as ykarel | 14:31 | |
| rloo | ?? | 14:31 |
| jeblair | AJaeger: yes, i believe it does that after the first job starts running | 14:31 |
| *** felipemonteiro_ has joined #openstack-infra | 14:31 | |
| AJaeger | rloo: why should it build release notes if nothing is touched? We only build releasenotes if certain files are changed. | 14:31 |
| mwhahaha | jeblair: seems like a dangerous flow that might allow for flakey changes to get merged | 14:31 |
| *** udesale has quit IRC | 14:31 | |
| AJaeger | rloo: and none of them are changed, so this is fine... | 14:32 |
| rloo | AJaeger: OH. why do we always build docs? Maybe something changes there all the time... | 14:32 |
| rloo | AJaeger: phew! | 14:32 |
| *** felipemonteiro__ has joined #openstack-infra | 14:32 | |
| rloo | AJaeger: I was going crazy, trying to figure out if it had to do with all this zuul v3 migration! | 14:32 |
| AJaeger | rloo: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n447 | 14:33 |
| jeblair | mwhahaha: i guess it depends on the circumstances. abandoning and restoring an approved change is pretty rare. | 14:33 |
| rloo | AJaeger: thx for explaining! | 14:33 |
| AJaeger | rloo: we always build docs since docs include often autogenerated code... | 14:33 |
| mwhahaha | jeblair: i still think it shouldn't gate until all check jobs have reported | 14:33 |
| * mwhahaha shrugs | 14:33 | |
| jeblair | mwhahaha: they had | 14:34 |
| jeblair | from the previous run | 14:34 |
| rloo | AJaeger: ah, that makes sense. thx again! | 14:34 |
| AJaeger | rloo: so, go ahead and let's get https://review.openstack.org/#/c/516060/ in ;) | 14:34 |
| rloo | AJaeger: yup, working on that, and other zuul v3 related patches for some of the ironic projects ;) | 14:35 |
| jeblair | AJaeger: 516997 lgtm | 14:35 |
| AJaeger | rloo: yeah, seen you've been busy... | 14:35 |
| *** yamamoto has quit IRC | 14:35 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: DO NOT MERGE - Testing specific DLRN hash tag Also testing dlrn_hash_tag_newest set to specific hash. https://review.openstack.org/516624 | 14:35 |
| mwhahaha | jeblair: imho an abandon/restore should completely clear previous state but ok. still seems like an odd interaction | 14:36 |
| *** felipemonteiro_ has quit IRC | 14:36 | |
| rloo | AJaeger: lesson learned, never ask for a volunteer to do things, you might get picked to do it yourself. Hence my involvement :D | 14:36 |
| *** yamamoto has joined #openstack-infra | 14:37 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/openstack-zuul-jobs master: Remove periodic tripleo ci jobs https://review.openstack.org/517002 | 14:37 |
| sshnaidm | weshay, EmilienM mwhahaha ^^ | 14:37 |
| AJaeger | jeblair: thanks. So, I'll have some dancing around between project-config and ozj in front of me with the help of the config-cores then ;) | 14:37 |
| EmilienM | sshnaidm: thx | 14:37 |
| EmilienM | clarkb, fungi: fyi a (little) effort to remove useless CI jobs https://review.openstack.org/517002 | 14:38 |
| jeblair | mwhahaha: yeah, that's not unreasonable. but i think in some cases, the status-quo is reasonable as well. | 14:38 |
| * AJaeger hugs rloo | 14:39 | |
| mwhahaha | jeblair: sure. I don't think it's critical but it does impact the system as a whole because it did cause a failure in the gate which reset the queue. | 14:40 |
| * rloo blushes | 14:40 | |
| jeblair | EmilienM: those jobs don't use the tripleo cloud do they? | 14:41 |
| *** yamamoto has quit IRC | 14:42 | |
| EmilienM | jeblair: they were using openstack-infra/multinode and now fully migrated to RDO cloud | 14:42 |
| EmilienM | jeblair: so we don't need them anymore in openstack zuuml | 14:42 |
| EmilienM | zuul even | 14:42 |
| *** cshastri has quit IRC | 14:43 | |
| jeblair | EmilienM: i guess thought you were moving the ones that needed the tripleo hardware, not all of them | 14:43 |
| AJaeger | config-core, please review the stack starting at https://review.openstack.org/516856 | 14:43 |
| EmilienM | jeblair: *all* periodic jobs move to RDO CI, because we want to run them every 4 hours | 14:43 |
| *** tesseract has joined #openstack-infra | 14:43 | |
| *** gongysh has joined #openstack-infra | 14:44 | |
| *** gongysh has quit IRC | 14:44 | |
| jeblair | EmilienM: oh. did our teams ever talk about this? like an email thread or meeting or something i can catch up on? | 14:44 |
| EmilienM | jeblair: I would ask that quesiton to the tripleo CI team, weshay ^ | 14:45 |
| jeblair | i've been pretty distracted lately and i haven't been able to keep up with everything | 14:45 |
| EmilienM | jeblair: let me look in my mailbox | 14:45 |
| EmilienM | jeblair: I don't think it was directly communicated to *you* but announced in the weekly (or so) "CI Squad Meeting Summary" | 14:46 |
| EmilienM | oh wait | 14:46 |
| EmilienM | it was discussed in [openstack-dev] [infra][tripleo] initial discussion for a new periodic pipeline | 14:46 |
| EmilienM | jeblair: http://lists.openstack.org/pipermail/openstack-dev/2017-March/113471.html | 14:47 |
| EmilienM | jeblair: conclusion was: move to RDO cloud | 14:47 |
| *** psachin has quit IRC | 14:48 | |
| *** alexchadin has quit IRC | 14:50 | |
| weshay | thanks sshnaidm | 14:52 |
| jeblair | EmilienM: indeed that seems to be the case. i'm not sure i agree with that approach. i'd much rather openstack projects continue to use the openstack ci system as much as they are able. third-party ci should only be for things that can not physically be tested in the public system due to hardware or software licensing limitations. | 14:52 |
| EmilienM | jeblair: it was clear infra wan't willing to run periodic jobs every 4 hours | 14:53 |
| weshay | jeblair, there are two seperate moves here.. The first was to move *all* the periodic jobs out of upstream infra | 14:53 |
| EmilienM | wasn't* | 14:53 |
| weshay | the second was to move ovb jobs ( the ones requiring tripleo hardware ) to third party | 14:54 |
| jeblair | EmilienM: i read early in the thread that it seemed folks were fine with it. just not doing so at the expense of the check pipeline. ie, keeping them at low precedence. | 14:55 |
| weshay | jeblair, right.. which is why ovb jobs will be third party voting | 14:56 |
| *** LindaWang has quit IRC | 14:57 | |
| *** LindaWang has joined #openstack-infra | 14:57 | |
| *** cody-somerville has joined #openstack-infra | 14:58 | |
| *** gordc has joined #openstack-infra | 14:58 | |
| *** csomerville has quit IRC | 14:59 | |
| *** vhosakot has joined #openstack-infra | 14:59 | |
| gordc | hey folks, anyone know how/where grenade figures out which stable branch to pull in for the 'old' clone. | 14:59 |
| jeblair | EmilienM: anyway, it's probably not useful to relitigate this now. i'm not going to -2 any changes or anything. but i would like to discuss this further once everyone is once again available. i'd like to make sure we all know where this is heading. | 14:59 |
| weshay | jeblair, like a group discussion or 1-1? | 15:02 |
| jeblair | weshay: group i think? | 15:05 |
| weshay | agree.. jeblair and pabelanger should be in on that as well | 15:05 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 15:05 |
| *** LindaWang has quit IRC | 15:06 | |
| mriedem | clarkb: i see the indexing behind by message here changed, looks like things were fixed? http://status.openstack.org/elastic-recheck/ - any idea what was wrong? | 15:07 |
| mriedem | thought it was some key issue last week | 15:08 |
| *** martinkopec has quit IRC | 15:08 | |
| *** mat128 has joined #openstack-infra | 15:08 | |
| *** qadaemon has joined #openstack-infra | 15:15 | |
| *** qadaemon has quit IRC | 15:15 | |
| *** ijw has joined #openstack-infra | 15:15 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for nova bug 1729371 https://review.openstack.org/517008 | 15:17 |
| openstack | bug 1729371 in OpenStack Compute (nova) "ResourceTracker races to delete instance allocations before instance is mapped to a cell" [High,Triaged] https://launchpad.net/bugs/1729371 | 15:17 |
| *** smatzek has joined #openstack-infra | 15:19 | |
| *** catintheroof has quit IRC | 15:21 | |
| *** dhinesh has quit IRC | 15:24 | |
| *** dhinesh has joined #openstack-infra | 15:25 | |
| *** markvoelker_ has joined #openstack-infra | 15:26 | |
| *** eharney has quit IRC | 15:27 | |
| AJaeger | config-core, please review https://review.openstack.org/516397 to fix tarball generation and https://review.openstack.org/516851 to upload artifacts only on success | 15:29 |
| *** markvoelker has quit IRC | 15:29 | |
| sshnaidm | jeblair, AJaeger if I remove periodic job, should I do changes in openstack-infra/project-config as well? https://review.openstack.org/#/c/517002/ | 15:31 |
| *** eharney has joined #openstack-infra | 15:31 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Retiring old OpenStack-Ansible projects https://review.openstack.org/517012 | 15:32 |
| *** dhinesh has quit IRC | 15:36 | |
| evrardjp | could anyone tell me what job replaced the ones explained here https://docs.openstack.org/infra/manual/drivers.html#step-1-end-project-gating ? | 15:36 |
| *** dhinesh has joined #openstack-infra | 15:36 | |
| evrardjp | looks like merge-check isn't defined anymore | 15:37 |
| pabelanger | mgagne: we are seeing some quota issues in nodepool, do you mind confirming the limits again? We are currently setup for 190 | 15:37 |
| evrardjp | I can replace the docs, but I don't know what this was before | 15:37 |
| *** spectr has quit IRC | 15:37 | |
| pabelanger | mgagne: seem to be hitting 160 instances, then quota error | 15:38 |
| *** ykarel is now known as ykarel|afk | 15:38 | |
| *** hashar has quit IRC | 15:39 | |
| *** yamamoto has joined #openstack-infra | 15:41 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for nova bug 1729371 https://review.openstack.org/517008 | 15:42 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test ci https://review.openstack.org/517016 | 15:42 |
| openstack | bug 1729371 in OpenStack Compute (nova) pike "ResourceTracker races to delete instance allocations before instance is mapped to a cell" [High,Triaged] https://launchpad.net/bugs/1729371 | 15:42 |
| sshnaidm | pabelanger, if I remove periodic job, should I do changes in openstack-infra/project-config as well? https://review.openstack.org/#/c/517002/ | 15:43 |
| *** dprince has quit IRC | 15:43 | |
| *** ykarel|afk has quit IRC | 15:44 | |
| clarkb | mriedem: it was a combination of things, new log format needed new queries in e-r to check lag, also we were double indexing things which made it really slow, and the gearman server had enough jobs in it that it was having performance issues | 15:44 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Update timeout on openstack-tox-compare-cover job https://review.openstack.org/516980 | 15:44 |
| pabelanger | sshnaidm: you need to remove a job first from the pipeline, in project-config zuul.d folder | 15:45 |
| clarkb | evrardjp: merge-check has been replaced by the system-required template | 15:45 |
| evrardjp | yeah just found it out with a good git blame | 15:45 |
| evrardjp | just at the second | 15:45 |
| evrardjp | I will fix that an the docs | 15:45 |
| *** hashar has joined #openstack-infra | 15:45 | |
| evrardjp | thanks clarkb! | 15:45 |
| evrardjp | you confirmed my investigation! | 15:45 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Retiring old OpenStack-Ansible projects https://review.openstack.org/517012 | 15:46 |
| clarkb | gordc: it is hardcoded in devstack-gate. Basically depedending on current branch and type of upgrade it just has a list of what is yhe branch ahead and what is the branch behind | 15:46 |
| mriedem | clarkb: ah i see https://github.com/openstack-infra/elastic-recheck/commit/69b588cb2b3847754f01f350d7aa9f4c2b476801 nice catch | 15:46 |
| *** yamamoto has quit IRC | 15:47 | |
| *** dtantsur|bbl is now known as dtantsur | 15:47 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Remove tripleo-ci periodic jobs https://review.openstack.org/517019 | 15:48 |
| clarkb | mriedem: I think something is still off in finding uncategorised failures | 15:49 |
| *** ralonsoh has joined #openstack-infra | 15:50 | |
| gordc | clarkb: hmm.. ok i'll take a closer look. thanks for the pointer. | 15:51 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/openstack-zuul-jobs master: Remove periodic tripleo ci jobs https://review.openstack.org/517002 | 15:51 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/infra-manual master: Replace merge-check by system-required https://review.openstack.org/517020 | 15:52 |
| evrardjp | AJaeger: ^ docs thing related to one of your patches :p | 15:53 |
| evrardjp | thought you might be interested . | 15:53 |
| *** ansiwen[q] has joined #openstack-infra | 15:54 | |
| *** yamahata has joined #openstack-infra | 15:54 | |
| *** ansiwen[q] is now known as ansiwen_q | 15:54 | |
| *** ansiwen_q is now known as ansiwen[q] | 15:55 | |
| mnaser | evrardjp: left a small note | 15:56 |
| *** ansiwen[q] is now known as ansiwen_q | 15:56 | |
| *** janki has quit IRC | 15:56 | |
| *** Sukhdev has joined #openstack-infra | 15:57 | |
| evrardjp | mnaser: https://github.com/openstack-infra/project-config/blob/master/zuul.d/pipelines.yaml#L336 | 15:57 |
| evrardjp | you're right! | 15:57 |
| evrardjp | consistency... | 15:57 |
| mnaser | :p | 15:57 |
| *** coolsvap has joined #openstack-infra | 15:58 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/infra-manual master: Replace merge-check by system-required https://review.openstack.org/517020 | 15:58 |
| evrardjp | mnaser: updated! | 15:58 |
| mnaser | evrardjp: super unrelated but im wondering if the template: - name: system-required format is valid at all | 15:59 |
| *** ansiwen_q is now known as ansiwen[q] | 15:59 | |
| evrardjp | mnaser: I don't think it is. Did I let that? | 15:59 |
| *** dhinesh has quit IRC | 15:59 | |
| evrardjp | darn! | 15:59 |
| evrardjp | I did | 15:59 |
| mnaser | evrardjp: but that was there beforehand | 15:59 |
| mnaser | in infra-docs | 15:59 |
| evrardjp | yeah | 16:00 |
| evrardjp | my bad | 16:00 |
| mnaser | oh you made the change beforehand? :p | 16:00 |
| evrardjp | <-- silly hobbit! | 16:00 |
| evrardjp | no, I just left my brain in the garage. Give me a sec I'll fetch it. | 16:00 |
| *** xarses has joined #openstack-infra | 16:00 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/infra-manual master: Replace merge-check by system-required https://review.openstack.org/517020 | 16:01 |
| *** ijw has quit IRC | 16:02 | |
| *** jascott1 has joined #openstack-infra | 16:03 | |
| mnaser | evrardjp: im so sorry i should have mentioned it earlier but s/template:/templates:/ and i wont bother you anymore :p | 16:04 |
| evrardjp | I should pay more attention | 16:04 |
| evrardjp | let me fix that quick | 16:04 |
| *** jascott1 has quit IRC | 16:04 | |
| *** ansiwen[q] is now known as ansiwen_q | 16:05 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Retiring old OpenStack-Ansible projects https://review.openstack.org/517012 | 16:05 |
| AJaeger | mnaser: could you review the stack starting at https://review.openstack.org/#/c/516856 , please? | 16:06 |
| mnaser | AJaeger: sure one sec | 16:06 |
| AJaeger | clarkb: care to give https://review.openstack.org/#/c/516010/ - the infra-publications change another +2, please? I had to rebase... | 16:06 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/infra-manual master: Replace merge-check by system-required https://review.openstack.org/517020 | 16:06 |
| *** iyamahat has joined #openstack-infra | 16:07 | |
| *** catintheroof has joined #openstack-infra | 16:07 | |
| *** kiennt26 has quit IRC | 16:09 | |
| *** trown is now known as trown|lunch | 16:10 | |
| *** ldnunes has quit IRC | 16:10 | |
| *** dhajare has joined #openstack-infra | 16:11 | |
| *** Apoorva has joined #openstack-infra | 16:12 | |
| AJaeger | evrardjp: thanks for the doc update. On the retirement change, I'd like to see PTL +1 | 16:12 |
| evrardjp | that's me :p | 16:12 |
| evrardjp | I can +1 my own patch if you want. | 16:12 |
| evrardjp | :p | 16:12 |
| AJaeger | evrardjp: not needed in that case ;) | 16:13 |
| AJaeger | evrardjp: indeed you are - congrats. Sorry, missed that change... | 16:13 |
| evrardjp | AJaeger: haha! I like the fact that you have checked :) | 16:13 |
| evrardjp | and thanks! | 16:14 |
| mnaser | AJaeger: i have reviewed the stack up to the point where we wait for merges to review the rest | 16:15 |
| AJaeger | mnaser: ok, will recheck once those are in and then ask again ;) | 16:15 |
| mnaser | :) | 16:16 |
| AJaeger | mnaser, could I trouble you for review https://review.openstack.org/516397 to fix tarball generation and https://review.openstack.org/516851 to upload artifacts only on success as well, please? | 16:16 |
| * AJaeger has too many open reviews... | 16:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add ansible-role-k8s-tripleo to zuul.d https://review.openstack.org/515317 | 16:16 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix periodic neutron jobs https://review.openstack.org/516856 | 16:17 |
| *** salv-orlando has joined #openstack-infra | 16:17 | |
| *** gouthamr has joined #openstack-infra | 16:20 | |
| *** Sukhdev has quit IRC | 16:20 | |
| *** salv-orlando has quit IRC | 16:21 | |
| *** jschlueter|znc is now known as jschlueter | 16:21 | |
| mnaser | AJaeger: dmsimard got the second one, first one took a lil while to understand why neutron and horizon were there :p | 16:22 |
| dmsimard | you beat me to the first one, I was already aware of context :) | 16:22 |
| AJaeger | mnaser: yeah, hope we get rid of those soon... | 16:22 |
| Shrews | jeblair: ah, that failed node was locked because it was part of the nodeset for the request that caused chocolate to be stuck | 16:22 |
| *** efried is now known as efried_rollin | 16:22 | |
| *** ldnunes has joined #openstack-infra | 16:23 | |
| Shrews | jeblair: all nodes (ready or failed) get unlocked together after the request is fulfilled | 16:23 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Rework periodic stable jobs https://review.openstack.org/516866 | 16:23 |
| jeblair | Shrews: it failed with a 'manager not running' error iirc | 16:24 |
| gordc | clarkb: you recall anything changing in the past day that would change how branches in grenade are set up? | 16:24 |
| *** ijw has joined #openstack-infra | 16:24 | |
| gordc | looking at a patch which previously was correctly pulling in stable branch but is now pulling in master: http://paste.openstack.org/show/625233/ | 16:24 |
| *** Rockyg has joined #openstack-infra | 16:25 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config master: Remove legacy networking-cisco jobs https://review.openstack.org/517035 | 16:25 |
| Shrews | jeblair: right | 16:25 |
| *** csomerville has joined #openstack-infra | 16:25 | |
| clarkb | gordc: not that recently | 16:25 |
| clarkb | last time changes should have been made was around release time | 16:26 |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/project-config master: Remove legacy-neutron-dsvm-api from Neutron https://review.openstack.org/516724 | 16:26 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/openstack-zuul-jobs master: Remove legacy networking-cisco jobs https://review.openstack.org/517036 | 16:27 |
| pabelanger | Shrews: re: pastebin, yes that lines up around the time I decresed the max-servers back to 56 | 16:27 |
| clarkb | gordc: there may have been a cleanup of newton? | 16:27 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config master: Remove legacy networking-cisco jobs https://review.openstack.org/517035 | 16:27 |
| gordc | clarkb: weird. the devstacklog says it's pulling in master but at some point it switches to stable... (and that switch behaviour seems to have disappeared now) | 16:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack/requirements to publish-openstack-python-branch-tarball https://review.openstack.org/516397 | 16:28 |
| gordc | clarkb: possibly? hmm.. i'll take a closer look at logs | 16:28 |
| pabelanger | Shrews: citycloud errors are known, we have 7 stuck nodes. Going to open a support ticket now | 16:28 |
| *** cody-somerville has quit IRC | 16:28 | |
| *** csomerville has quit IRC | 16:28 | |
| dmsimard | mnaser: https://review.openstack.org/#/c/515974/ needs rebased | 16:29 |
| jeblair | Shrews: so the launcher thread should have been able to delete that node and release the request, yeah? but it just left it there in failed+locked | 16:30 |
| Shrews | pabelanger: ok. we should get that nodepool fix in so that doesn't happen again. i don't know the exact circumstances that led to so many chocolate nodes being locked, but that fix is certainly the cause of the entire pool being paused | 16:30 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Replace merge-check by system-required https://review.openstack.org/517020 | 16:30 |
| openstackgerrit | Merged openstack-infra/project-config master: Upload artifacts only on success https://review.openstack.org/516851 | 16:30 |
| mnaser | dmsimard: of course it did a conflict was just a matter of time in a change like that :p | 16:30 |
| mnaser | i'll get to it shortly | 16:30 |
| Shrews | jeblair: chocolate was paused trying to satisfy a min-ready request (forces a new launch), but we were over capacity of max-servers. the min-ready request should have been declined | 16:31 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/openstack-zuul-jobs master: Remove legacy networking-cisco jobs https://review.openstack.org/517036 | 16:31 |
| jeblair | Shrews: yeah, but it also should have been able to reclaim the failed node | 16:31 |
| Shrews | jeblair: not until the request it was for had been fulfilled or failed | 16:32 |
| openstackgerrit | Merged openstack-infra/project-config master: Retiring old OpenStack-Ansible projects https://review.openstack.org/517012 | 16:32 |
| jeblair | Shrews: yes, though it should have been able to do that -- it had exhausted its 3 launch attempts for the second node. | 16:33 |
| evrardjp | thanks AJaeger and dmsimard :) | 16:33 |
| dmsimard | clarkb: so I don't have any ideas for the .gz thing. Happy to try your approach. | 16:34 |
| dmsimard | clarkb: oh it already merged | 16:36 |
| dmsimard | the logstash queue is still growing endlessly though ? | 16:36 |
| *** thiagolib has joined #openstack-infra | 16:36 | |
| clarkb | I dont think so e-r says we are only 5 hours behind | 16:36 |
| clarkb | but I havent checked gearman itself | 16:37 |
| *** slaweq has joined #openstack-infra | 16:37 | |
| *** dhinesh has joined #openstack-infra | 16:37 | |
| *** felipemonteiro__ has quit IRC | 16:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix networking periodic jobs https://review.openstack.org/516858 | 16:40 |
| *** slaweq has quit IRC | 16:42 | |
| pabelanger | okay, citycloud emailed, I've cc'd fungi and clarkb | 16:43 |
| *** dsariel__ has quit IRC | 16:44 | |
| openstackgerrit | Merged openstack-infra/project-config master: Cleanup legacy-grenade-dsvm-neutron* branch restrictions https://review.openstack.org/516705 | 16:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Refactor nonha multinode ooq into project template https://review.openstack.org/515972 | 16:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Drop irrelevant-files from TripleO jobs https://review.openstack.org/515973 | 16:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Drop files from TripleO jobs https://review.openstack.org/515985 | 16:45 |
| *** catintheroof has quit IRC | 16:45 | |
| *** ykarel|afk has joined #openstack-infra | 16:46 | |
| *** wolverineav has joined #openstack-infra | 16:47 | |
| andreaf | jeblair: hi - anything holding https://review.openstack.org/#/c/509233/ back? I'd really like to see that moving forward | 16:51 |
| *** catintheroof has joined #openstack-infra | 16:52 | |
| *** pcaruana has quit IRC | 16:52 | |
| fungi | jeblair: EmilienM: weshay: catching up while i have a few minutes waiting for a plane... what pushed tripleo to move their periodic jobs to third-party is that they wanted them _reliably_ run every 4 hours, and we wouldn't guarantee that under eavy gate/check volume those might get get queued for many hours until resources were available to run them | 16:53 |
| *** dhinesh has quit IRC | 16:53 | |
| fungi | mriedem: we merge a performance fix for the gear library which sped up processing for our logstash and subunit workers | 16:53 |
| fungi | oh, clarkb also answered | 16:53 |
| EmilienM | fungi: to be fair, RDO cloud is so busy now that our jobs don't run every 4h | 16:54 |
| EmilienM | fungi: have a safe flight | 16:54 |
| clarkb | dmsimard: just checked and the job count is up to ~125k | 16:54 |
| clarkb | dmsimard: so ya not entirely sure we've sorted out all the slowness | 16:54 |
| AJaeger | mnaser: do you want to continue with the periodic changes? https://review.openstack.org/#/c/516972/ is ready now | 16:55 |
| mnaser | AJaeger: sure one sec | 16:55 |
| clarkb | pabelanger: will you be spinning up new logstash workers today? | 16:55 |
| AJaeger | thanks, mnaser | 16:55 |
| clarkb | dmsimard: I think ^ is the next step in trying to address the backlog | 16:56 |
| *** ijw has quit IRC | 16:56 | |
| dmsimard | clarkb: how many workers do we have ? On phone right but I remember seeing like 18 on cacti | 16:56 |
| pabelanger | clarkb: yes! I was able to get logstash-worker16 online | 16:56 |
| * fungi is hating the terrible keyboard on this burner laptop | 16:56 | |
| fungi | dmsimard: 20 logstash workers last time i counted | 16:57 |
| clarkb | ya 20 is what we ran with for a long time, but we are/were down to 15 | 16:57 |
| *** gyee has joined #openstack-infra | 16:58 | |
| clarkb | worker 14 has a full / so I am looking into that now | 16:58 |
| *** dave-mccowan has quit IRC | 16:58 | |
| *** panda|ruck is now known as panda|ruck|bbl | 16:58 | |
| dmsimard | Wow, and they're not able to keep up ? | 16:58 |
| clarkb | currently not keeping up | 16:59 |
| dmsimard | What's the bottleneck ? | 16:59 |
| clarkb | I think its a combo of network io and regexing ~billion strings a day | 17:00 |
| clarkb | ES itself seems fine and will take what we throw at it, its the stuff upstream of it | 17:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Add os_congress role to openstack-ansible https://review.openstack.org/516874 | 17:00 |
| *** sbezverk has quit IRC | 17:01 | |
| clarkb | crm114 is what filled the disk | 17:02 |
| *** e0ne has quit IRC | 17:02 | |
| clarkb | it looks like the explosion of tripleo log files is at least partially to blame | 17:02 |
| *** dave-mccowan has joined #openstack-infra | 17:03 | |
| clarkb | 15G extra | 17:03 |
| *** openstackgerrit has quit IRC | 17:03 | |
| clarkb | 3.7G undercloud | 17:03 |
| clarkb | EmilienM: we are indexing all of the tripleo log files individually now? | 17:03 |
| clarkb | I think that may account for the increase in volume | 17:03 |
| clarkb | and is enough unique files that crm114 is exploding its disk use | 17:04 |
| clarkb | I think we need 13MB per file we index with crm114 | 17:04 |
| clarkb | and since the tables crm114 uses are fixed that can't grow, what grows is adding more files to it | 17:04 |
| *** wolverineav has quit IRC | 17:05 | |
| *** openstackgerrit has joined #openstack-infra | 17:05 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove stable periodic docs/py27 jobs https://review.openstack.org/516972 | 17:05 |
| clarkb | looks like we also have log files that are based on hostname so we just create crm114 dirs for them and then never reuse them | 17:05 |
| clarkb | eg ceph-mon-centos-7-ovh-gra1-0000632103 | 17:06 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Improve error handling in webapp /keys https://review.openstack.org/517053 | 17:06 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use generic stable periodic jobs https://review.openstack.org/516997 | 17:06 |
| *** jcoufal has quit IRC | 17:07 | |
| clarkb | I'm going to push up a patch to disable crm114 then we'll need to delete all teh crm114 content and restart the workers | 17:07 |
| *** jcoufal has joined #openstack-infra | 17:07 | |
| EmilienM | clarkb: I'm not aware of any change on that regard | 17:08 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Retire old OSA projects (phase 3) https://review.openstack.org/517054 | 17:08 |
| *** wolverineav has joined #openstack-infra | 17:09 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add more logstash-workers https://review.openstack.org/517055 | 17:10 |
| *** mdbooth has quit IRC | 17:10 | |
| *** ari[m] has quit IRC | 17:10 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Disable crm114 on the logstash worker nodes https://review.openstack.org/517056 | 17:10 |
| clarkb | EmilienM: I thought you had a change to make use of the regex system? | 17:11 |
| clarkb | infra-root review on 517056 is appreciated | 17:11 |
| *** ijw has joined #openstack-infra | 17:12 | |
| *** ari[m] has joined #openstack-infra | 17:12 | |
| pabelanger | clarkb: fungi: 517055 brings logstash-workers back online, doing logstash-worker18 now | 17:12 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add additional launcher logging https://review.openstack.org/517057 | 17:12 |
| clarkb | EmilienM: ya one sec I see it on disk | 17:12 |
| *** armaan has joined #openstack-infra | 17:12 | |
| clarkb | EmilienM: Id5b9be6130e4e4506418a254a5e57e105fa4d03c that one | 17:12 |
| *** mdbooth has joined #openstack-infra | 17:12 | |
| clarkb | EmilienM: we may need to revert that, stablize logstash then work to find a way to make that viable | 17:12 |
| EmilienM | clarkb: we wanted to save log files for all services in /var/log | 17:13 |
| clarkb | EmilienM: yes I know, but there are a few bugs in there I see | 17:13 |
| EmilienM | ok cool, revert it | 17:13 |
| clarkb | EmilienM: first is that there are many logs that aren't console logs getting picked up | 17:13 |
| *** lucasagomes is now known as lucas-afk | 17:13 | |
| clarkb | another is we are indexing unique filenames like ceph-mon-centos-7-ovh-gra1-0000632103/something/or/other which means crm114 will just add those to disk until it runs out | 17:14 |
| fungi | better to do like devstack-gate and move/copy them to common paths and names before retrieval, i suppose | 17:15 |
| clarkb | it also looks like we may be indexing redundant copies of some logs, let me find some example paths | 17:16 |
| *** dhajare has quit IRC | 17:16 | |
| *** caphrim007 has joined #openstack-infra | 17:16 | |
| clarkb | /var/lib/crm114/logs/subnode-2/var/log/extra/docker/containers/cinder_scheduler/log/cinder/cinder-manage_log_txt_gz and /var/lib/crm114/logs/subnode-2/var/log/extra/docker/containers/logrotate_crond/log/containers/cinder/cinder-manage_log_txt_gz | 17:17 |
| clarkb | on the crm114 side I think we want to treat those as being the same log so have a single entry for cinder-manage_log_txt_gz | 17:17 |
| clarkb | on the tripleo side you should make sure we aren't copying multiple copies of the same log | 17:17 |
| weshay | re: containers probably | 17:17 |
| *** salv-orlando has joined #openstack-infra | 17:18 | |
| weshay | yup containers | 17:18 |
| pabelanger | logstash-worker18.o.o online, moving to 19 | 17:20 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Revert "tripleo: index /var/log/*.log.txt files" https://review.openstack.org/517058 | 17:21 |
| clarkb | there is a revert so that we can stabilize | 17:21 |
| *** jcoufal has quit IRC | 17:21 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test image build https://review.openstack.org/517059 | 17:21 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo: fix release jobs layout https://review.openstack.org/517060 | 17:22 |
| *** jcoufal has joined #openstack-infra | 17:22 | |
| *** salv-orlando has quit IRC | 17:22 | |
| *** jpich has quit IRC | 17:22 | |
| frickler | gordc: stable/newton no longer exists, thus running grenade on stable/ocata won't work for most projects anymore, so you probably have to disable that job like in https://review.openstack.org/516885 | 17:23 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo: fix release jobs layout https://review.openstack.org/517060 | 17:23 |
| EmilienM | can someone familiar with zuul v3 jobs for release review that ^ please | 17:23 |
| clarkb | EmilienM: weshay I think step 0 on unreverting is to produce a complete list of all the files that match those regexes on today's jobs. From that we can look at what we want to get indexed, apply the correct tags to those, and then make any changes to crm114 necessary to actually do that reliably | 17:23 |
| weshay | k | 17:23 |
| weshay | that sounds reasonable | 17:23 |
| AJaeger | mnaser, frickler, could you review now https://review.openstack.org/#/c/516996/ to move forward with periodic change, please? | 17:26 |
| *** salv-orlando has joined #openstack-infra | 17:26 | |
| weshay | pabelanger, remove tripleo periodic upstream jobs https://review.openstack.org/#/c/517019/ | 17:28 |
| gordc | frickler: hmm... is this new? i don't remember ever disabling grenade on N-2 gate. that said, our grenade does little to nothing so i'm ok with this.lol | 17:28 |
| openstackgerrit | Merged openstack-infra/system-config master: Include retired projects/deliverables in elections https://review.openstack.org/488543 | 17:29 |
| openstackgerrit | Merged openstack-infra/system-config master: Use foundation member IDs in invitations list https://review.openstack.org/488555 | 17:29 |
| clarkb | EmilienM: commented on the release change | 17:30 |
| *** slaweq has joined #openstack-infra | 17:31 | |
| *** trown|lunch is now known as trown | 17:34 | |
| *** camunoz has joined #openstack-infra | 17:34 | |
| *** hashar is now known as hasharAway | 17:35 | |
| weshay | pabelanger, ovb migration is blocked atm by a newton bug on rdo-cloud | 17:35 |
| weshay | we're getting the patch on soon | 17:35 |
| pabelanger | weshay: okay, left comment and +2, not to block. Thought it was only OVB periodic moving too | 17:35 |
| pabelanger | weshay: k | 17:35 |
| frickler | gordc: I think this may have happened more globally with zuul v2, matching any *grenade* job or the like, but I haven't checked in detail | 17:36 |
| *** mdbooth has quit IRC | 17:36 | |
| *** ari[m] has quit IRC | 17:36 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Drop TripleO jobs from Puppet modules https://review.openstack.org/516794 | 17:36 |
| *** ari[m] has joined #openstack-infra | 17:37 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Drop TripleO jobs from Puppet modules https://review.openstack.org/516794 | 17:37 |
| EmilienM | clarkb: how do you explain this error then? http://logs.openstack.org/28/517028/1/check/openstack-tox-validate/2b00b1e/job-output.txt.gz#_2017-11-01_16_18_40_185727 | 17:38 |
| *** mdbooth has joined #openstack-infra | 17:38 | |
| gordc | frickler: cool cool. i'll disable and quote you if we need someone to blame. :) | 17:38 |
| clarkb | EmilienM: I don't know, just pointing out if you look at pypi there is what appears to be long histories of publishing there | 17:39 |
| clarkb | EmilienM: could be a bug in the release scripts themselves? | 17:39 |
| gordc | frickler: actually, yeah i notice grenade was active on stable/newton patches either. works for me. | 17:39 |
| gordc | wasnt* | 17:39 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add os_congress role to openstack-ansible https://review.openstack.org/516884 | 17:40 |
| *** ykarel|afk has quit IRC | 17:40 | |
| frickler | gordc: actually, it is strange that newton-eol has been tagged 8 days ago, but your job succeeded 2 days ago and fails only now | 17:40 |
| gordc | frickler: yeah, that's what was confusing me. i guess it was getting lucky. i'm still not sure how it manages to switch to correct stable when it clones master. | 17:41 |
| EmilienM | smcginnis, dhellmann : if you're around, can you help me to figure out this one please? http://logs.openstack.org/28/517028/1/check/openstack-tox-validate/2b00b1e/job-output.txt.gz#_2017-11-01_16_18_40_185727 | 17:41 |
| clarkb | I've stopped the lgostash worker processes and am in the process of cleaning out dirs to free disk space | 17:42 |
| fungi | newton eol is sort of a rolling thing, not all repos have been eol'd yet as they're working to land some final fixes and get a last point release tagged first | 17:42 |
| EmilienM | clarkb: sounds like all tripleo projects have wrong release-type, I'm looking at the script now | 17:42 |
| EmilienM | clarkb: honestly I don't know why we push tripleo project son pypi. Removing this jobs makes sense to me | 17:43 |
| EmilienM | clarkb: we don't deploy from source and always use rpms from delorean. | 17:44 |
| *** rosmaita has quit IRC | 17:46 | |
| frickler | fungi: but does deletion of stable/newton happen later than tagging newton-eol? I can't see when the former happen, the later was 8 days ago here http://git.openstack.org/cgit/openstack/ceilometer | 17:46 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo: fix release jobs layout https://review.openstack.org/517060 | 17:46 |
| EmilienM | clarkb: commit message update | 17:46 |
| frickler | fungi: still the jobs 2 day ago has stable/newton pushed http://logs.openstack.org/43/516443/1/check/grenade-dsvm-ceilometer/71dfb24/job-output.txt.gz#_2017-10-30_21_21_26_473918 | 17:46 |
| frickler | fungi: while todays hasn't http://logs.openstack.org/43/516443/2/check/grenade-dsvm-ceilometer/5b742ca/job-output.txt.gz#_2017-11-01_14_53_33_307088 | 17:47 |
| pabelanger | clarkb: we're ready to land: https://review.openstack.org/517055/ all new logstash-workers are online. | 17:47 |
| frickler | fungi: or did the zuul-mergers cache the branch after it has been deleted? | 17:47 |
| fungi | frickler: one possibility, i suppose, is that zuul executors aren't pruning branch removals | 17:47 |
| pabelanger | fungi: https://review.openstack.org/517055/ too when you have a moment | 17:48 |
| clarkb | pabelanger: thanks | 17:48 |
| frickler | fungi: yes, that would explain the delayed failures. | 17:48 |
| *** hasharAway is now known as hasharDinner | 17:48 | |
| clarkb | 01 02 03 are running workers against, I may have to restart them depending on if puppet races me and starts running them with crm114 again | 17:48 |
| clarkb | config-core can I get a second review on https://review.openstack.org/#/c/517058/ please? | 17:48 |
| clarkb | pabelanger: did you accept their new ssh keys so that my crm disablement will get applied? | 17:49 |
| pabelanger | clarkb: just doing that now | 17:50 |
| *** sshnaidm is now known as sshnaidm|afk | 17:50 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo: fix release jobs layout https://review.openstack.org/517060 | 17:50 |
| *** rosmaita has joined #openstack-infra | 17:51 | |
| pabelanger | #status log logstash-worker16.o.o to logstash-worker20.o.o now online and SSH keys accepted | 17:52 |
| openstackstatus | pabelanger: finished logging | 17:52 |
| frickler | config-core: /me has another newton-eol fix needing a second review: https://review.openstack.org/516885 | 17:52 |
| openstackgerrit | Merged openstack-infra/system-config master: Disable crm114 on the logstash worker nodes https://review.openstack.org/517056 | 17:53 |
| clarkb | 2017-11-01 17:38:24,822 Retrieving: http://logs.openstack.org/81/516581/2/gate/legacy-heat-dsvm-functional-convg-mysql-lbaasv2/1c9accf/logs/libvirt/libvirtd.txt.gz?level=INFO 2017-11-01 17:39:06,906 Pushing 988855 log lines. | 17:54 |
| clarkb | that ^ is also going to be very slow | 17:54 |
| clarkb | looks like it may have actually OOM'd it | 17:55 |
| *** dhinesh has joined #openstack-infra | 17:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "tripleo: index /var/log/*.log.txt files" https://review.openstack.org/517058 | 17:57 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Do not run legacy-grenade-dsvm-designate-pdns4 on stable/ocata https://review.openstack.org/516885 | 17:58 |
| *** felipemonteiro_ has joined #openstack-infra | 17:59 | |
| *** felipemonteiro__ has joined #openstack-infra | 18:00 | |
| *** ralonsoh has quit IRC | 18:01 | |
| *** felipemonteiro_ has quit IRC | 18:04 | |
| *** panda|ruck|bbl is now known as panda|ruck | 18:04 | |
| pabelanger | infra-root: can I get a +3 on https://review.openstack.org/517055/ I'd like to finish that up this afternoon. Happy to self-approve too | 18:04 |
| AJaeger | clarkb, config-core, please review https://review.openstack.org/#/c/516010/ to fix infra-publish jobs - I had to rebase ;( | 18:04 |
| *** salv-orl_ has joined #openstack-infra | 18:05 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add BaseSource.getProjectReadonly and refactor https://review.openstack.org/517067 | 18:06 |
| clarkb | ok the jenkins log worker processes are running on 01-05 and I expect they will be fine from now on. For 06-20 they are not running and I am just going to clear out the crm data so that when puppet runs it will get it all started there | 18:07 |
| * AJaeger will move fixed comments from zuulv3-issues to zuulv3-fixed now | 18:07 | |
| *** salv-orlando has quit IRC | 18:08 | |
| EmilienM | clarkb: could you please take a second look? https://review.openstack.org/#/c/517060/ | 18:09 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Use >= in instance count comparison to max-servers https://review.openstack.org/516988 | 18:10 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add additional launcher logging https://review.openstack.org/517057 | 18:10 |
| clarkb | AJaeger: I've +2'd it but won't approve as I actually need to finish packing to catch my flight later today | 18:12 |
| AJaeger | dmsimard: shall I https://review.openstack.org/#/c/513943 +A now? Wanted to wait until you're around | 18:12 |
| pabelanger | clarkb: okay, | 18:13 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Unpause a declined request https://review.openstack.org/517068 | 18:13 |
| pabelanger | err | 18:13 |
| clarkb | EmilienM: you probably want the release team to sign off on that? Are we sure no one is consuming it from pypi? | 18:13 |
| pabelanger | clarkb: okay, I've just +3'd 517055 to open up firewalls on logstash-workers | 18:13 |
| clarkb | just beacuse red hat doesn't doesn't mean someone else isn't | 18:13 |
| dmsimard | AJaeger: sure, hadn't seen your comment | 18:13 |
| AJaeger | clarkb: safe travels. do you mind if I approve to move forward here? | 18:13 |
| clarkb | AJaeger: ya I thinkyou can approve if you are able to monitor it | 18:14 |
| EmilienM | clarkb: I pinged dhellmann and smcginnis but not reply yet | 18:14 |
| EmilienM | clarkb: I'm happy to restore pypi later if that makes sense | 18:14 |
| EmilienM | but for now, we need to release things and I don't see anyone around using pypi for tripleo projects, we ship everything via rpm, upstream and downstream | 18:14 |
| AJaeger | clarkb: it needs content from openstack-infra/publications to merge - will keep an eye on that one... | 18:14 |
| EmilienM | we don't have any tooling deploing tripleo from pypi | 18:15 |
| *** ansiwen_q is now known as ansiwen[q] | 18:17 | |
| EmilienM | mwhahaha: can you sign-off https://review.openstack.org/#/c/517060/ please? | 18:18 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove tripleo-ci periodic jobs https://review.openstack.org/517019 | 18:20 |
| *** dhinesh_ has joined #openstack-infra | 18:22 | |
| AJaeger | pabelanger, mnaser, dmsimard, next change for periodic jobs - could one of you review https://review.openstack.org/#/c/516996/ , please? | 18:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix openstack-infra publishing https://review.openstack.org/516010 | 18:22 |
| *** Sukhdev has joined #openstack-infra | 18:23 | |
| *** dhinesh has quit IRC | 18:25 | |
| *** dprince has joined #openstack-infra | 18:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire old OSA projects (phase 3) https://review.openstack.org/517054 | 18:25 |
| *** yamahata has quit IRC | 18:26 | |
| *** Sukhdev_ has joined #openstack-infra | 18:28 | |
| *** tesseract has quit IRC | 18:28 | |
| *** Sukhdev_ has quit IRC | 18:29 | |
| *** Sukhdev has quit IRC | 18:29 | |
| *** iyamahat has quit IRC | 18:29 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add more logstash-workers https://review.openstack.org/517055 | 18:29 |
| *** vhosakot has quit IRC | 18:29 | |
| *** vhosakot_ has joined #openstack-infra | 18:30 | |
| pabelanger | ^ okay, firewalls should be reloading here in 30mins, I can then start processes on the new workers | 18:30 |
| *** ansiwen[q] has quit IRC | 18:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy jobs from Barbican https://review.openstack.org/510390 | 18:30 |
| *** bnemec has quit IRC | 18:35 | |
| *** Sukhdev has joined #openstack-infra | 18:35 | |
| clarkb | pabelanger: make sure crm is disabled first | 18:36 |
| clarkb | pabelanger: files in /etc/logprocessor | 18:36 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Use cgit more in contributor listing https://review.openstack.org/517077 | 18:37 |
| pabelanger | clarkb: yah, puppet just ran on those hosts and updated configs | 18:38 |
| pabelanger | so, next run firewalls will open | 18:38 |
| pabelanger | then we can start | 18:38 |
| clarkb | cool | 18:38 |
| clarkb | I'm starting workers on 06-15 really quickly | 18:38 |
| clarkb | then back to packing | 18:38 |
| pabelanger | clarkb: direct flight or down to LA? | 18:39 |
| AJaeger | config-core, a few job removals from openstack-zuul-jobs for review: https://review.openstack.org/#/c/512635/ https://review.openstack.org/#/c/516011/ https://review.openstack.org/#/c/51700 | 18:39 |
| fungi | boarding my first flight now. looking forward to seeing a lot of you next week! | 18:39 |
| pabelanger | fungi: ++ | 18:40 |
| clarkb | pabelanger: down to LA then direct from there | 18:40 |
| clarkb | I actually get to run into fungi in LA later tonight | 18:40 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Do not add invalid projets via the /keys API https://review.openstack.org/517078 | 18:41 |
| pabelanger | clarkb: nice | 18:41 |
| *** rwsu has quit IRC | 18:42 | |
| * SpamapS is filled with FOMO not going to Sydney :-/ | 18:42 | |
| *** e0ne has joined #openstack-infra | 18:44 | |
| mwhahaha | EmilienM: i want to say we do publish to pypi for like tripleo-common and instack-undercloud | 18:44 |
| mwhahaha | EmilienM: will need to verify tho. I think we need it for the unit tests or something silly like that | 18:44 |
| *** salv-orl_ has quit IRC | 18:45 | |
| EmilienM | mwhahaha: indeed, this is a valid use case that I missed | 18:46 |
| *** salv-orlando has joined #openstack-infra | 18:46 | |
| EmilienM | mwhahaha: not sure why release scripts see these projects as python-service though | 18:46 |
| mwhahaha | EmilienM: no idea about that part | 18:46 |
| *** MasterOfBugs has joined #openstack-infra | 18:46 | |
| *** coolsvap has quit IRC | 18:47 | |
| *** iyamahat has joined #openstack-infra | 18:48 | |
| clarkb | we are at a 145k logstash gearamn job right now. I think we compare against that watermark in 24 hours or so | 18:48 |
| clarkb | we'll have new workers, disks are no longer full, and so on | 18:48 |
| *** salv-orlando has quit IRC | 18:50 | |
| *** dtantsur is now known as dtantsur|afk | 18:53 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Use playbook from tripleo-quickstart-extras for OVB https://review.openstack.org/513508 | 18:53 |
| *** sbezverk has joined #openstack-infra | 18:57 | |
| EmilienM | clarkb: I think I found out | 18:58 |
| EmilienM | clarkb: I'm preparing a new patchset | 18:58 |
| EmilienM | clarkb: we're missing release-type in all stable deliverables in openstack/releases for tripleo project | 18:59 |
| *** Apoorva has quit IRC | 19:00 | |
| *** Apoorva has joined #openstack-infra | 19:01 | |
| *** salv-orlando has joined #openstack-infra | 19:01 | |
| *** sambetts is now known as sambetts|afk | 19:02 | |
| *** rlandy is now known as rlandy|brb | 19:03 | |
| *** openstackgerrit has quit IRC | 19:03 | |
| *** bnemec has joined #openstack-infra | 19:03 | |
| AJaeger | dmsimard: thanks! | 19:03 |
| dmsimard | ? | 19:03 |
| *** openstackgerrit has joined #openstack-infra | 19:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Replace legacy-tox-build with native job https://review.openstack.org/517081 | 19:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Add os_congress role to openstack-ansible https://review.openstack.org/516884 | 19:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy networking-cisco jobs https://review.openstack.org/517035 | 19:04 |
| AJaeger | dmsimard: thanks for reviewing my long list of requests ;) | 19:05 |
| *** Apoorva has quit IRC | 19:05 | |
| dmsimard | AJaeger: oh I'm just trying to churn through a bit of review backlog in no particular order :/ | 19:06 |
| *** yamahata has joined #openstack-infra | 19:06 | |
| AJaeger | dmsimard: ah,ok - yeah, there's a long list but we're getting there ;) | 19:06 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove periodic tripleo ci jobs https://review.openstack.org/517002 | 19:06 |
| *** jascott1 has joined #openstack-infra | 19:07 | |
| dmsimard | AJaeger: you're a machine | 19:07 |
| dmsimard | (an awesome machine) | 19:07 |
| *** efried_rollin is now known as efried | 19:07 | |
| AJaeger | ;) | 19:08 |
| dmsimard | definitely can't keep up | 19:08 |
| dmsimard | I thought we cached the cirros image in the nodepool image, did I hallucinate that ? | 19:08 |
| dmsimard | oh wait, nevermind, it's through the devstack thing | 19:09 |
| * dmsimard was looking in the wrong place | 19:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use periodic-stable-jobs https://review.openstack.org/516996 | 19:11 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Persist iptables rules https://review.openstack.org/513943 | 19:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Merge project-config linter jobs https://review.openstack.org/517086 | 19:15 |
| AJaeger | config-core, I think we can merge the linter jobs now for project-config ^ | 19:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove project-config-zuul-jobs-linters https://review.openstack.org/517087 | 19:16 |
| *** Rockyg has quit IRC | 19:16 | |
| *** catintheroof has quit IRC | 19:16 | |
| *** ijw has left #openstack-infra | 19:17 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo: fix release jobs layout https://review.openstack.org/517060 | 19:17 |
| *** owalsh is now known as owalsh_biab | 19:18 | |
| *** catintheroof has joined #openstack-infra | 19:18 | |
| AJaeger | dmsimard: could you review https://review.openstack.org/517087 as well, please? Then my periodic job cleanup is completely done... I rechecked already but am confident it will pass ;) | 19:18 |
| dmsimard | AJaeger: hunting something down right now, will look after | 19:19 |
| AJaeger | dmsimard: sure - thanks | 19:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo: fix release jobs layout https://review.openstack.org/517060 | 19:19 |
| *** rlandy|brb is now known as rlandy | 19:21 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Also add logstash-worker works to cacti https://review.openstack.org/517088 | 19:21 |
| AJaeger | infra-root, I just see that on nova, the post job publish-openstack-python-branch-tarball has error "error" without log file, see http://logs.openstack.org/ef/ef487de22204fe391e0f536a43ed30df95374540/post/ | 19:22 |
| AJaeger | We recently merged https://review.openstack.org/#/c/516397/ - is there an error in it? | 19:22 |
| AJaeger | or can you get a logfile, please? | 19:22 |
| *** Goneri has quit IRC | 19:25 | |
| *** Goneri has joined #openstack-infra | 19:26 | |
| *** jcoufal_ has joined #openstack-infra | 19:27 | |
| *** jcoufal has quit IRC | 19:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "Add openstack/requirements to publish-openstack-python-branch-tarball" https://review.openstack.org/517093 | 19:30 |
| *** jcoufal has joined #openstack-infra | 19:31 | |
| AJaeger | infra-root, config-core, that's the short-term fix - but would like to know what the real problem is ^ | 19:31 |
| *** hasharDinner has quit IRC | 19:31 | |
| pabelanger | 2017-11-01 19:07:34,801 DEBUG zuul.AnsibleJob: [build: 115f586041234056a827e9d518e6a49a] Sending result: {"error_detail": "Unable to find playbook /var/lib/zuul/builds/115f586041234056a827e9d518e6a49a/trusted/project_0/git.openstack.org/openstack-infra/project-config/playbooks/python-branch-tarball/post.yaml", "result": "ERROR"} | 19:31 |
| pabelanger | I see that in logs | 19:31 |
| pabelanger | not sure if related | 19:31 |
| AJaeger | pabelanger: looks related, let me check. thanks | 19:32 |
| *** salv-orlando has quit IRC | 19:32 | |
| AJaeger | indeed, that does not exist... | 19:32 |
| pabelanger | did we delete or move something? | 19:32 |
| *** jcoufal_ has quit IRC | 19:33 | |
| AJaeger | pabelanger: yes, you did ;( publish-openstack-puppet-branch-tarball removed the wrong dir... | 19:34 |
| AJaeger | pabelanger: change I7ca54a7045023596ec77682214afad5f210ac964 - I'll fix... | 19:34 |
| pabelanger | AJaeger: oh, ya, that will do it | 19:34 |
| *** ijw has joined #openstack-infra | 19:36 | |
| *** adreznec has joined #openstack-infra | 19:36 | |
| *** hashar has joined #openstack-infra | 19:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Readd playbooks/python-branch-tarball/ https://review.openstack.org/517094 | 19:36 |
| AJaeger | pabelanger: ^ | 19:36 |
| pabelanger | AJaeger: did you also want to delete puppet-branch-tarball, that was the one we wanted to delete | 19:37 |
| AJaeger | pabelanger: that's still used | 19:37 |
| AJaeger | oh, it's not - let me add | 19:38 |
| pabelanger | AJaeger: Ya, it should have been puppet-branch-tarball that was to be original deleted | 19:38 |
| *** bobh has quit IRC | 19:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Readd playbooks/python-branch-tarball/ https://review.openstack.org/517094 | 19:39 |
| AJaeger | pabelanger: I did some grep and found usage, must have done something wrong - so here you go ^ | 19:39 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/elastic-recheck master: grenade: wait for service logs before matching against patterns https://review.openstack.org/493987 | 19:39 |
| pabelanger | AJaeger: +3 | 19:40 |
| pabelanger | thanks | 19:40 |
| pabelanger | clarkb: okay, new logstash-workers started and processing | 19:41 |
| pabelanger | at least I think they are | 19:41 |
| AJaeger | pabelanger: thanks | 19:41 |
| *** electrofelix has quit IRC | 19:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Merge project-config linter jobs (1/2) https://review.openstack.org/517086 | 19:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Merge project-config linter jobs (2/2) https://review.openstack.org/517095 | 19:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove project-config-zuul-jobs-linters https://review.openstack.org/517087 | 19:44 |
| * AJaeger calls it a day | 19:45 | |
| SamYaple | o/ | 19:49 |
| *** bnemec has quit IRC | 19:53 | |
| *** makowals has joined #openstack-infra | 19:55 | |
| *** rwsu has joined #openstack-infra | 19:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Readd playbooks/python-branch-tarball/ https://review.openstack.org/517094 | 19:57 |
| *** esberglu has quit IRC | 19:57 | |
| *** esberglu has joined #openstack-infra | 19:57 | |
| *** esberglu has quit IRC | 19:58 | |
| *** ihrachys_ is now known as ihrachys | 20:06 | |
| *** eharney has quit IRC | 20:08 | |
| openstackgerrit | Merged openstack-infra/system-config master: Also add logstash-worker works to cacti https://review.openstack.org/517088 | 20:08 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/project-config master: Remove legacy ara job https://review.openstack.org/517099 | 20:10 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: puppetmaster: rotate /var/log/puppet_run_all_infracloud.log https://review.openstack.org/516165 | 20:10 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/openstack-zuul-jobs master: Remove ara job https://review.openstack.org/517100 | 20:11 |
| *** edmondsw has quit IRC | 20:11 | |
| *** edmondsw has joined #openstack-infra | 20:11 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Remove legacy ara job https://review.openstack.org/517099 | 20:11 |
| *** esberglu has joined #openstack-infra | 20:13 | |
| *** Goneri has quit IRC | 20:14 | |
| *** ldnunes has quit IRC | 20:15 | |
| *** jcoufal_ has joined #openstack-infra | 20:18 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add new backup host to puppet https://review.openstack.org/516148 | 20:19 |
| *** jcoufal__ has joined #openstack-infra | 20:20 | |
| *** jcoufal has quit IRC | 20:20 | |
| *** smatzek has quit IRC | 20:21 | |
| openstackgerrit | Merged openstack-infra/system-config master: Backup zuulv3 with bup https://review.openstack.org/516157 | 20:22 |
| *** felipemonteiro_ has joined #openstack-infra | 20:22 | |
| *** jcoufal_ has quit IRC | 20:22 | |
| *** rwsu has quit IRC | 20:23 | |
| *** mlavalle has joined #openstack-infra | 20:24 | |
| *** felipemonteiro__ has quit IRC | 20:24 | |
| mlavalle | clarkb: ping | 20:24 |
| clarkb | mlavalle: hi | 20:25 |
| mlavalle | clarkb: I recently merged this change https://review.openstack.org/#/c/512856/ | 20:26 |
| *** dprince has quit IRC | 20:26 | |
| mlavalle | But the meeting is still on Thursday: http://eavesdrop.openstack.org/#Neutron_drivers_Meeting | 20:26 |
| *** edmondsw has quit IRC | 20:27 | |
| clarkb | there is probably a bug in the meeting page update publishing | 20:27 |
| clarkb | mlavalle: can you add it to https://etherpad.openstack.org/p/zuulv3-issues as something that we should track down/triage? | 20:27 |
| *** jcoufal__ has quit IRC | 20:27 | |
| *** edmondsw has joined #openstack-infra | 20:27 | |
| clarkb | I'm going to drive to the airport soon and won't be able to look at it now | 20:28 |
| mlavalle | clarkb: sure, thanks | 20:28 |
| *** Apoorva has joined #openstack-infra | 20:28 | |
| *** rbrndt has quit IRC | 20:29 | |
| *** Goneri has joined #openstack-infra | 20:30 | |
| EmilienM | could i have a review on https://review.openstack.org/#/c/517060/ please? | 20:31 |
| *** edmondsw has quit IRC | 20:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Merge project-config linter jobs (1/2) https://review.openstack.org/517086 | 20:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Merge project-config linter jobs (2/2) https://review.openstack.org/517095 | 20:32 |
| *** salv-orlando has joined #openstack-infra | 20:33 | |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/project-config master: Remove legacy ara job https://review.openstack.org/517099 | 20:34 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/openstack-zuul-jobs master: Remove ara job https://review.openstack.org/517100 | 20:35 |
| *** bnemec has joined #openstack-infra | 20:35 | |
| *** rbrndt has joined #openstack-infra | 20:36 | |
| *** pramodrj07 has joined #openstack-infra | 20:36 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/bindep master: Fix logic for groups https://review.openstack.org/517105 | 20:38 |
| *** salv-orlando has quit IRC | 20:38 | |
| *** edmondsw has joined #openstack-infra | 20:39 | |
| *** MasterOfBugs has quit IRC | 20:40 | |
| *** rosmaita has quit IRC | 20:41 | |
| *** rwsu has joined #openstack-infra | 20:42 | |
| *** rfolco_ has quit IRC | 20:43 | |
| *** thiagolib has quit IRC | 20:46 | |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/project-config master: Remove legacy ara job https://review.openstack.org/517099 | 20:46 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/infra-manual master: Add "openstack-tox-cover (PTI)" to list of things not to convert https://review.openstack.org/517106 | 20:47 |
| dmsimard | AJaeger: ^ | 20:47 |
| *** threestrands has joined #openstack-infra | 20:48 | |
| *** adreznec has quit IRC | 20:51 | |
| *** makowals has quit IRC | 20:53 | |
| *** adreznec has joined #openstack-infra | 20:53 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/bindep master: Fix logic for groups https://review.openstack.org/517105 | 20:55 |
| SamYaple | are we in the midst of another infra mirror vs upstream mirror issue? http://logs.openstack.org/05/517105/1/check/legacy-bindep-fallback-debian-jessie/3db711c/job-output.txt.gz#_2017-11-01_20_42_23_809079 | 20:55 |
| *** kgiusti has left #openstack-infra | 20:55 | |
| *** xingchao has joined #openstack-infra | 20:56 | |
| *** armaan has quit IRC | 20:56 | |
| *** jascott1 has quit IRC | 20:57 | |
| *** armaan has joined #openstack-infra | 20:57 | |
| *** armaan has quit IRC | 20:57 | |
| *** armaan has joined #openstack-infra | 20:58 | |
| *** derekh has quit IRC | 20:59 | |
| *** mat128 has quit IRC | 21:00 | |
| *** xingchao has quit IRC | 21:00 | |
| *** smatzek has joined #openstack-infra | 21:02 | |
| *** smatzek has quit IRC | 21:04 | |
| *** xingchao has joined #openstack-infra | 21:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Merge project-config linter jobs (1/2) https://review.openstack.org/517086 | 21:05 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Merge project-config linter jobs (2/2) https://review.openstack.org/517095 | 21:05 |
| jeblair | ianw, pabelanger: did we ever set our image builds to use our mirrors? | 21:06 |
| pabelanger | jeblair: we did for ubuntu | 21:07 |
| pabelanger | I think we still need to do centos | 21:07 |
| pabelanger | and others | 21:07 |
| jeblair | and maybe debian? since that's the one that's broken now | 21:07 |
| jeblair | and also the one that broke last time | 21:07 |
| pabelanger | sure, we can do that one too | 21:08 |
| *** trown is now known as trown|outtypewww | 21:08 | |
| *** thorst has quit IRC | 21:08 | |
| jeblair | pabelanger: can you pause and drop the latest jessie image builds in nodepool? | 21:08 |
| pabelanger | jeblair: yes | 21:08 |
| jeblair | i'm trying to see what's going on with mirror-update | 21:09 |
| *** eharney has joined #openstack-infra | 21:11 | |
| jeblair | i think mirror-update may have been rebooted during a repo check and left the lockfile around | 21:12 |
| *** priteau has quit IRC | 21:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Pause debian-jessie images https://review.openstack.org/517110 | 21:13 |
| pabelanger | jeblair: ^ | 21:13 |
| jeblair | pabelanger: +3 | 21:14 |
| jeblair | pabelanger: looks like your timeout increase landed | 21:14 |
| jeblair | pabelanger: so i'm going to go ahead and just delete the lock file and we'll see how the next run looks. | 21:14 |
| jeblair | pabelanger: sound good? | 21:14 |
| *** vhosakot_ has quit IRC | 21:14 | |
| pabelanger | jeblair: sure | 21:14 |
| jeblair | done. next run is in 45m | 21:15 |
| *** catintheroof has quit IRC | 21:16 | |
| jeblair | i could go ahead and run it now in screen i guess? would that be a good idea? | 21:16 |
| jeblair | (the cronjob flocks, so it should be safe) | 21:16 |
| pabelanger | yah, it should be safe | 21:16 |
| jeblair | flock -n /var/run/reprepro/debian.lock reprepro-mirror-update /etc/reprepro/debian mirror.debian >>/var/log/reprepro/debian-mirror.log 2>&1 | 21:17 |
| jeblair | i'll just go ahead and run that in screen ^ | 21:17 |
| *** salv-orlando has joined #openstack-infra | 21:18 | |
| *** adreznec has quit IRC | 21:20 | |
| *** PramodJ has joined #openstack-infra | 21:20 | |
| *** pramodrj07 has quit IRC | 21:21 | |
| *** adreznec has joined #openstack-infra | 21:22 | |
| SamYaple | jeblair: is that 45m before i should recheck, or 45m before it starts to build? | 21:23 |
| *** xingchao has quit IRC | 21:24 | |
| *** PramodJ has quit IRC | 21:24 | |
| *** e0ne has quit IRC | 21:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Pause debian-jessie images https://review.openstack.org/517110 | 21:30 |
| jeblair | SamYaple: i'm kicking it off now, but as soon as pabelanger is done deleting our new jessie images, it should be ok to recheck | 21:30 |
| *** dave-mccowan has quit IRC | 21:30 | |
| *** adreznec has quit IRC | 21:31 | |
| pabelanger | yah, just looking now at nb03 / nb04 now | 21:31 |
| jeblair | SamYaple: (the fastest fix is to revert to our old images which weren't built with too-new packages, so they can update to what's in the mirror) | 21:31 |
| pabelanger | oh, darn | 21:31 |
| jeblair | SamYaple: (once the mirror is updated, we can build new images again) | 21:31 |
| pabelanger | made a mistake | 21:31 |
| *** adreznec has joined #openstack-infra | 21:32 | |
| SamYaple | cool. thanks for the info | 21:33 |
| jeblair | pabelanger: what's that? | 21:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Also pause debian-jessie in nodepool.yaml https://review.openstack.org/517112 | 21:33 |
| pabelanger | jeblair: SamYaple: ^ | 21:33 |
| pabelanger | builders are using that file | 21:33 |
| jeblair | pabelanger: ah, as mistakes go that's not a bad one :) | 21:33 |
| pabelanger | :) | 21:34 |
| jeblair | i was expecting "deleted all images" or something | 21:34 |
| pabelanger | ah | 21:34 |
| pabelanger | not yet | 21:34 |
| jeblair | lol | 21:34 |
| jeblair | pabelanger: looks like 0000004588 is the one to delete, yeah? | 21:35 |
| *** xingchao has joined #openstack-infra | 21:36 | |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/openstack-zuul-jobs master: Remove ara job https://review.openstack.org/517100 | 21:38 |
| pabelanger | jeblair: yes | 21:41 |
| *** rcernin has joined #openstack-infra | 21:42 | |
| *** bocon has joined #openstack-infra | 21:45 | |
| *** bocon has quit IRC | 21:45 | |
| jeblair | pabelanger, SamYaple: it looks like the mirror release just happened successfully | 21:48 |
| SamYaple | :+1: | 21:49 |
| jeblair | pabelanger: so if you haven't deleted the images yet, i don't think we have to | 21:49 |
| jeblair | SamYaple: can you go ahead and recheck and let us know if it looks better? | 21:49 |
| SamYaple | your wish is my task i will do at some point | 21:50 |
| pabelanger | jeblair: oh, okay, I was waiting for patch to land | 21:51 |
| pabelanger | jeblair: but I actually need to afk now, do you mind if I hand off to you? | 21:51 |
| *** Sukhdev has quit IRC | 21:51 | |
| jeblair | pabelanger: yep. thanks! | 21:52 |
| pabelanger | ianw: dmsimard: do you mind stepping in? See back scroll about debian-jessie | 21:52 |
| dmsimard | let me see | 21:52 |
| ianw | pabelanger: ok, i'm dropping kids to school, back in 10 minutes | 21:52 |
| jeblair | i don't think there's anything to do at the moment except wait for feedback from SamYaple | 21:52 |
| dmsimard | pretty sure our configure-mirror role works for every supported distro | 21:53 |
| * dmsimard wrote that | 21:53 | |
| jeblair | the second patch had a syntax error preventing it from landing | 21:53 |
| jeblair | nono that's not the problem | 21:53 |
| dmsimard | oh I understand | 21:54 |
| jeblair | the summary is: our debian mirror got stuck and was not updating, but our images built with newer packages, so they can't upgrade anymore. the procedure in this situation is to pause image builds and revert to last known good image. | 21:54 |
| dmsimard | yeah, I realized that by re-reading slowly | 21:54 |
| jeblair | we were half-way through that procedure when i unstuck the mirror. so instead of rolling back, we *should* be able to roll-forward | 21:54 |
| SamYaple | jeblair: pabelanger the news is good. jessie test passed | 21:54 |
| jeblair | i just want to get confirmation that the newly updated mirror fixes the problem before i revert pabelanger's pause patch. | 21:54 |
| SamYaple | thanks for the quick response :) | 21:55 |
| jeblair | which we have! | 21:55 |
| *** adreznec has quit IRC | 21:55 | |
| dmsimard | So the gist of the issue is that we should probably be using our own mirrors when building images | 21:55 |
| SamYaple | i thought we were going to do that already | 21:55 |
| dmsimard | because I can see this kind of issue being a bit racy, even if the mirrors aren't stuck | 21:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Revert "Pause debian-jessie images" https://review.openstack.org/517114 | 21:55 |
| SamYaple | after what happened with the buntus | 21:55 |
| jeblair | dmsimard: yes. apparently we only got as far as doing that for ubuntu, not debian | 21:56 |
| dmsimard | SamYaple: It might already be the case, I'm not sure.. I'll dig | 21:56 |
| jeblair | dmsimard: now that we have the all clear from SamYaple, can you +3 517114 ? | 21:56 |
| *** felipemonteiro_ has quit IRC | 21:56 | |
| dmsimard | I can and I did | 21:56 |
| *** adreznec has joined #openstack-infra | 21:57 | |
| *** rwsu has quit IRC | 21:57 | |
| jeblair | #status log jessie mirror not updated since oct 10 due to reboot of server mid-update. manually removed stale lockfile for debian jessie reprepro; mirror updated and release sucessfully. | 21:57 |
| openstackstatus | jeblair: finished logging | 21:57 |
| jeblair | okay, that should be everything, aside from the future-proofing by using mirrors during builds | 21:58 |
| *** eharney has quit IRC | 21:59 | |
| *** bobh has joined #openstack-infra | 21:59 | |
| dmsimard | jeblair: for centos/fedora I think this does the trick: https://git.openstack.org/cgit/openstack/diskimage-builder/commit/diskimage_builder/elements/centos-minimal?id=649f0b66d9011bc041f33281c1b25949b8d8628d | 22:00 |
| dmsimard | it seems like the support for the different distros is there as well https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/openstack-ci-mirrors/environment.d/11-dib-distribution-mirror.bash | 22:01 |
| fungi | if we ever end up with a community monitoring system, i can imagine polling and tracking the last updated timestamp across the various mirrors could be useful... or maybe we tweak the mirror builds to emit events to graphite | 22:01 |
| dmsimard | fungi: I still need to write that spec :( | 22:01 |
| jeblair | fungi: ++ | 22:01 |
| dmsimard | on my todo | 22:01 |
| *** shardy has quit IRC | 22:01 | |
| * fungi disappears back into the sky | 22:01 | |
| jeblair | mirror volume not released in > 48h is a really good and safe metric. | 22:01 |
| *** bobh has quit IRC | 22:03 | |
| *** florianf has quit IRC | 22:04 | |
| dmsimard | ianw: can you help a DIB noob out ? pretend we include 'openstack-ci-mirrors' in our images, what does it actually do ? i.e, https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/openstack-ci-mirrors/environment.d/11-dib-distribution-mirror.bash | 22:06 |
| dmsimard | I'm not clear on what "environment.d" ends up doing | 22:07 |
| dmsimard | I *think* it's meant to be included /before/, say, ubuntu-minimal, centos-minimal, so that these end up configuring the openstack-infra mirrors | 22:07 |
| jeblair | dmsimard: without answering that question -- i will note i believe that element is designed to aid in gate testing, and not necessarily in actual image building | 22:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Pause debian-jessie images" https://review.openstack.org/517114 | 22:08 |
| dmsimard | but I don't think the necessary variables, like NODEPOOL_UBUNTU_MIRROR, would be available for building | 22:08 |
| dmsimard | jeblair: yeah I'm trying to understand :) | 22:08 |
| jeblair | dmsimard: if it helps, i think https://review.openstack.org/511492 is the change to use the ubuntu mirror for our image builds | 22:09 |
| dmsimard | jeblair: regardless, I'm not sure to what extent we want to persist openstack-infra mirrors inside the image since they end up being distributed in every cloud with different mirrors anyway.. | 22:09 |
| *** xingchao has quit IRC | 22:09 | |
| jeblair | dmsimard: yeah, we still want the update | 22:09 |
| jeblair | gr | 22:09 |
| jeblair | dmsimard: we still want the configure-mirror role to update the config on the node, we just want the image built from one of the mirrors to start with. | 22:10 |
| jeblair | dmsimard: i believe pabelanger hard-coded rax into our nodepool config for building that because all our image builds happen in rax | 22:10 |
| dmsimard | jeblair: the openstack-ci-mirrors element basically does what that patch you sent me, it sets up DIB_DISTRIBUTION_MIRROR | 22:10 |
| *** xingchao has joined #openstack-infra | 22:10 | |
| dmsimard | jeblair: but it expects NODEPOOL_*_MIRROR to exist, which is not a thing on nodepool builders | 22:11 |
| jeblair | dmsimard: exactly. it's a thing on built images, but not on the builders. so pabelanger's patch adds it to the DIB environment via nodepool. | 22:11 |
| * dmsimard nod | 22:11 | |
| *** edmondsw has quit IRC | 22:13 | |
| *** edmondsw has joined #openstack-infra | 22:13 | |
| *** panda|ruck is now known as panda|ruck|off | 22:13 | |
| *** rbrndt has quit IRC | 22:13 | |
| SamYaple | just an fyi, before we tag a new bindep this should merge because it changes the group logic a touch and i dont want to see a released version where we have to support the old logic (which hasnt been tagged yet) https://review.openstack.org/#/c/517105/ | 22:17 |
| *** rlandy is now known as rlandy|afk | 22:17 | |
| *** edmondsw has quit IRC | 22:18 | |
| *** baoli has quit IRC | 22:19 | |
| ianw | dmsimard: hey, yeah opestack-ci-mirrors takes the /etc/ci-mirror file that's written out (now by ansible, used to be slave scripts) and translates that into some .repo files and environment variables for dib to use mirrors during the build | 22:20 |
| *** xarses has quit IRC | 22:20 | |
| *** mlavalle has quit IRC | 22:21 | |
| *** gordc has left #openstack-infra | 22:23 | |
| *** slaweq has quit IRC | 22:24 | |
| *** salv-orlando has quit IRC | 22:25 | |
| *** owalsh_biab is now known as owalsh | 22:26 | |
| ianw | yeah, so in dib functional tests, we don't care about the final image | 22:26 |
| ianw | the thing i've never been convinced we have a good solution for not leaving the mirror behind in the image. just because you build against it, doesn't mean you want your final image to reference it | 22:27 |
| *** rbrndt has joined #openstack-infra | 22:28 | |
| jeblair | well, in our case it *should* be overwritten by our configure_mirror role so it shouldn't matter too much. but yeah, ideally we'd leave something more neutral in place. especially if we wanted to publish the images. | 22:29 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Prime github app install map on connection load https://review.openstack.org/517121 | 22:29 |
| ianw | jeblair: yeah, that was what i mean from a generic dib mechanism point of view | 22:30 |
| ianw | we have DIB_DISTRIBUTION_MIRROR ... but that change from pabelanger added another bit to the element to strip the build mirrors out, a function dib should really provide | 22:31 |
| ianw | also we got all tangled up trying to make one global "use this variable to point to mirrors" but it just didn't work across platforms. i remember discussions with greghaynes | 22:32 |
| *** lbragstad has quit IRC | 22:33 | |
| greghaynes | I think I was trying to undo the single global variable to make it easier for a user to export one set of env vars and depending on the distro being made the right env var would be used | 22:38 |
| greghaynes | been a while though | 22:38 |
| greghaynes | https://review.openstack.org/#/c/408850/ | 22:38 |
| greghaynes | looks like thats a thing now :) | 22:38 |
| *** MasterOfBugs has joined #openstack-infra | 22:40 | |
| *** esberglu has quit IRC | 22:43 | |
| *** tpsilva has quit IRC | 22:43 | |
| *** thorst has joined #openstack-infra | 22:45 | |
| *** owalsh is now known as owalsh_pto | 22:46 | |
| *** thorst has quit IRC | 22:46 | |
| ianw | greghaynes: yeah, we never really covered the clearing out bit though | 22:48 |
| *** aeng has joined #openstack-infra | 22:50 | |
| *** ijw has quit IRC | 22:50 | |
| *** smatzek has joined #openstack-infra | 22:52 | |
| *** rbrndt has quit IRC | 22:53 | |
| *** wolverineav has quit IRC | 22:54 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add BaseSource.getProjectReadonly and refactor https://review.openstack.org/517067 | 22:56 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Do not add invalid projets via the /keys API https://review.openstack.org/517078 | 22:56 |
| *** smatzek has quit IRC | 22:57 | |
| *** adreznec has quit IRC | 22:57 | |
| *** salv-orlando has joined #openstack-infra | 22:59 | |
| *** adreznec has joined #openstack-infra | 22:59 | |
| *** 5EXAAPFFA has joined #openstack-infra | 23:01 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test repositories https://review.openstack.org/517130 | 23:03 |
| *** hongbin has quit IRC | 23:09 | |
| *** gildub has joined #openstack-infra | 23:12 | |
| *** armax has quit IRC | 23:13 | |
| *** calbers has quit IRC | 23:22 | |
| *** calbers has joined #openstack-infra | 23:24 | |
| jlvillal | Has this been asked before? We download files from repo.tinycorelinux.net to build our TinyIPA. Would it be possible for these files to be mirrored on the openstack.org infrastructure somewhere so we don't get errors if the server is down? | 23:24 |
| *** xingchao has quit IRC | 23:24 | |
| *** gmann_afk is now known as gmann | 23:24 | |
| jlvillal | For example just got an error like this: http://logs.openstack.org/60/516060/1/gate/ipa-tempest-dsvm-partition-bios-pxe_ipmitool-tinyipa-src/2dabfad/logs/devstacklog.txt.gz#_2017-11-01_21_34_52_049 | 23:24 |
| jlvillal | I see that repo.tinycorelinux.net does support rsync | 23:25 |
| *** gyee has quit IRC | 23:25 | |
| *** calbers has quit IRC | 23:27 | |
| *** hashar has quit IRC | 23:27 | |
| *** mikal has quit IRC | 23:28 | |
| *** jklare has quit IRC | 23:30 | |
| ianw | jlvillal: it has not been asked, but it would be possible | 23:33 |
| ianw | (not asked afaik) | 23:34 |
| jlvillal | ianw: Cool. I'll discuss with the Ironic team. | 23:34 |
| jlvillal | ianw: we have had erratic failures for sure due to issues with their site being down or just hiccups | 23:34 |
| ianw | jlvillal: is it a deb repo? | 23:34 |
| jlvillal | ianw: No. It is tar.gz files :) | 23:34 |
| ianw | ahh, http://repo.tinycorelinux.net/8.x/x86/release/ ? | 23:35 |
| jlvillal | ianw: For example a file might be: texinfo.tcz | 23:35 |
| jlvillal | ianw: yeah! | 23:35 |
| ianw | jlvillal: ok, in some cases we've done caching reverse proxies on a certain port, but if they have rsync and no special signing requirements, etc, then a mirror job is probably best | 23:38 |
| jlvillal | ianw: Cool. Yeah, seems like I can just rsync the files. I did some directory walking with rsync without issues. | 23:39 |
| jlvillal | ianw: I would say it is slightly insecure method, but then again they only offer http: as it is :( So rsync isn't any less secure than that. | 23:40 |
| *** salv-orl_ has joined #openstack-infra | 23:41 | |
| *** larainema has quit IRC | 23:41 | |
| *** salv-orlando has quit IRC | 23:43 | |
| ianw | jlvillal: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/mirror_update.pp is where to look | 23:44 |
| jlvillal | ianw: Thanks | 23:44 |
| ianw | reverse proxy can work too, but if the remote end is intermittent or does things like partial updates that you tend to catch, having a periodic mirror is probably better | 23:44 |
| jlvillal | So since I just saw a TIMED_OUT failure on: https://review.openstack.org/#/c/517063/ I was wondering if could get a review for: https://review.openstack.org/#/c/513444/ it does have one +2 :) | 23:45 |
| *** jklare has joined #openstack-infra | 23:47 | |
| ianw | oh nice one, meaning to look at that. I also feel like i've seen issues highlighting MERGE_FAILURE | 23:48 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Make encrypt_secret.py work with OpenSSL 0.x https://review.openstack.org/517133 | 23:50 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Prime github app install map on connection load https://review.openstack.org/517121 | 23:50 |
| *** 5EXAAPFFA has quit IRC | 23:50 | |
| jlvillal | ianw: Oh that might be another one to add. Not sure I have seen it or not myself. | 23:50 |
| jlvillal | Add as in a new patch at some point... | 23:51 |
| *** jaypipes has quit IRC | 23:54 | |
| *** links has joined #openstack-infra | 23:56 | |
| *** pramodrj07 has joined #openstack-infra | 23:56 | |
| *** stakeda has joined #openstack-infra | 23:57 | |
| *** MasterOfBugs has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!