| clarkb | I checked bandersnatch one last time and now ist doing something different maybe I will stick around another 20 minute sto see what happens | 00:01 |
|---|---|---|
| ianw | pabelanger: there's a couple of others in other regions, but not as many | 00:03 |
| *** yamamoto has joined #openstack-infra | 00:03 | |
| pabelanger | ianw: yah, we should try to understand why we are leaking them. Or why nodepool isn't cleaning them up | 00:05 |
| *** r-daneel has joined #openstack-infra | 00:05 | |
| clarkb | ok it finished | 00:05 |
| pabelanger | ianw: alien-list was recently removed, so that doesn't help any more | 00:05 |
| clarkb | ianw: I guess I'll go ahead and start it again now and it should finish much more quickly | 00:05 |
| *** r-daneel has quit IRC | 00:05 | |
| ianw | pabelanger: well, i started on a script to show old nodes, but it's *sooo* slow due to the number of images it tries to iterate in rax. i will clean that up after we've sorted out this glean issue | 00:05 |
| clarkb | it is running again | 00:07 |
| *** yamamoto has quit IRC | 00:07 | |
| fungi | if memory serves, a full bandersnatch resync needs a rerun to perform its finalization pass or something like that, so it's like the third or fourth run where you should finally see it properly no-op | 00:10 |
| clarkb | fun, in any case it is on its second pass now | 00:10 |
| clarkb | processing mxnet something or other so its jumped well ahead | 00:10 |
| clarkb | I'll pick it up in the morning but hope to do a vos release as soon as bandersnatch steady states | 00:11 |
| clarkb | then I'll add in the blacklist and reenable the cron | 00:11 |
| pabelanger | looks like airbnb just got cancelled for pre summit days. new rules kicking in | 00:11 |
| *** caphrim007_ has joined #openstack-infra | 00:12 | |
| clarkb | pabelanger: ya they have to have a business license and operate out of their primary residence only | 00:12 |
| clarkb | pabelanger: I guess the expect at least 1000 new units on the long term rental market as a result | 00:12 |
| *** caphrim007 has quit IRC | 00:15 | |
| ianw | i dunno what nodepool thinks is going on with xenial arm64; it's built and image but not uploaded it | 00:15 |
| pabelanger | clarkb: yah, that's what I heard too | 00:16 |
| *** caphrim007_ has quit IRC | 00:16 | |
| *** Goneri has quit IRC | 00:19 | |
| ianw | ok, finally uploading new image | 00:20 |
| *** liusheng has quit IRC | 00:20 | |
| *** liusheng has joined #openstack-infra | 00:21 | |
| *** yamahata has quit IRC | 00:23 | |
| *** yamamoto has joined #openstack-infra | 00:24 | |
| *** iyamahat has quit IRC | 00:25 | |
| *** yamamoto has quit IRC | 00:29 | |
| *** kiennt2680 has joined #openstack-infra | 00:30 | |
| pabelanger | checkpool is still running on reprepro, I expect it to take a while, since we are doing a stat on each file | 00:34 |
| ianw | pabelanger: ssh root@103.20.251.143 | 00:36 |
| *** bobh has joined #openstack-infra | 00:37 | |
| ianw | so that's double confirmation that it's the initramfs update -- that's a xenial built on nb03.o.o reverted back to dib 2.12.1 | 00:37 |
| pabelanger | ianw: yay, at least we know what made it break | 00:38 |
| ianw | pabelanger: but that's a dhcp environment ... | 00:39 |
| pabelanger | ah, right | 00:39 |
| ianw | why o why would this trigger in multiple providers, but *not* during our nodepool tests | 00:39 |
| ianw | is it just that nested emulated boot in devstack is so much slower? | 00:39 |
| *** wolverin_ has quit IRC | 00:39 | |
| *** wolverineav has joined #openstack-infra | 00:40 | |
| *** markmcclain has quit IRC | 00:44 | |
| *** yamamoto has joined #openstack-infra | 00:44 | |
| *** markmcclain has joined #openstack-infra | 00:49 | |
| *** yamamoto has quit IRC | 00:51 | |
| *** stakeda has joined #openstack-infra | 00:56 | |
| *** salv-orlando has joined #openstack-infra | 01:02 | |
| ianw | we really should install at least vi on the base system | 01:03 |
| *** yamahata has joined #openstack-infra | 01:04 | |
| *** salv-orl_ has quit IRC | 01:04 | |
| pabelanger | ianw: minimal :) | 01:05 |
| pabelanger | guess we could add it to infra-package-needs | 01:05 |
| *** cshastri has joined #openstack-infra | 01:05 | |
| *** dmsimard is now known as dmsimard|off | 01:06 | |
| *** yamamoto has joined #openstack-infra | 01:07 | |
| *** bobh has quit IRC | 01:09 | |
| *** bobh has joined #openstack-infra | 01:09 | |
| *** yamamoto has quit IRC | 01:12 | |
| ianw | initramfs-tools-core: /usr/share/initramfs-tools/scripts/init-top/framebuffer | 01:13 |
| ianw | it is making no sense to me how that isn't in the initramfs | 01:13 |
| *** bobh has quit IRC | 01:14 | |
| ianw | well, it appears the initrd i've booted with doesn't have it, and glean has worked :/ | 01:14 |
| *** harlowja has quit IRC | 01:16 | |
| *** yamamoto has joined #openstack-infra | 01:28 | |
| *** bobh has joined #openstack-infra | 01:31 | |
| *** armaan has quit IRC | 01:32 | |
| *** yamamoto has quit IRC | 01:32 | |
| *** janki has joined #openstack-infra | 01:33 | |
| *** jamesmcarthur has joined #openstack-infra | 01:36 | |
| *** zhangfei has joined #openstack-infra | 01:38 | |
| *** kiennt2680 has quit IRC | 01:38 | |
| *** spsurya has joined #openstack-infra | 01:38 | |
| *** hongbin_ has joined #openstack-infra | 01:40 | |
| *** salv-orlando has quit IRC | 01:41 | |
| *** ihar has quit IRC | 01:41 | |
| *** salv-orlando has joined #openstack-infra | 01:42 | |
| *** bobh has quit IRC | 01:43 | |
| *** kiennt2680 has joined #openstack-infra | 01:43 | |
| *** ihar has joined #openstack-infra | 01:43 | |
| *** kiennt2680 has quit IRC | 01:44 | |
| *** kiennt2695 has joined #openstack-infra | 01:45 | |
| clarkb | bandersnatch had stopped. There were timeout errors downloading what I'm assuming were large packages. I have restarted it | 01:46 |
| *** kiennt2695 has quit IRC | 01:46 | |
| *** kiennt2651 has joined #openstack-infra | 01:46 | |
| *** kiennt2651 has quit IRC | 01:47 | |
| *** yamamoto has joined #openstack-infra | 01:49 | |
| *** kiennt2609 has joined #openstack-infra | 01:49 | |
| *** jamesmcarthur has quit IRC | 01:50 | |
| *** slaweq has joined #openstack-infra | 01:50 | |
| *** wolverineav has quit IRC | 01:54 | |
| *** wolverineav has joined #openstack-infra | 01:54 | |
| *** slaweq has quit IRC | 01:55 | |
| *** yamamoto has quit IRC | 01:55 | |
| *** rlandy|bbl is now known as rlandy | 01:55 | |
| *** rlandy has quit IRC | 01:56 | |
| *** ianychoi_ is now known as ianychoi | 01:57 | |
| *** pcichy has joined #openstack-infra | 01:58 | |
| *** wolverineav has quit IRC | 01:59 | |
| *** salv-orl_ has joined #openstack-infra | 02:00 | |
| *** pcichy has quit IRC | 02:02 | |
| *** caphrim007 has joined #openstack-infra | 02:03 | |
| *** salv-orlando has quit IRC | 02:03 | |
| *** nicolasbock has quit IRC | 02:06 | |
| *** caphrim007 has quit IRC | 02:08 | |
| *** yamamoto has joined #openstack-infra | 02:12 | |
| *** dingyichen has joined #openstack-infra | 02:14 | |
| *** yamamoto has quit IRC | 02:17 | |
| ianw | i've taken a "bad" initramfs, put it into my image in rdo cloud, the damn thing still boots and glean runs | 02:20 |
| *** caphrim007 has joined #openstack-infra | 02:24 | |
| *** caphrim007_ has joined #openstack-infra | 02:25 | |
| clarkb | ianw: timing issue then maybe? | 02:26 |
| clarkb | would explain why arm64 kvm and xen x86 appear to be affected but others arent | 02:27 |
| ianw | it's a big pain because those are a complete PITA to test | 02:27 |
| clarkb | ya | 02:28 |
| ianw | i don't have a separate arm builder to iterate on, and i still haven't got a vhd upload working properly i don't think | 02:28 |
| *** caphrim007 has quit IRC | 02:29 | |
| *** dklyle has joined #openstack-infra | 02:31 | |
| *** yamamoto has joined #openstack-infra | 02:32 | |
| ianw | i'm going to try one more time with a custom vhd upload and see if i can get it to trigger | 02:34 |
| clarkb | ianw: how are you converting to vhd? | 02:36 |
| ianw | just with dib's "-t vhd" option | 02:37 |
| ianw | then uploading via shade | 02:37 |
| clarkb | I've always copied my qcow2 to a nodepool builder and converted it there then uploaded with shad | 02:37 |
| ianw | i wish i could remember my personal login for rax | 02:37 |
| clarkb | That way I am using the vhdutil or whatever it is that I know works | 02:37 |
| ianw | yeah, this is our vhd converter on this host | 02:38 |
| ianw | from the ppa | 02:38 |
| *** yamamoto has quit IRC | 02:39 | |
| scas | are 'timeouts' currently a known thing? | 02:44 |
| *** ianychoi has quit IRC | 02:44 | |
| scas | my jobs start to execute, then... just... time... out. nothing really more useful that i can see from my end | 02:45 |
| *** psachin has joined #openstack-infra | 02:48 | |
| clarkb | scas have links to an example? | 02:48 |
| scas | https://review.openstack.org/561629 is the problem review in question. the logs aren't too useful | 02:49 |
| scas | a recheck just kicked some builds | 02:49 |
| scas | if it holds to pattern, the 'rake' job will timeout in ~30 minutes | 02:50 |
| *** jchhatbar has joined #openstack-infra | 02:51 | |
| clarkb | the logs seem to show it doesnt just timeout. its running the whole time | 02:52 |
| clarkb | is the timeout just not long enough for the job? | 02:52 |
| scas | perhaps? i don't think they're something we're configuring, not deliberately | 02:52 |
| clarkb | ya zuul gives every job a 30 minute timeout by default | 02:53 |
| scas | the compute cookbook takes a while to run through the unit tests | 02:53 |
| clarkb | you can increase or decrease this in your jobs | 02:53 |
| *** janki has quit IRC | 02:53 | |
| scas | okay. i can do some local benchmarking to compare and contrast | 02:53 |
| *** yamamoto has joined #openstack-infra | 02:55 | |
| *** VW has joined #openstack-infra | 02:59 | |
| *** yamamoto has quit IRC | 03:00 | |
| *** dave-mccowan has quit IRC | 03:02 | |
| scas | my own testing yields significantly faster times, nowhere near the timeout | 03:06 |
| scas | but, in ci, the same task seems to take orders of magnitude longer, at least enough to bump up against the natural timeout | 03:07 |
| scas | for a vm with the size that is in use, i thought i would have something relatively comparable to a quad core processor. is this an incorrect presumption? | 03:10 |
| scas | it's just that 8 minutes vs 30+ 'feels' unreasonable | 03:11 |
| scas | if i need to check my expectations at the gate, that's fine, too | 03:12 |
| *** kiennt2609 has quit IRC | 03:13 | |
| *** TobbeCN has joined #openstack-infra | 03:13 | |
| *** kiennt2609 has joined #openstack-infra | 03:13 | |
| *** yamamoto has joined #openstack-infra | 03:16 | |
| clarkb | running bandersnatch one more time before calling it a day then will resume in the morning and work to get it cron'd again | 03:17 |
| clarkb | scas: we really don't get much insight into how the clouds portion up their computes | 03:18 |
| clarkb | scas: but generally decent local hardware is faster | 03:18 |
| *** TobbeCN has quit IRC | 03:18 | |
| *** ianychoi has joined #openstack-infra | 03:20 | |
| *** yamamoto has quit IRC | 03:21 | |
| ianw | nope, the vhd image i uploaded doesn't work | 03:26 |
| *** yamamoto has joined #openstack-infra | 03:37 | |
| *** caphrim007_ has quit IRC | 03:37 | |
| *** caphrim007 has joined #openstack-infra | 03:39 | |
| *** lpetrut_ has joined #openstack-infra | 03:41 | |
| scas | clarkb: i get that, but i have little ways to gauge how much longer the timeout needs to be without a measure. i could guess and double it, but i don't need to elaborate on what that can lead to | 03:43 |
| *** harlowja has joined #openstack-infra | 03:43 | |
| *** yamamoto has quit IRC | 03:43 | |
| *** caphrim007_ has joined #openstack-infra | 03:45 | |
| *** VW has quit IRC | 03:46 | |
| *** caphrim007 has quit IRC | 03:49 | |
| *** jamesmcarthur has joined #openstack-infra | 03:50 | |
| *** dsariel has joined #openstack-infra | 03:51 | |
| *** slaweq has joined #openstack-infra | 03:51 | |
| scas | i'm going to chalk it up to slow cloud. i've been rechecking every few hours and this one might get it merged | 03:52 |
| scas | i'm trying to avoid too many unnecessary changes to ci since i'm currently working on retooling it | 03:54 |
| *** jamesmcarthur has quit IRC | 03:54 | |
| *** slaweq has quit IRC | 03:56 | |
| *** ramishra has joined #openstack-infra | 03:58 | |
| *** yamamoto has joined #openstack-infra | 04:00 | |
| *** salv-orl_ has quit IRC | 04:04 | |
| *** yamamoto has quit IRC | 04:04 | |
| *** salv-orlando has joined #openstack-infra | 04:04 | |
| *** kiennt2609 has quit IRC | 04:05 | |
| scas | this time it worked. my gut still thinks slow leak in a cloud somewhere | 04:06 |
| *** pbourke has quit IRC | 04:06 | |
| *** pbourke has joined #openstack-infra | 04:06 | |
| *** hongbin_ has quit IRC | 04:07 | |
| *** harlowja has quit IRC | 04:12 | |
| *** lpetrut_ has quit IRC | 04:13 | |
| *** lpetrut_ has joined #openstack-infra | 04:19 | |
| *** gyee has quit IRC | 04:20 | |
| *** yamamoto has joined #openstack-infra | 04:21 | |
| *** udesale has joined #openstack-infra | 04:21 | |
| *** yamamoto has quit IRC | 04:25 | |
| *** dsariel has quit IRC | 04:27 | |
| ianw | OMG i did it! I got a vhd uploaded and booting | 04:28 |
| *** cshastri_ has joined #openstack-infra | 04:33 | |
| *** cshastri has quit IRC | 04:35 | |
| *** yamamoto has joined #openstack-infra | 04:41 | |
| *** lpetrut_ has quit IRC | 04:44 | |
| *** yamamoto has quit IRC | 04:48 | |
| *** yamamoto has joined #openstack-infra | 04:48 | |
| *** aeng has joined #openstack-infra | 04:51 | |
| *** claudiub|2 has joined #openstack-infra | 04:51 | |
| *** cshastri_ has quit IRC | 04:52 | |
| *** jchhatbar has quit IRC | 04:59 | |
| *** jchhatbar has joined #openstack-infra | 05:00 | |
| *** jchhatbar has quit IRC | 05:01 | |
| *** jchhatbar has joined #openstack-infra | 05:01 | |
| *** jchhatbar has quit IRC | 05:02 | |
| *** jchhatbar has joined #openstack-infra | 05:03 | |
| *** pcaruana has joined #openstack-infra | 05:05 | |
| *** cshastri_ has joined #openstack-infra | 05:06 | |
| *** pcichy has joined #openstack-infra | 05:10 | |
| *** cshastri has joined #openstack-infra | 05:11 | |
| *** cshastri_ has quit IRC | 05:13 | |
| *** pcaruana has quit IRC | 05:19 | |
| *** lpetrut_ has joined #openstack-infra | 05:23 | |
| *** pgadiya has joined #openstack-infra | 05:24 | |
| *** pgadiya has quit IRC | 05:24 | |
| *** udesale_ has joined #openstack-infra | 05:33 | |
| *** lpetrut_ has quit IRC | 05:35 | |
| *** udesale has quit IRC | 05:36 | |
| *** zhurong has joined #openstack-infra | 05:36 | |
| *** hamzy has quit IRC | 05:37 | |
| *** krtaylor has quit IRC | 05:37 | |
| *** e0ne has joined #openstack-infra | 05:42 | |
| *** agopi has quit IRC | 05:43 | |
| *** agopi has joined #openstack-infra | 05:43 | |
| *** dsariel has joined #openstack-infra | 05:44 | |
| *** kjackal has joined #openstack-infra | 05:46 | |
| *** hamzy has joined #openstack-infra | 05:47 | |
| *** krtaylor has joined #openstack-infra | 05:50 | |
| *** quiquell|off is now known as quiquell|ruck | 05:57 | |
| *** hashar has joined #openstack-infra | 06:01 | |
| *** kjackal has quit IRC | 06:06 | |
| *** psachin has quit IRC | 06:11 | |
| *** gtrxcb has joined #openstack-infra | 06:13 | |
| *** armaan has joined #openstack-infra | 06:18 | |
| *** gmann has quit IRC | 06:21 | |
| *** TobbeCN has joined #openstack-infra | 06:21 | |
| *** kambiz has quit IRC | 06:23 | |
| *** kiennt2609 has joined #openstack-infra | 06:24 | |
| *** aojea has joined #openstack-infra | 06:25 | |
| *** dhajare|busy has joined #openstack-infra | 06:25 | |
| *** lpetrut has joined #openstack-infra | 06:26 | |
| *** TobbeCN has quit IRC | 06:26 | |
| *** kambiz has joined #openstack-infra | 06:26 | |
| *** jtomasek has joined #openstack-infra | 06:26 | |
| *** jchhatba_ has joined #openstack-infra | 06:30 | |
| *** e0ne has quit IRC | 06:31 | |
| *** dklyle has quit IRC | 06:32 | |
| *** dklyle has joined #openstack-infra | 06:32 | |
| *** jchhatbar has quit IRC | 06:33 | |
| *** dpawlik has joined #openstack-infra | 06:34 | |
| *** dpawlik has quit IRC | 06:38 | |
| *** kiennt2609 has quit IRC | 06:38 | |
| *** shardy has joined #openstack-infra | 06:41 | |
| *** kjackal has joined #openstack-infra | 06:42 | |
| *** ildikov has joined #openstack-infra | 06:43 | |
| *** TobbeCN has joined #openstack-infra | 06:43 | |
| *** jchhatba_ is now known as janki | 06:44 | |
| *** mordred has quit IRC | 06:45 | |
| *** links has joined #openstack-infra | 06:46 | |
| *** dklyle has quit IRC | 06:47 | |
| *** mordred has joined #openstack-infra | 06:48 | |
| *** pcaruana has joined #openstack-infra | 06:50 | |
| *** alexchadin has joined #openstack-infra | 06:59 | |
| *** rcernin has quit IRC | 07:00 | |
| *** zigo has quit IRC | 07:11 | |
| *** gmann has joined #openstack-infra | 07:13 | |
| *** zigo has joined #openstack-infra | 07:14 | |
| *** quiquell|ruck is now known as quiquell|ruck|af | 07:15 | |
| *** quiquell|ruck|af is now known as quique|ruck|afk | 07:15 | |
| *** florianf has joined #openstack-infra | 07:16 | |
| *** jpich has joined #openstack-infra | 07:25 | |
| *** ihar has quit IRC | 07:25 | |
| *** ihar has joined #openstack-infra | 07:26 | |
| *** dhajare|busy has quit IRC | 07:26 | |
| *** links has quit IRC | 07:29 | |
| *** tesseract has joined #openstack-infra | 07:29 | |
| *** udesale__ has joined #openstack-infra | 07:29 | |
| *** zhurong has quit IRC | 07:30 | |
| *** udesale_ has quit IRC | 07:32 | |
| *** psachin has joined #openstack-infra | 07:33 | |
| *** vtapia has joined #openstack-infra | 07:34 | |
| *** dpawlik has joined #openstack-infra | 07:35 | |
| *** amoralej|off is now known as amoralej | 07:38 | |
| *** shardy has quit IRC | 07:39 | |
| *** shardy has joined #openstack-infra | 07:39 | |
| *** dpawlik has quit IRC | 07:40 | |
| *** gtrxcb has quit IRC | 07:42 | |
| *** dpawlik has joined #openstack-infra | 07:45 | |
| *** links has joined #openstack-infra | 07:47 | |
| *** jpena|off is now known as jpena | 07:47 | |
| *** tosky has joined #openstack-infra | 07:48 | |
| *** dizquierdo has joined #openstack-infra | 07:48 | |
| *** yamamoto has quit IRC | 07:49 | |
| *** yamamoto has joined #openstack-infra | 07:49 | |
| *** links has quit IRC | 07:52 | |
| *** yamamoto_ has joined #openstack-infra | 07:56 | |
| *** quique|ruck|afk is now known as quiquell|ruck | 07:58 | |
| *** kjackal has quit IRC | 07:58 | |
| *** dpawlik has quit IRC | 08:00 | |
| *** kjackal has joined #openstack-infra | 08:00 | |
| *** yamamoto has quit IRC | 08:00 | |
| *** dhajare|busy has joined #openstack-infra | 08:02 | |
| *** lucas-afk is now known as lucasagomes | 08:05 | |
| *** dpawlik has joined #openstack-infra | 08:05 | |
| *** dpawlik has quit IRC | 08:09 | |
| *** mugsie has quit IRC | 08:09 | |
| *** electrofelix has joined #openstack-infra | 08:11 | |
| *** sambetts|afk is now known as sambetts | 08:19 | |
| *** mugsie has joined #openstack-infra | 08:21 | |
| *** mugsie has quit IRC | 08:21 | |
| *** mugsie has joined #openstack-infra | 08:21 | |
| *** stakeda has quit IRC | 08:23 | |
| *** slaweq has joined #openstack-infra | 08:23 | |
| *** ssbarnea_ has joined #openstack-infra | 08:27 | |
| *** pcichy has quit IRC | 08:30 | |
| *** markus_z has joined #openstack-infra | 08:35 | |
| *** udesale_ has joined #openstack-infra | 08:35 | |
| *** e0ne has joined #openstack-infra | 08:36 | |
| *** udesale__ has quit IRC | 08:38 | |
| *** zhurong has joined #openstack-infra | 08:39 | |
| *** derekh has joined #openstack-infra | 08:43 | |
| *** agopi has quit IRC | 08:44 | |
| *** dingyichen has quit IRC | 08:46 | |
| *** markus_z has left #openstack-infra | 08:48 | |
| *** links has joined #openstack-infra | 08:48 | |
| *** pgadiya has joined #openstack-infra | 08:57 | |
| *** pgadiya has quit IRC | 08:57 | |
| *** panda|rover|off is now known as panda|rover | 08:59 | |
| *** annp has quit IRC | 09:04 | |
| *** annp has joined #openstack-infra | 09:05 | |
| *** jtomasek has quit IRC | 09:07 | |
| *** kjackal has quit IRC | 09:08 | |
| *** gfidente has joined #openstack-infra | 09:19 | |
| *** gfidente has joined #openstack-infra | 09:19 | |
| *** alexchadin has quit IRC | 09:24 | |
| *** alexchadin has joined #openstack-infra | 09:26 | |
| lucasagomes | hi all, the docs job in the stable/queens for networking-ovn is failing due to http://logs.openstack.org/93/560893/3/check/build-openstack-sphinx-docs/6ad4b0a/job-output.txt.gz#_2018-04-17_07_35_05_375569 | 09:38 |
| lucasagomes | the task where this is failing is the TASK [neutron-horizon-hack : Install extra things needed by tox_install.sh] | 09:39 |
| lucasagomes | anyone has seem this problem ? | 09:39 |
| *** efoley has joined #openstack-infra | 09:40 | |
| *** alexchadin has quit IRC | 09:49 | |
| *** zhurong has quit IRC | 09:57 | |
| *** dizquierdo has quit IRC | 10:04 | |
| *** efoley has quit IRC | 10:04 | |
| *** efoley has joined #openstack-infra | 10:04 | |
| *** zoli is now known as zoli|lunch | 10:07 | |
| *** aluria has quit IRC | 10:21 | |
| *** udesale__ has joined #openstack-infra | 10:21 | |
| lucasagomes | AJaeger, ^ any idea on that neutron-horizon-hack ? | 10:22 |
| *** udesale_ has quit IRC | 10:24 | |
| *** aluria has joined #openstack-infra | 10:25 | |
| *** zhipeng has joined #openstack-infra | 10:27 | |
| *** aluria has quit IRC | 10:27 | |
| *** aluria has joined #openstack-infra | 10:27 | |
| *** pcaruana has quit IRC | 10:27 | |
| *** nicolasbock has joined #openstack-infra | 10:31 | |
| *** boden has joined #openstack-infra | 10:33 | |
| *** dsariel has quit IRC | 10:37 | |
| *** kjackal has joined #openstack-infra | 10:38 | |
| *** slagle has quit IRC | 10:44 | |
| AJaeger | lucasagomes, see : https://etherpad.openstack.org/p/pip10-mitigation - start line 42 | 10:46 |
| AJaeger | mordred: ^ | 10:46 |
| lucasagomes | AJaeger, thanks | 10:46 |
| *** e0ne has quit IRC | 10:48 | |
| *** dsariel has joined #openstack-infra | 10:50 | |
| *** e0ne has joined #openstack-infra | 10:51 | |
| *** rh-jelabarre has joined #openstack-infra | 11:05 | |
| *** zhipeng has quit IRC | 11:08 | |
| *** slagle has joined #openstack-infra | 11:11 | |
| *** HeOS has joined #openstack-infra | 11:13 | |
| *** zoli|lunch is now known as zoli | 11:14 | |
| *** zoli is now known as zoli|wfh | 11:14 | |
| *** zoli|wfh is now known as zoli | 11:14 | |
| *** udesale__ has quit IRC | 11:16 | |
| *** lucasagomes is now known as lucas-hungry | 11:18 | |
| *** zhangfei has quit IRC | 11:19 | |
| *** dhill_ has quit IRC | 11:26 | |
| *** zhipeng has joined #openstack-infra | 11:27 | |
| *** Kevin_Zheng has quit IRC | 11:30 | |
| *** ramishra has quit IRC | 11:32 | |
| *** zhipeng has quit IRC | 11:34 | |
| *** yamamoto_ has quit IRC | 11:36 | |
| *** ldnunes has joined #openstack-infra | 11:37 | |
| *** gmann has quit IRC | 11:51 | |
| *** armaan has quit IRC | 11:52 | |
| *** armaan has joined #openstack-infra | 11:53 | |
| *** neiloy has joined #openstack-infra | 11:54 | |
| *** ramishra has joined #openstack-infra | 11:57 | |
| *** katkapilatova has joined #openstack-infra | 11:57 | |
| *** jpena is now known as jpena|lunch | 12:00 | |
| *** rfolco|off is now known as rfolco | 12:01 | |
| *** katkapilatova has quit IRC | 12:02 | |
| *** yamamoto has joined #openstack-infra | 12:03 | |
| *** tpsilva has joined #openstack-infra | 12:05 | |
| *** armaan has quit IRC | 12:06 | |
| *** armaan has joined #openstack-infra | 12:06 | |
| *** alexchadin has joined #openstack-infra | 12:08 | |
| *** pcichy has joined #openstack-infra | 12:09 | |
| *** amoralej is now known as amoralej|lunch | 12:10 | |
| *** gmann has joined #openstack-infra | 12:15 | |
| *** edmondsw has joined #openstack-infra | 12:15 | |
| *** jtomasek has joined #openstack-infra | 12:15 | |
| *** dbecker has joined #openstack-infra | 12:19 | |
| *** pcaruana has joined #openstack-infra | 12:19 | |
| frickler | infra-root: when someone has time to look into ethercalc.o.o, it seems unresponsive and the log is full of this: [Wed Apr 18 12:18:54.754678 2018] [mpm_event:error] [pid 1663:tid 140180487960448] AH00485: scoreboard is full, not at MaxRequestWorkers | 12:19 |
| *** cshastri has quit IRC | 12:24 | |
| *** alexchadin has quit IRC | 12:26 | |
| *** efoley has quit IRC | 12:27 | |
| *** trown|outtypewww is now known as trown | 12:29 | |
| *** rlandy has joined #openstack-infra | 12:30 | |
| *** lucas-hungry is now known as lucasagomes | 12:31 | |
| *** dsariel has quit IRC | 12:36 | |
| *** alexchadin has joined #openstack-infra | 12:36 | |
| *** quiquell|ruck is now known as quique|ruck|food | 12:37 | |
| *** armaan has quit IRC | 12:40 | |
| *** ramishra_ has joined #openstack-infra | 12:41 | |
| *** ramishra has quit IRC | 12:44 | |
| *** apetrich_ has joined #openstack-infra | 12:48 | |
| *** zhipeng has joined #openstack-infra | 12:48 | |
| *** jamesmcarthur has joined #openstack-infra | 12:51 | |
| *** armaan has joined #openstack-infra | 12:53 | |
| *** apetrich_ has quit IRC | 12:55 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove installed packages before pip install https://review.openstack.org/561479 | 12:56 |
| *** efoley has joined #openstack-infra | 13:00 | |
| *** ssbarnea_ has quit IRC | 13:00 | |
| *** jpena|lunch is now known as jpena | 13:01 | |
| *** kgiusti has joined #openstack-infra | 13:02 | |
| *** ssbarnea_ has joined #openstack-infra | 13:04 | |
| *** Goneri has joined #openstack-infra | 13:04 | |
| *** amoralej|lunch is now known as amoralej | 13:05 | |
| *** dave-mccowan has joined #openstack-infra | 13:06 | |
| *** dhill_ has joined #openstack-infra | 13:09 | |
| *** dave-mccowan has quit IRC | 13:11 | |
| fungi | frickler: the "AH00485: scoreboard is full, not at MaxRequestWorkers" is one we've dealt with before on other servers... if memory serves an apache restart should clear the problem for now but i'll dig for some notes on how we've determined to solve it properly in the past | 13:13 |
| *** pcichy has quit IRC | 13:13 | |
| *** asettle has left #openstack-infra | 13:14 | |
| *** dave-mccowan has joined #openstack-infra | 13:17 | |
| *** cshastri has joined #openstack-infra | 13:18 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Localize build time in build overview https://review.openstack.org/562273 | 13:19 |
| *** dhill_ has quit IRC | 13:19 | |
| *** dhill_ has joined #openstack-infra | 13:19 | |
| fungi | frickler: aha, ethercalc01 is running ubuntu trusty. i want to say moving to xenial gets us an apache where that bug is fixed | 13:20 |
| dhellmann | mordred, corvus, smcginnis: are we still in a 1/2 complete state with the changes around siblings and publishing services to PyPI? We still have project teams confused about how to handle cross-service dependencies: http://lists.openstack.org/pipermail/openstack-dev/2018-April/129455.html | 13:21 |
| *** zoli is now known as zoli|afk | 13:21 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Localize build time in build overview https://review.openstack.org/562273 | 13:23 |
| smcginnis | dhellmann: I thought that was all sorted, so interested if mordred and corvus have any input. | 13:23 |
| dhellmann | smcginnis : I have no idea, that's why I'm highlighting the thread. If we do have an answer, a link to docs should help resolve the confusion. | 13:24 |
| smcginnis | ++ | 13:24 |
| *** quique|ruck|food is now known as quiquell|ruck | 13:24 | |
| mnaser | hmm | 13:25 |
| mnaser | did we get into a change to figure out what's the issue with the debian images | 13:25 |
| AJaeger | dhellmann: I don't think it's sorted - so far I'm aware of the following missing things: push more projects to pypi (see https://review.openstack.org/#/c/531825/ and followups ) , write an announcement and update docs. There might be more - mordred ? | 13:25 |
| dhellmann | do we need to find a volunteer to help move that forward? | 13:26 |
| dhellmann | someone should at least respond to that ML thread explaining that this is something that's in progress, so we don't have teams coming up with a bunch of unique half-baked solutions | 13:27 |
| AJaeger | dhellmann: yes, a volunteer to help might be the best way forward. That one should first discuss with mordred what's missing. Sorry, I can't help further here. | 13:28 |
| * dhellmann wishes we would write more plans down before we start, to avoid situations like this | 13:30 | |
| *** germs has joined #openstack-infra | 13:31 | |
| *** germs has quit IRC | 13:31 | |
| *** germs has joined #openstack-infra | 13:31 | |
| *** mrtenio has joined #openstack-infra | 13:31 | |
| *** mrtenio_ has joined #openstack-infra | 13:31 | |
| *** e0ne has quit IRC | 13:34 | |
| *** trown|brb has joined #openstack-infra | 13:35 | |
| *** trown has quit IRC | 13:35 | |
| frickler | fungi: yes, we started that migration during the december sprint, but got stuck. maybe ianw wants to continue on that | 13:37 |
| *** jamesmcarthur has quit IRC | 13:37 | |
| *** jamesmcarthur has joined #openstack-infra | 13:38 | |
| *** e0ne has joined #openstack-infra | 13:44 | |
| *** trown|brb is now known as trown | 13:45 | |
| *** Kevin_Zheng has joined #openstack-infra | 13:45 | |
| *** udesale__ has joined #openstack-infra | 13:45 | |
| *** psachin has quit IRC | 13:46 | |
| *** mrtenio_ has quit IRC | 13:46 | |
| *** pcichy has joined #openstack-infra | 13:48 | |
| *** eharney has joined #openstack-infra | 13:53 | |
| *** jamesmcarthur has quit IRC | 13:57 | |
| mwhahaha | hey it looks like we're having problems with the build-openstacks-phinx-docs job for our stable/pike branch in tripleo-common, http://logs.openstack.org/51/560851/1/check/build-openstack-sphinx-docs/9102d2a/job-output.txt.gz#_2018-04-18_09_55_52_130609 | 14:00 |
| mwhahaha | any thoughts on who we can poke on that? | 14:01 |
| *** esberglu has joined #openstack-infra | 14:03 | |
| *** kiennt26_ has joined #openstack-infra | 14:04 | |
| frickler | mwhahaha: see L43++ here https://etherpad.openstack.org/p/pip10-mitigation | 14:05 |
| mwhahaha | hrm so the workaround has been sitting for 2 days | 14:06 |
| dulek | frickler: Regarding https://etherpad.openstack.org/p/pip10-mitigation and fixing tools/tox-install.sh - what happens if there's no tools/tox-install.sh? Kuryr-Libnetwork has none: https://github.com/openstack/kuryr-libnetwork/tree/stable/queens | 14:06 |
| frickler | mwhahaha: seems folks don't like the workaround and would want projects to fix their tools/tox-install.sh instead | 14:07 |
| frickler | dulek: if there is none, the whole task is skipped and no error happens | 14:07 |
| mwhahaha | we don't have a tools/tox-install.sh | 14:07 |
| *** pcichy has quit IRC | 14:07 | |
| mwhahaha | oh wait maybe we do in a stable version | 14:07 |
| dulek | frickler: Crap, nevermind, my bad, looking at wrong repo. :P | 14:08 |
| mwhahaha | fine we'll hack it | 14:08 |
| mwhahaha | stupid pip10 | 14:08 |
| frickler | yes, it has been removed from most projects in master and replaced with tox-siblings things iiuc | 14:10 |
| clarkb | dhellmann: that particular case is special isn't it because its depending on an external third party service. Even if we wanted to publish it to pypi I'm not sure we would/could | 14:10 |
| clarkb | or wait I'm mixing up gnocchi and ceilomter nevermind | 14:10 |
| clarkb | in that case /me looks at pypi | 14:11 |
| clarkb | it is far too early in the morning | 14:11 |
| clarkb | looks like we've squatted the name, ceilometer just needs to publish to it. I'm guessing the issue is lack of release jobs? | 14:11 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update test instructions https://review.openstack.org/562289 | 14:12 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update test instructions https://review.openstack.org/562289 | 14:13 |
| *** dave-mccowan has quit IRC | 14:13 | |
| *** bobh has joined #openstack-infra | 14:13 | |
| pabelanger | morning | 14:13 |
| pabelanger | looks like checkpool passed without issue on reprepro for debian, I'm going to push up patches this morning for new update method | 14:14 |
| *** zoli|afk is now known as zoli | 14:14 | |
| *** zoli is now known as zoli|wfh | 14:14 | |
| *** dklyle has joined #openstack-infra | 14:15 | |
| *** zhangfei has joined #openstack-infra | 14:15 | |
| clarkb | bandersnatch ran in ~14 minutes, which is quick enough I've gone ahead and approved the package blacklisting change. I'll get that tested, then do a vos release then release the lock | 14:18 |
| *** rpioso|afk is now known as rpioso | 14:20 | |
| *** eernst has joined #openstack-infra | 14:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create tripleo-ha-utils project (zuul) https://review.openstack.org/556598 | 14:21 |
| pabelanger | mnaser: yah, just finishing up the last of the process for debian. Took longer then expected | 14:21 |
| fungi | clarkb: any idea if we would need to do a complete re-mirror of pypi to clean out the older artifacts for the blacklisted projects? | 14:23 |
| clarkb | fungi: I think so either that or manually clean them out. I'm noticing cupy is another package that may be up for blacklisting | 14:24 |
| clarkb | all these packages are linking against cuda and huge | 14:24 |
| fungi | i wonder how easy it is to manually clean any files out given that they're organized by checksum rather than name | 14:25 |
| fungi | we'd need to map them from the index to figure out what to delete i guess | 14:26 |
| clarkb | or use find but that would probably be slow | 14:26 |
| clarkb | (the filenames don't change just the path) | 14:26 |
| mnaser | pabelanger: np thank you | 14:26 |
| *** dhajare|busy is now known as dhajare | 14:32 | |
| *** jamesmcarthur has joined #openstack-infra | 14:32 | |
| clarkb | and current bandersnatch run almost looks like it is going through the entire list again? | 14:37 |
| clarkb | not sure I quite understand what it is doing | 14:38 |
| clarkb | but looking at strace it appears to be reading all the files all over again for some reason | 14:39 |
| pabelanger | some on mirror-update01.o.o is also using up a lot of memory | 14:39 |
| pabelanger | we look to be swapping | 14:39 |
| pabelanger | just a little | 14:39 |
| clarkb | pabelanger: its 7.7GB of buffers/cache. Likely due to bandersnatch reading all the files | 14:40 |
| clarkb | that memory should be freeable though. I don't expect its causing many issues | 14:40 |
| *** lpetrut has quit IRC | 14:40 | |
| *** hongbin_ has joined #openstack-infra | 14:40 | |
| clarkb | ya 7.6GB available | 14:41 |
| clarkb | now bandersnatch is doing writes so maybe it has to read all those packages if a new one arrives I dunno, its weird but I'll try to be patient with it | 14:42 |
| *** neiloy has quit IRC | 14:42 | |
| *** neiloy has joined #openstack-infra | 14:43 | |
| *** VW has joined #openstack-infra | 14:45 | |
| *** zhangfei has quit IRC | 14:45 | |
| mnaser | infra people might know this but there used to be a domain that you could prefix your ip to and it would resolve a hostname to your ip | 14:46 |
| mnaser | sigh, as i say it, http://xip.io/ | 14:46 |
| clarkb | aha! I can follow th todo file to see where it is in progress | 14:47 |
| *** alexchadin has quit IRC | 14:47 | |
| pabelanger | mnaser: neat, TIL | 14:49 |
| mnaser | yeah it's a neat little hack when you need a hostname that resolves | 14:49 |
| pabelanger | The SSL certificate for ethercalc.openstack.org "(CN: )" has expired! | 14:51 |
| pabelanger | fungi: clarkb: ^ just seen that in email | 14:51 |
| clarkb | pabelanger: neat | 14:51 |
| pabelanger | hmm, seems down to me also | 14:52 |
| clarkb | pabelanger: frickler mentioned it wasn't responding earlier in scrollback | 14:52 |
| clarkb | looked like apache issues | 14:52 |
| pabelanger | ah, right | 14:52 |
| clarkb | fungi suggested a webserver restart | 14:52 |
| pabelanger | trusty too | 14:52 |
| pabelanger | k, let me do that now | 14:52 |
| clarkb | yes etherpad is trusty too | 14:52 |
| clarkb | did I miss adding that server to our cert expiration checker script? | 14:53 |
| *** yamamoto has quit IRC | 14:53 | |
| pabelanger | I actually wonder if it failed because of 404 | 14:53 |
| clarkb | maybe | 14:54 |
| pabelanger | cert is still green for my browser | 14:54 |
| pabelanger | https://ethercalc.openstack.org/ | 14:54 |
| clarkb | that would explain why it went straight to expired instead of will expire too | 14:54 |
| pabelanger | yah | 14:54 |
| *** felipemonteiro has joined #openstack-infra | 14:54 | |
| *** efoley has quit IRC | 14:55 | |
| *** efoley has joined #openstack-infra | 14:55 | |
| fungi | clarkb: yeah, i saw the certcheck e-mail for it but not until after frickler mentioned apache was kaput, so i figured it was a false expiration warning | 14:55 |
| fungi | otherwise we'd have been getting daily warnings about upcoming expiration for something like at least two months before | 14:55 |
| clarkb | ya that is why I was confused | 14:55 |
| clarkb | certcheck is like the one cronspam like email that goes to the top of the pile | 14:56 |
| *** zhangfei has joined #openstack-infra | 14:57 | |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch master: Support for package blacklists https://review.openstack.org/561692 | 14:57 |
| openstackgerrit | Merged openstack-infra/system-config master: Stop mirroring tensorflow and mxnet pypi packages https://review.openstack.org/561694 | 14:57 |
| *** dizquierdo has joined #openstack-infra | 14:57 | |
| *** ramishra_ has quit IRC | 14:59 | |
| *** rajinir has joined #openstack-infra | 15:00 | |
| *** ramishra has joined #openstack-infra | 15:01 | |
| clarkb | there is a package called pyClark | 15:02 |
| clarkb | I don't know what it does but we mirror it | 15:02 |
| *** alexchadin has joined #openstack-infra | 15:02 | |
| clarkb | "Remote error reporting that helps save the day" I approve | 15:03 |
| *** eernst has quit IRC | 15:06 | |
| *** cshastri has quit IRC | 15:09 | |
| *** TobbeCN has quit IRC | 15:09 | |
| clarkb | blacklist appears to have applied as expected, now just waiting for bandersnatch to finish current run so it can run again with blacklist to test it | 15:09 |
| clarkb | we are unfortunately taking forever to mirror one of the blacklisted packages :/ | 15:10 |
| *** eernst has joined #openstack-infra | 15:10 | |
| *** eernst has quit IRC | 15:12 | |
| clarkb | https://pypi.org/project/cupy-cuda80/#files basically anything touching cuda seems to come in at at least 200mb or so | 15:14 |
| fungi | yeesh | 15:15 |
| clarkb | and then there is the different versions of cuda x different versions of python x all the things using cuda | 15:15 |
| *** alexchadin has quit IRC | 15:16 | |
| *** ramishra has quit IRC | 15:17 | |
| *** quiquell|ruck is now known as quiquell|off | 15:17 | |
| clarkb | x the different platforms wheels are built for | 15:17 |
| *** ramishra has joined #openstack-infra | 15:19 | |
| *** zhipeng has quit IRC | 15:19 | |
| clarkb | I'm worried this won't actually finish before I have to head out to the dentist later today | 15:20 |
| fungi | see, you should have gone to the dentist last wednesday instead, like i did ;) | 15:21 |
| *** zhangfei has quit IRC | 15:21 | |
| clarkb | infra-root can I get reviews on https://review.openstack.org/#/c/562070/ I believe this to be the real proper fix for neutrons lack of indexed log files | 15:22 |
| *** kiennt26_ has quit IRC | 15:22 | |
| *** HeOS has quit IRC | 15:23 | |
| pabelanger | +2 | 15:24 |
| corvus | clarkb: oh i get it. neat. :) | 15:24 |
| *** alexchadin has joined #openstack-infra | 15:25 | |
| *** tosky has quit IRC | 15:27 | |
| clarkb | it appears that when syncing package foo, bandersnatch reads every existing local artifact for foo (and checks the hash?) in addition to pulling down whatever things may be missing | 15:28 |
| clarkb | this makes these large pacakges that release often even more problematic beacuse you aren't just syncing the the new stuff but reading all of the existing stuff off the network too | 15:29 |
| *** tosky has joined #openstack-infra | 15:29 | |
| clarkb | corvus: while you are here, what is the incantation for running a vos release the safe way when you aren't sure if the command will run in the auth token's allotted time? | 15:30 |
| clarkb | run vos release -root as root on afs01.dfw.openstack.org? | 15:30 |
| openstackgerrit | Merged openstack-infra/os-loganalyze master: Better supports sev checking https://review.openstack.org/562070 | 15:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/553596 | 15:31 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed summit validation dates rules https://review.openstack.org/562318 | 15:33 |
| corvus | clarkb: run it as root with '-localauth' on any afs server machine. afs01.dfw will work. | 15:33 |
| corvus | clarkb: i think we documented that too... | 15:33 |
| corvus | clarkb: https://docs.openstack.org/infra/system-config/afs.html search for "mirror.foo -localauth" | 15:34 |
| *** lpetrut has joined #openstack-infra | 15:34 | |
| *** eernst has joined #openstack-infra | 15:35 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Refactor run_handler to be generic https://review.openstack.org/535554 | 15:38 |
| *** dklyle has quit IRC | 15:38 | |
| dhellmann | clarkb : this feels like a case where some folks agreed on a plan, and started, and haven't finished, but also haven't really shared the plan clearly. I could be wrong, maybe I just missed an email or spec or whatever. but no one seems to be able to point to a document that describes what folks *should* be doing. :-/ | 15:40 |
| clarkb | dhellmann: ya clarification would be useful | 15:41 |
| clarkb | corvus: thanks@ | 15:41 |
| *** yamahata has quit IRC | 15:42 | |
| *** ramishra has quit IRC | 15:43 | |
| *** armaan has quit IRC | 15:43 | |
| *** armaan has joined #openstack-infra | 15:43 | |
| corvus | dhellmann, clarkb: i see pings to me about this from earlier; i don't mean to ignore it, but i also don't have much information beyond AJaeger -- i have heard that "publish servers to pypi so we can have real deps" is an important first step. | 15:44 |
| dhellmann | corvus : ok, thanks. I think I've heard that, but wasn't sure if there was anything blocking it and don't have time to take it on myself. Perhaps smcginnis or ttx or one of the other release team folks can get involved. | 15:45 |
| corvus | dhellmann: apparently ceilometer is already registered, so maybe we could push that one ahead independently | 15:46 |
| *** beagles has left #openstack-infra | 15:46 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Proposed spec: tenant-scoped admin web API https://review.openstack.org/562321 | 15:46 |
| dhellmann | jd_ : maybe you can help? ^^ | 15:46 |
| *** dhajare has quit IRC | 15:46 | |
| clarkb | ianw: when you wake, was the omg I did it got a vhd to boot meaning you got the pre initramfs update images to boot or did you get a post update image to boot too? | 15:47 |
| fungi | corvus: dhellmann: the only plan i was aware of was "stop not uploading them" (except for a couple where we don't control the name on pypi, and there we follow pep 541 to hopefully obtain control first) | 15:48 |
| jd_ | dhellmann: I'm confused, about what? | 15:48 |
| jd_ | dhellmann: oh ceilometer name on PyPI (I read that backlog too fast) | 15:48 |
| dhellmann | jd_ : the problem described in http://lists.openstack.org/pipermail/openstack-dev/2018-April/129455.html appears to be related to not having ceilometer published to pypi? | 15:48 |
| corvus | dhellmann, jd_, smcginnis, clarkb: my *guess* is that if we made this change https://review.openstack.org/531825 for only ceilometer, then got an infra-root to manually trigger a publish for the latest tag, we could move forward with having a normal dependency in (networking-odl i think?). that's the part of the "plan" that makes sense to me, but anything beyond that is vague. | 15:48 |
| clarkb | corvus: ya that would be my expectation | 15:49 |
| corvus | so, i mean, i provide no warranty on that :) | 15:49 |
| clarkb | unless they need unreleased code | 15:49 |
| jd_ | dhellmann: we could publish to PyPI, it seems me, you and openstackci owns the project | 15:49 |
| dhellmann | corvus : ok. without really understanding what changed that caused this to stop working, I'm not sure what we really need to do. | 15:49 |
| jd_ | dhellmann: corvus: isn't that weird to depend on Ceilometer in a requirement? do they load code from it? | 15:49 |
| dhellmann | jd_ : yeah, I already have 3 other "touch all the projects" things going on right now so I don't have time to take on "publish all service projects" | 15:50 |
| AJaeger | corvus: we don't even need publishing now for CI - if ceilometer is in required projects, we install it from there. But we need to publish for sure later... | 15:50 |
| dhellmann | jd_ : that may be something to investigate | 15:50 |
| corvus | jd_: i'm not familiar enough with the problem to say. | 15:50 |
| jd_ | ok let me do a quickcheck | 15:50 |
| AJaeger | jd_: it's weird - I checked they do use it but didn't check for what reason | 15:51 |
| clarkb | dhellmann: I think what likely changed is with tox siblings we are pip installing the siblings, that coupled with the new dep leads to this problem | 15:51 |
| corvus | AJaeger: okay, i somehow assumed, i guess, that the problem was that we were trying to do an install-from-source on a project that doesn't have an old tox_install.sh script. | 15:51 |
| clarkb | dhellmann: the thread explains the differences between pip install -r reqs-file and pip install $project | 15:51 |
| jd_ | they provide a set of plugin for Ceilometer, that's why they need it | 15:51 |
| *** udesale__ has quit IRC | 15:52 | |
| jd_ | I think it was designed to be done this way at some point, but nobody did it, and we broke it multiple times, so that's a bold move. :) | 15:52 |
| *** hemna_ has quit IRC | 15:52 | |
| dhellmann | clarkb : yeah, I don't fully understand the siblings changes. I know we've changed how we deal with those dependencies but I haven't had time to learn the new system | 15:52 |
| jd_ | but it'd make sense to publish to pypi for that use case | 15:52 |
| *** eernst has quit IRC | 15:52 | |
| clarkb | dhellmann: the short version is we run tox but tell it to only install deps. Then for each required project on the job we pip install that project into the tox env. Then finally we run tox again telling it to only run the env command and not install any deps | 15:53 |
| corvus | i wish i knew which job was failing here, the thread never states that? | 15:53 |
| *** alexchadin has quit IRC | 15:53 | |
| *** yamamoto has joined #openstack-infra | 15:53 | |
| clarkb | the bug doesn't say either | 15:54 |
| clarkb | (it may not even be a siblings issue I guess) | 15:54 |
| *** links has quit IRC | 15:55 | |
| AJaeger | So, indeed the pip change broke the old tox-isntall scripts - but some might have been broken before but nobody noticed. With tox-siblings calling tox-install if needed, we had a different path of invocation | 15:55 |
| *** agopi has joined #openstack-infra | 15:57 | |
| openstackgerrit | Merged openstack-infra/zuul master: Make gearman queries asynchronous in github driver https://review.openstack.org/560044 | 15:57 |
| *** yamamoto has quit IRC | 16:00 | |
| corvus | i have to afk for a few hours | 16:05 |
| *** eernst has joined #openstack-infra | 16:06 | |
| *** harlowja has joined #openstack-infra | 16:06 | |
| *** slaweq has quit IRC | 16:07 | |
| *** jpich has quit IRC | 16:07 | |
| *** r-daneel has joined #openstack-infra | 16:07 | |
| *** jamesmcarthur has quit IRC | 16:07 | |
| openstackgerrit | Emilien Macchi proposed openstack/diskimage-builder master: Run tripleo-ci-centos-7-nonha-multinode-oooq as voting https://review.openstack.org/562328 | 16:07 |
| *** efried is now known as efried_rollin | 16:07 | |
| *** jamesmcarthur has joined #openstack-infra | 16:07 | |
| *** slaweq has joined #openstack-infra | 16:07 | |
| *** r-daneel has quit IRC | 16:08 | |
| *** owalsh is now known as owalsh_afk | 16:08 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed summit validation dates rules https://review.openstack.org/562318 | 16:09 |
| pabelanger | mnaser: zigo: I've released the debian mirror in AFS, stretch should be working again | 16:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 16:11 |
| *** slaweq has quit IRC | 16:13 | |
| *** florianf has quit IRC | 16:16 | |
| *** yamahata has joined #openstack-infra | 16:17 | |
| *** jamesmcarthur has quit IRC | 16:19 | |
| *** gyee has joined #openstack-infra | 16:30 | |
| fungi | i may need help from someone with significantly more ansible background to help me figure out why launch_node.py's --environment option has regressed | 16:32 |
| fungi | i've dug pretty far into ansible-role-puppet and am not spotting anything seriously wrong with how this is plumbed through from the ad_hoc playbook in system-config | 16:33 |
| *** lpetrut has quit IRC | 16:35 | |
| *** lucasagomes is now known as lucas-afk | 16:35 | |
| *** panda|rover is now known as panda|rover|off | 16:36 | |
| fungi | http://paste.openstack.org/show/719493/ is what the tail end of the failed run looks like | 16:36 |
| *** jamesmcarthur has joined #openstack-infra | 16:37 | |
| fungi | at line 60 you can see that it's looking for /opt/system-config/production/manifests/site.pp and tripping the error checking at https://git.openstack.org/cgit/openstack-infra/ansible-role-puppet/tree/library/puppet#n194 | 16:38 |
| fungi | however line 84 of the paste indicates it did include -e puppet_environment=fungi in the ansible-playbook command line | 16:40 |
| anteaya | fungi does site.pp appear elsewhere? | 16:40 |
| anteaya | or no site.pp anywhere? | 16:40 |
| fungi | yes, if i log into the broken instance i can see it created /opt/system-config/fungi/manifests/site.pp | 16:40 |
| fungi | but not /opt/system-config/production/manifests/site.pp | 16:40 |
| anteaya | hrumph | 16:41 |
| clarkb | fungi: it seems that library/puppet doesn't use puppet_environment at all? | 16:41 |
| clarkb | ah it becomes just environment | 16:41 |
| fungi | clarkb: it's set by the playbook | 16:41 |
| fungi | er, by teh task | 16:41 |
| fungi | yeah | 16:41 |
| fungi | the task sets environment from puppet_environment | 16:42 |
| fungi | or at least is trying to as far as i can tell | 16:42 |
| *** shardy has quit IRC | 16:42 | |
| fungi | if i can work out how to bring my own ansible-role-puppet source, i'd try stuffing some debugging statements in there | 16:43 |
| fungi | but not sure how to go about telling system-config to use something other than the upstream role | 16:43 |
| *** eernst has quit IRC | 16:44 | |
| mtreinish | fungi: did mosquitto crash this morning when cron.daily ran? | 16:46 |
| mtreinish | things still look up to me | 16:46 |
| fungi | clarkb: on a whim i tried reverting corvus's https://review.openstack.org/371759 to see if that's what inadvertently introduced the regression, but no dice | 16:47 |
| clarkb | fungi: it is coming from system-config/playbooks/group_vars/all.yaml | 16:47 |
| fungi | mtreinish: no crash today | 16:48 |
| fungi | maybe crash tomorrow | 16:48 |
| clarkb | fungi: I think if you set another -e manifest=/path/to/fungi/manifest that would override the default in group vars | 16:49 |
| clarkb | bandersnatch is moving again finally go through mxnet | 16:49 |
| clarkb | some of mxnet, I guess it has multiple workers | 16:50 |
| mtreinish | fungi: hmm, ok | 16:50 |
| clarkb | another improvement may be to increase the worker count | 16:50 |
| *** jamesmcarthur has quit IRC | 16:50 | |
| fungi | clarkb: ooh, that's a fun observation... strangely we already have sane defaults set for these at https://git.openstack.org/cgit/openstack-infra/ansible-role-puppet/tree/defaults/main.yml#n15 so i wonder if system-config can just stop passing those in? | 16:51 |
| clarkb | oh they want it set at 3 to not overwhelm pypi itself, so thats set to the worker value we want I guess | 16:51 |
| fungi | clarkb: or, yeah, interpret them based on the puppet_environment value? | 16:51 |
| clarkb | fungi: manifest isn't set in the role, its the one required parameter I think | 16:51 |
| clarkb | but ya maybe interpret it based on the environment parameter | 16:52 |
| fungi | so something like manifest: /opt/system-config/{{ puppet_environment | default(omit) }}/manifests/site.pp | 16:52 |
| clarkb | default(production) I think | 16:53 |
| fungi | er, no, default(production) i guess | 16:53 |
| fungi | yeah, that | 16:53 |
| *** szaher has quit IRC | 16:53 | |
| clarkb | arg the tensorflow packages are also in the todo list for bandersnatch definitely not going to get this done before I have to dentist | 16:54 |
| clarkb | the blacklist is in place now though | 16:54 |
| pabelanger | clarkb: fungi: okay, my fixes for debian-security don't see to work. At this point, I'm of the mind we might just want to stop mirroring that as reprepro doesn't seem to support that setup | 16:55 |
| *** yamamoto has joined #openstack-infra | 16:56 | |
| anteaya | when you have to dentist, you have to dentist | 16:56 |
| clarkb | pabelanger: is it mirroring something different? | 16:56 |
| pabelanger | clarkb: i don't think so, but we are getting checksum failures for packages between jessie and stretch, removing them doesn't seem to fix it as reprepro will redownload them and error next run | 16:57 |
| pabelanger | I am unsure how upstream security.debian.org works, but our mirror fails | 16:57 |
| pabelanger | Oh | 16:58 |
| pabelanger | you know what | 16:58 |
| pabelanger | I think it is because stretch-updates and stretch-security are actually 2 different pools | 16:58 |
| pabelanger | upstream, and we are trying to do one downstream | 16:58 |
| pabelanger | so, I think we could try having mirror.debian and mirror.debian-security for AFS volumes | 16:59 |
| *** bhujay has joined #openstack-infra | 17:00 | |
| *** derekh has quit IRC | 17:00 | |
| fungi | yes, updates and security are different package lists | 17:00 |
| *** yamamoto has quit IRC | 17:00 | |
| pabelanger | yah, we also need to do that | 17:00 |
| fungi | security is basically the high-priority channel for updates which provide security fixes | 17:01 |
| pabelanger | so, I propose, we remove security from mirror.debian, update zuul-jobs to just use upstream security.debian.org, then if we want to mirror, create mirror.debian-security and update zuul-jobs again to use that | 17:01 |
| fungi | and proposed-updates is for non-security-related updates | 17:01 |
| *** markmcd has quit IRC | 17:01 | |
| fungi | and both eventually get rolled down into the normal stable list periodically (every few months or so) as stable point releases | 17:02 |
| *** e0ne has quit IRC | 17:02 | |
| *** bhujay has quit IRC | 17:03 | |
| *** szaher has joined #openstack-infra | 17:04 | |
| *** dhajare has joined #openstack-infra | 17:04 | |
| *** bhujay has joined #openstack-infra | 17:04 | |
| *** markmcd has joined #openstack-infra | 17:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove debian-security from reprepro https://review.openstack.org/562348 | 17:07 |
| pabelanger | fungi: clarkb: ^is step 1 | 17:07 |
| pabelanger | I'll update zuul-jobs in a moment | 17:07 |
| *** amoralej is now known as amoralej|off | 17:07 | |
| fungi | clarkb: so substituting like manifest: /opt/system-config/{{ puppet_environment | default(production) }}/manifests/site.pp doesn't seem to have helped, but regardless the error isn't getting raised in playbooks/group_vars/all.yaml it's coming from ansible-role-puppet via playbooks/remote_puppet_adhoc.yaml which doesn't seem to use the group_vars playbook | 17:08 |
| fungi | unless i'm completely misunderstanding the relationship between those | 17:09 |
| clarkb | fungi: I think the group_vars are just defaults for the playbooks that we do run | 17:09 |
| clarkb | fungi: since the file is called all.yaml it applies to the group all (which matches all hosts) | 17:09 |
| *** eernst has joined #openstack-infra | 17:09 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Support winrm hosts in static driver https://review.openstack.org/561256 | 17:10 |
| clarkb | fungi: I don't know how it knows to lookup the group_vars via a specific path though. Maybe it is relative to your edits? | 17:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Switch to http://security.debian.org/ for debian https://review.openstack.org/562349 | 17:11 |
| pabelanger | clarkb: fungi: ^2nd | 17:11 |
| *** gyee has quit IRC | 17:12 | |
| *** jpena is now known as jpena|off | 17:12 | |
| *** trown is now known as trown|lunch | 17:12 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 17:12 |
| fungi | clarkb: what's interesting is that in this task ansible-role-puppet seems to dynamically work out the manifest path based on puppet_environment: https://git.openstack.org/cgit/openstack-infra/ansible-role-puppet/tree/tasks/main.yml#n32 | 17:14 |
| *** eernst has quit IRC | 17:15 | |
| pabelanger | clarkb: do we want to move forward on https://review.openstack.org/562004/ (revert to fix glean) and take a new release of DIB? ianw left a +2 | 17:17 |
| clarkb | fungi: ya I think its just the manifest value itself that is not being set properly. I would expect your change above to fix it but that may depend on what group vars are actually being applied when it runs the playbook | 17:17 |
| clarkb | pabelanger: I think I'm ok with it if ianw is ok with it. I won't be around very soon to help babysit any additional fallout of the pip10 rollout though | 17:17 |
| clarkb | so I probably shouldn't +3 it | 17:17 |
| fungi | clarkb: right, which it may not be finding i guess if it doesn't look in the current checkout/source tree for the playbook to do the group vars setting? | 17:18 |
| clarkb | fungi: ya it might be relative to the /etc/ansible config? | 17:18 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 17:19 |
| fungi | interesting there's a /etc/ansible/roles/puppet.pabelanger and /etc/ansible/roles/puppet.clarkb.bak on there... i guess that's how i would create a temporary environment-specific fork of ansible-role-puppet? | 17:20 |
| clarkb | fungi: more likely I made a copy of the old version and edited the actual one in place | 17:21 |
| fungi | ahh | 17:21 |
| fungi | nm | 17:21 |
| pabelanger | clarkb: ack, ianw is online shortly, lets see what he says | 17:22 |
| *** sambetts is now known as sambetts|afk | 17:23 | |
| fungi | clarkb: as a workaround, next i'm trying to explicitly add -e manifest=... in the ansible-playbook command line at https://git.openstack.org/cgit/openstack-infra/system-config/tree/launch/launch-node.py#n205 | 17:26 |
| fungi | fingers crossed | 17:26 |
| clarkb | fungi: aiui -e vars always win so that should work | 17:26 |
| *** wolverineav has joined #openstack-infra | 17:30 | |
| fungi | clarkb: indeed, that has gotten me further along. now i'm battling my inability to correctly set things in hiera ;) | 17:30 |
| fungi | i'm going to go grab a quick lunch and then pick this back up when i return | 17:31 |
| *** wolverineav has quit IRC | 17:32 | |
| *** wolverineav has joined #openstack-infra | 17:32 | |
| *** electrofelix has quit IRC | 17:33 | |
| *** Kevin_Zheng has quit IRC | 17:34 | |
| *** esberglu has quit IRC | 17:35 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 17:35 |
| anteaya | fungi yay progress, happy lunch | 17:36 |
| *** janki has quit IRC | 17:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Switch to http://security.debian.org/ for debian https://review.openstack.org/562349 | 17:38 |
| *** bobh has quit IRC | 17:41 | |
| clarkb | ok I need to afk now to get ready for dentist then drive over there. bandersnatch is still chewing away at things. I'm hoping it is done by the time I am done with dentist then will restart it with the blacklist, make sure that works, vos release the results and then release the file lock | 17:43 |
| clarkb | udata and cupy are two addtional packages we might consider blacklisting | 17:43 |
| clarkb | basically there are a lot of packages all releasing daily updates now and they are not small | 17:43 |
| clarkb | part of me wonders if the warehouse switch is to blame | 17:43 |
| *** nikeshm has joined #openstack-infra | 17:43 | |
| clarkb | (but I doubt it ) | 17:44 |
| *** harlowja has quit IRC | 17:44 | |
| nikeshm | hi | 17:46 |
| nikeshm | did any one face below issue | 17:46 |
| nikeshm | 2018-04-17 10:17:12.330 | venv-tempest installdeps: -chttps://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt, -r/opt/stack/new/tempest/requirements.txt 2018-04-17 11:18:29.209 | venv-tempest develop-inst: /opt/stack/new/tempest | 17:46 |
| nikeshm | venv-tempest installdeps: was stucking for more than 1 hour | 17:46 |
| *** esberglu has joined #openstack-infra | 17:47 | |
| *** aojea has quit IRC | 17:47 | |
| nikeshm | then it is automatically fixed after some hours | 17:47 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 17:51 |
| *** jamesmcarthur has joined #openstack-infra | 17:54 | |
| anteaya | clarkb, happy dentist | 17:56 |
| *** jcoufal has joined #openstack-infra | 17:56 | |
| *** yamamoto has joined #openstack-infra | 17:56 | |
| *** dave-mccowan has joined #openstack-infra | 17:57 | |
| anteaya | nikeshm, that is from a log from yesterday, are you experiencing the same issue today? | 17:57 |
| nikeshm | no | 17:59 |
| *** dhajare has quit IRC | 17:59 | |
| nikeshm | just wanted to know that did any one faced that | 17:59 |
| anteaya | sometimes installing dependencies can take a while, I'm not sure if taking one hour is usual or not | 18:00 |
| anteaya | nikeshm, well I can't speak for anyone else but I haven't seen anyone else mention it, some of our other infra folks are offline at the moment so someone may have a different response in a bit | 18:01 |
| nikeshm | ok no problem, i just thought to share my experience :) | 18:01 |
| *** yamamoto has quit IRC | 18:02 | |
| nikeshm | i faced this for couple of hours and then it fixed automatically | 18:02 |
| *** lpetrut has joined #openstack-infra | 18:03 | |
| *** efoley has quit IRC | 18:03 | |
| *** zoli|wfh is now known as zoli|gone | 18:03 | |
| *** eernst has joined #openstack-infra | 18:04 | |
| anteaya | nikeshm, always nice to hear other's experiences, thank you :) | 18:07 |
| *** eernst has quit IRC | 18:08 | |
| *** eernst has joined #openstack-infra | 18:09 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 18:09 |
| *** lpetrut has quit IRC | 18:10 | |
| *** pcaruana has quit IRC | 18:12 | |
| *** gfidente is now known as gfidente|afk | 18:12 | |
| *** eernst has quit IRC | 18:13 | |
| *** jcoufal has quit IRC | 18:13 | |
| *** bobh has joined #openstack-infra | 18:16 | |
| *** bhujay has quit IRC | 18:17 | |
| *** lpetrut has joined #openstack-infra | 18:18 | |
| *** harlowja has joined #openstack-infra | 18:18 | |
| *** rkukura has quit IRC | 18:22 | |
| *** harlowja_ has joined #openstack-infra | 18:23 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 18:24 |
| *** harlowja has quit IRC | 18:24 | |
| *** trown|lunch is now known as trown | 18:30 | |
| *** e0ne has joined #openstack-infra | 18:32 | |
| *** bobh has quit IRC | 18:39 | |
| *** bobh has joined #openstack-infra | 18:40 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 18:40 |
| pabelanger | No viable v4 or v6 route found to git.openstack.org. The build node is assumed to be invalid. | 18:41 |
| pabelanger | 2nd time I've seen that today | 18:41 |
| pabelanger | rax-dfw | 18:43 |
| *** rkukura has joined #openstack-infra | 18:44 | |
| *** Goneri has quit IRC | 18:50 | |
| *** tosky has quit IRC | 18:52 | |
| *** tosky has joined #openstack-infra | 18:55 | |
| *** yamamoto has joined #openstack-infra | 18:57 | |
| clarkb | ya I pushed a zuul jobs change to debug that more | 19:01 |
| clarkb | by capturing the traceroute output | 19:01 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 19:01 |
| *** felipemonteiro_ has joined #openstack-infra | 19:01 | |
| *** yamamoto has quit IRC | 19:03 | |
| *** HeOS has joined #openstack-infra | 19:03 | |
| *** bobh has quit IRC | 19:04 | |
| *** felipemonteiro has quit IRC | 19:05 | |
| *** nikeshm has quit IRC | 19:07 | |
| *** ssbarnea_ has quit IRC | 19:10 | |
| *** numans has quit IRC | 19:12 | |
| *** kmadac has joined #openstack-infra | 19:12 | |
| *** numans has joined #openstack-infra | 19:13 | |
| *** pcaruana has joined #openstack-infra | 19:13 | |
| kmadac | Hi, i'm trying to run tempest tests against my Pike version of private openstack cloud. Each test fails with error NotFound: Object not found, Details: {u'message': u'Could not find domain: default.', u'code': 404, u'title': u'Not Found'}. Domain `default` is created. Any clue where could be the problem? | 19:15 |
| *** tesseract has quit IRC | 19:15 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:16 | |
| edmondsw | PowerVMStackers was recently created, and with that nova-powervm came under governance. We will use openstack/release commits to tag Rocky and future releases, but should we continue to tag older releases the way we did before? | 19:17 |
| *** hrw has quit IRC | 19:19 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 19:19 |
| AJaeger | edmondsw: best ask on #openstack-release channel | 19:20 |
| *** felipemonteiro_ has quit IRC | 19:20 | |
| AJaeger | edmondsw: there was also a recent email on openstack-dev | 19:20 |
| edmondsw | AJaeger oh drat, I clicked the wrong one, meant to be there :) | 19:20 |
| *** pcaruana has quit IRC | 19:20 | |
| AJaeger | edmondsw: see http://lists.openstack.org/pipermail/openstack-dev/2018-April/129362.html | 19:20 |
| fungi | kmadac: you might have more luck in #openstack-qa as that's where the qa team who maintains tempest hangs out | 19:21 |
| edmondsw | AJaeger I saw that, but not sure it's really related | 19:21 |
| kmadac | fungi: thanks, i'll try to ask there | 19:21 |
| edmondsw | I guess maybe it is... if you lose all rights to tag separately, you'd have to tag via openstack/releases | 19:21 |
| fungi | edmondsw: that ml thread was acknowledging that official deliverables will only be able to be tagged through release automation now, and teams can't push their own tags directly any longer | 19:22 |
| edmondsw | fungi yeah, dots connecting... | 19:23 |
| edmondsw | thanks | 19:23 |
| *** efried_rollin is now known as efried | 19:24 | |
| AJaeger | edmondsw, fungi, nova-powervm was not changed according to that email - smcginnis waits for an answer by the powervm team AFAIU | 19:28 |
| fungi | aha, makes sense | 19:28 |
| openstackgerrit | Merged openstack-infra/python-storyboardclient master: remove sphinx-isms from README.rst and clean up docs https://review.openstack.org/545447 | 19:28 |
| edmondsw | AJaeger I'm actually looking for that email and I can't find it in my inbox | 19:28 |
| edmondsw | the one I had seen was the earlier thread... I never got this one | 19:28 |
| edmondsw | so I'm not sure how to reply... | 19:28 |
| fungi | AJaeger: edmondsw: so anyway, this is an opportunity to provide an answer i suppose ;) | 19:28 |
| fungi | edmondsw: you could just let smcginnis know in #openstack-release | 19:29 |
| edmondsw | sure | 19:29 |
| corvus | clarkb, fungi: i'm back | 19:29 |
| fungi | corvus: we sort of figured out a workaround to the launch-node.py --environment regression | 19:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of secrets https://review.openstack.org/553041 | 19:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of nodesets https://review.openstack.org/553088 | 19:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 19:32 |
| corvus | fungi: cool, anything i should review? | 19:34 |
| *** bobh has joined #openstack-infra | 19:35 | |
| *** imacdonn has quit IRC | 19:35 | |
| *** imacdonn has joined #openstack-infra | 19:36 | |
| fungi | corvus: not yet but i'll upload a ~one-liner here shortly once i figure out whether anything else is broken in relation to it | 19:38 |
| *** felipemonteiro__ has quit IRC | 19:39 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:40 | |
| *** kmadac has quit IRC | 19:44 | |
| *** caphrim007_ has quit IRC | 19:45 | |
| *** caphrim007 has joined #openstack-infra | 19:45 | |
| *** ssbarnea_ has joined #openstack-infra | 19:47 | |
| *** e0ne has quit IRC | 19:56 | |
| corvus | now that those tests are passing, i need to try to pick up where i left off a month ago :| | 19:57 |
| openstackgerrit | Merged openstack-infra/storyboard master: Fix PEP8 failures for H236, remove unneeded ignores, use newer hacking https://review.openstack.org/558607 | 19:58 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 19:58 |
| *** yamamoto has joined #openstack-infra | 19:59 | |
| *** eharney has quit IRC | 20:01 | |
| *** yamamoto has quit IRC | 20:04 | |
| clarkb | bandersnatch is still going :( working on tf-nightly though which we will blacklist | 20:07 |
| openstackgerrit | Merged openstack-infra/zuul-website-media master: Add license and zuul.yaml https://review.openstack.org/561747 | 20:12 |
| *** wolverin_ has joined #openstack-infra | 20:14 | |
| *** wolverineav has quit IRC | 20:14 | |
| SotK | does anyone have a bit of time to approve https://review.openstack.org/#/c/555237/ ? | 20:18 |
| openstackgerrit | Merged openstack-infra/zuul-website master: Merge zuul-website-media when publishing site https://review.openstack.org/561749 | 20:18 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 20:19 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Pass manifest path to launch with --environment https://review.openstack.org/562375 | 20:21 |
| fungi | corvus: clarkb: that ^ was the workaround | 20:21 |
| fungi | SotK: i'll take a look | 20:21 |
| openstackgerrit | megan guiney proposed openstack-infra/subunit2sql master: Amend releasenotes for new subunit2sql functions and options https://review.openstack.org/562376 | 20:22 |
| *** kgiusti has left #openstack-infra | 20:24 | |
| fungi | SotK: i'll whip up a quick follow-on series to move that and the default above it to use example.com after overriding in system-config | 20:25 |
| corvus | fungi: i have withdrawn my +A; we should modify in place. | 20:26 |
| tonyb | kolla (jeffrey4l): Have asked to EOL openstack/kolla for newton http://lists.openstack.org/pipermail/openstack-dev/2018-April/129366.html can we do the bootstrappers dance at somestage today ? | 20:26 |
| corvus | fungi, SotK: oh my, that module is full of openstackisms | 20:27 |
| fungi | corvus: yeah, i'm fine moving out the openstackist defaults in a follow-on | 20:27 |
| *** edwarnicke_ is now known as edwarnicke | 20:27 | |
| fungi | i'm hacking on that real quick | 20:27 |
| corvus | fungi, SotK: tell you what, i'm going to +0 it. i can't approve a change like that, but i can ignore it for a brief time. | 20:27 |
| fungi | wfm | 20:27 |
| fungi | i'll have that up in just a moment anyway | 20:28 |
| clarkb | bandersnatch is almost finally done | 20:28 |
| clarkb | I'll rerun it again just to confirm the blacklist works and then vos release. Do we want to point mirrors at the RW volume before I vos release? from what I can tell none of the files moved | 20:29 |
| *** lpetrut has quit IRC | 20:30 | |
| anteaya | fungi thanks for finding a fix for setting --environment | 20:33 |
| *** mriedem has joined #openstack-infra | 20:34 | |
| fungi | anteaya: yep, getting much further along booting the trial survey01.openstack.org now | 20:34 |
| anteaya | yay progress | 20:34 |
| *** eernst has joined #openstack-infra | 20:36 | |
| clarkb | ok I think I'm ready to vos release unless people want to check the output or test the RW volume first | 20:36 |
| *** ssbarnea_ has quit IRC | 20:37 | |
| clarkb | (I've restarted bandersnatch again just to catch up since the previous one was quick) | 20:37 |
| *** ssbarnea_ has joined #openstack-infra | 20:37 | |
| clarkb | ianw: pabelanger: have we done any specialized testing in the past or just gone for it? | 20:38 |
| corvus | clarkb: you could run your own pip install pointing at the rw volume | 20:41 |
| clarkb | corvus: does that work when you use the hashed indexes? | 20:42 |
| clarkb | I guess I could put a web server in front of it too | 20:42 |
| *** yamamoto has joined #openstack-infra | 20:44 | |
| clarkb | I'm going to take the easy route and update one of the citycloud mirrors to use RW volume and test that | 20:45 |
| clarkb | since citycloud isn't being used | 20:45 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 20:45 |
| corvus | clarkb: sounds good | 20:45 |
| *** dmellado has quit IRC | 20:45 | |
| *** gouthamr has quit IRC | 20:46 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-storyboard master: Set default_url for email notification https://review.openstack.org/555237 | 20:47 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-storyboard master: Clean up OpenStack-isms https://review.openstack.org/562381 | 20:47 |
| *** armaan has quit IRC | 20:47 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Override the default_url for StoryBoard servers https://review.openstack.org/562382 | 20:47 |
| fungi | corvus: SotK: ^ there we go. squeaky clean | 20:48 |
| *** armaan has joined #openstack-infra | 20:48 | |
| *** stevebaker has quit IRC | 20:48 | |
| clarkb | http://mirror.la1.citycloud.openstack.org/pypi/ is now using the rw volume | 20:49 |
| clarkb | testing an install now | 20:49 |
| *** armaan has quit IRC | 20:53 | |
| clarkb | it helps to have the /simple on the end of your url | 20:54 |
| *** lyz has joined #openstack-infra | 20:54 | |
| *** dbecker has quit IRC | 20:55 | |
| clarkb | if this ansible install works I'm going to do the vos release | 20:55 |
| clarkb | I have successfully updated pip to pip 10 in my venv | 20:55 |
| ianw | clarkb: i've always just gone for it :) | 20:55 |
| *** armaan has joined #openstack-infra | 20:56 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Use charset=utf8mb4 in pymysql DBURIs https://review.openstack.org/557498 | 20:56 |
| ianw | pabelanger: i'm guessing no further thoughts on https://review.openstack.org/#/c/562004/? i dumped a bunch of info in there, but still has me stumped | 20:57 |
| clarkb | ansible successfully installed. I'm going to do the vos release now then when that is done remove my bandersnatch lock | 20:58 |
| *** dave-mccowan has quit IRC | 20:59 | |
| clarkb | the human readable package index pages are a nice fix too. But looks like we'll only generate those for packages when they update | 21:00 |
| ianw | clarkb: does that increase the size? | 21:02 |
| pabelanger | ianw: yah, same. I have no idea why that affected glean | 21:02 |
| clarkb | ianw: http://mirror.la1.citycloud.openstack.org/pypi/simple/tf-nightly/ vs http://mirror.dfw.rax.openstack.org/pypi/simple/pip/ just some new <br> or <p> tags I expect | 21:02 |
| pabelanger | I am guessing we'll want to spend a bunch of time trying to break glean and see if we need to change anything | 21:02 |
| clarkb | ianw: probably not much bigger | 21:02 |
| clarkb | I should've added a -v to the vos release. Oh well | 21:03 |
| *** eernst has quit IRC | 21:05 | |
| ianw | clarkb: oh ok, not bundling angular :) | 21:06 |
| *** slaweq has joined #openstack-infra | 21:08 | |
| clarkb | corvus: vos release is typically very quiet I take it? | 21:08 |
| *** mrtenio has quit IRC | 21:08 | |
| ianw | fungi / clarkb: in between this, could you see if you have any other thoughts on the glean boot 562004 as i think you two have had some interaction with it | 21:08 |
| corvus | clarkb: yep | 21:09 |
| *** bnemec has quit IRC | 21:09 | |
| *** lyz has left #openstack-infra | 21:09 | |
| clarkb | ianw: I'm ok with it if you are. I don't want to break any lvm users either but I expect we have fewer of those than glean users? I'm not really in a great spot to help debug any new pip10 fallout that happens if we get xenial builds working again | 21:09 |
| ianw | i can dib release when that is it ... but that may release the pip10 kraken more fully. i don't think there is an ideal time to do that; we're probably far enough from 5pm friday to do it soon | 21:09 |
| corvus | clarkb: vos status afs01.dfw.openstack.org | 21:09 |
| clarkb | ianw: ya I think we need to rip that bandaid off | 21:10 |
| clarkb | corvus: thansk | 21:10 |
| *** eandersson has quit IRC | 21:10 | |
| clarkb | ianw: I've +2'd it but not approved in case anyone else wants to review it but also because etoomanydistractions | 21:10 |
| ianw | np, thanks, i'll merge soon | 21:11 |
| *** bnemec has joined #openstack-infra | 21:11 | |
| *** trown is now known as trown|outtypewww | 21:11 | |
| corvus | clarkb: looks like it's done now? | 21:12 |
| *** slaweq has quit IRC | 21:12 | |
| clarkb | Released volume mirror.pypi successfully <- yup | 21:13 |
| clarkb | I'm releasing my hold of the lock now | 21:13 |
| clarkb | next run in a couple minutes | 21:13 |
| *** eandersson has joined #openstack-infra | 21:13 | |
| *** stevebaker has joined #openstack-infra | 21:14 | |
| *** ldnunes has quit IRC | 21:14 | |
| *** jamesmcarthur has quit IRC | 21:14 | |
| clarkb | I've restored mirror.la1.citycloud to its previous use RO volume state | 21:16 |
| *** eernst has joined #openstack-infra | 21:16 | |
| clarkb | bandersnatch is running, has 284 packages to update | 21:16 |
| clarkb | a followup if anyone is interested would be to run the run-bandersnatch script under python3 instead of python2 | 21:18 |
| clarkb | but its working fine to have python2 fork and exec a python3 so not urgent beyond the fact that python2 is going away in the future | 21:18 |
| clarkb | #status log Pypi mirroring with bandersnatch is now running with Bandersnatch 2.2.0 under python3. This allows us to blacklist packages if necessary (which we are doing to exclude very large packages with very frequent updates to reduce disk needs) | 21:21 |
| openstackstatus | clarkb: finished logging | 21:21 |
| corvus | clarkb: thanks! | 21:25 |
| *** apetrich_ has joined #openstack-infra | 21:27 | |
| *** bdodd has joined #openstack-infra | 21:28 | |
| *** hashar has quit IRC | 21:32 | |
| clarkb | and cron run of bandersnatch seems to have done what we expected \o/ | 21:32 |
| *** eernst has quit IRC | 21:32 | |
| *** timburke_ is now known as timburke | 21:33 | |
| * fungi rejoices | 21:33 | |
| corvus | it's nice to see the word "expected" without its usual prefix | 21:34 |
| clarkb | now I can start to context switch my brain space onto other problems. | 21:35 |
| clarkb | ianw: I guess if you want to rip that dib bandaid off now is as good a time as any | 21:35 |
| *** jamesmcarthur has joined #openstack-infra | 21:40 | |
| ianw | clarkb: ok :) | 21:41 |
| *** gfidente|afk has quit IRC | 21:41 | |
| ianw | i'm going to write something to cleanup the rax images today. apart from about 180TB of space, it really didn't help when trying to upload testing images etc | 21:42 |
| *** armaan has quit IRC | 21:42 | |
| *** rh-jelabarre has quit IRC | 21:42 | |
| clarkb | :/ | 21:42 |
| *** jamesmcarthur has quit IRC | 21:42 | |
| *** tobberydberg has quit IRC | 21:43 | |
| *** bobh has quit IRC | 21:44 | |
| *** bobh has joined #openstack-infra | 21:44 | |
| *** eharney has joined #openstack-infra | 21:46 | |
| *** VW_ has joined #openstack-infra | 21:47 | |
| *** apetrich_ has quit IRC | 21:49 | |
| *** tobberydberg has joined #openstack-infra | 21:51 | |
| *** VW has quit IRC | 21:51 | |
| *** boden has quit IRC | 21:51 | |
| *** neiloy has quit IRC | 21:51 | |
| *** VW_ has quit IRC | 21:53 | |
| *** hemna_ has joined #openstack-infra | 21:57 | |
| *** bdodd has quit IRC | 21:57 | |
| *** rfolco is now known as rfolco|off | 21:59 | |
| *** dizquierdo has quit IRC | 22:00 | |
| *** yamamoto_ has joined #openstack-infra | 22:04 | |
| *** felipemonteiro_ has joined #openstack-infra | 22:06 | |
| *** yamamoto has quit IRC | 22:08 | |
| *** afred312 has joined #openstack-infra | 22:09 | |
| *** felipemonteiro__ has quit IRC | 22:10 | |
| *** aeng has quit IRC | 22:14 | |
| *** aeng has joined #openstack-infra | 22:14 | |
| *** rlandy has quit IRC | 22:18 | |
| *** afred312 has quit IRC | 22:19 | |
| *** esberglu has quit IRC | 22:22 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 22:25 |
| hongbin_ | hi infra team, a question, how to run tempest jobs against stable branch? i tried to checkout the stable branch in here: https://review.openstack.org/#/c/562364/ but it looks the job still checked out the master branch | 22:26 |
| *** claudiub|2 has quit IRC | 22:29 | |
| *** felipemonteiro_ has quit IRC | 22:29 | |
| *** felipemonteiro_ has joined #openstack-infra | 22:29 | |
| *** rcernin has joined #openstack-infra | 22:31 | |
| *** yolanda has quit IRC | 22:32 | |
| clarkb | hongbin_: http://logs.openstack.org/64/562364/2/check/neutron-tempest-plugin-api-queens/d18cec2/zuul-info/inventory.yaml seems to show it checking out queens | 22:33 |
| clarkb | where do you see it doing the wrong thing? | 22:33 |
| hongbin_ | clarkb: http://logs.openstack.org/64/562364/2/check/neutron-tempest-plugin-api-queens/d18cec2/logs/devstacklog.txt.gz#_2018-04-18_19_30_29_016 | 22:34 |
| hongbin_ | i knew it because the neutron stable branch is currently broken but the job succeeded, there must be something wrong | 22:35 |
| clarkb | hongbin_: looks like zuul did the right thing but that is a legacy devstack gate job so you have to tell it to use the stable branch too | 22:36 |
| corvus | it would be great if folks didn't remove the word "legacy-" from jobs which are legacy. | 22:36 |
| hongbin_ | clarkb: i see, how to tell the legacy job to do it, you have an example? | 22:37 |
| corvus | hongbin_: is anyone working on porting that to the v3 native tempest job? | 22:38 |
| clarkb | hongbin_: I'd look at tempests stable branch jobs | 22:38 |
| hongbin_ | corvus: i remembered there is a bug, but not sure if anyone is working on it | 22:38 |
| hongbin_ | clarkb: cool, i will look at it, thanks | 22:39 |
| openstackgerrit | Merged openstack/diskimage-builder master: Revert "debootstrap: Call update-initramfs explicitly" https://review.openstack.org/562004 | 22:40 |
| hongbin_ | corvus: https://bugs.launchpad.net/neutron/+bug/1747510 | 22:40 |
| openstack | Launchpad bug 1747510 in neutron "Migrate legacy jobs to zuul v3 syntax" [Medium,Confirmed] | 22:40 |
| hongbin_ | corvus: noboy is taking it so far | 22:40 |
| corvus | clarkb, hongbin_: i believe tempest has migrated to v3 native | 22:40 |
| *** edmondsw has quit IRC | 22:41 | |
| *** jamesmcarthur has joined #openstack-infra | 22:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 22:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 22:47 |
| *** bobh has quit IRC | 22:47 | |
| notmyname | is there any way to bypass " commit subject >65 characters; use shorter first paragraph" | 22:47 |
| *** bobh has joined #openstack-infra | 22:47 | |
| notmyname | oh! wait. I misread that | 22:48 |
| corvus | notmyname: i thought that was just a warning? or is it 50 chars that's a warning... | 22:48 |
| notmyname | yeah. it was rejected for a different reason. investigating... | 22:48 |
| *** bobh has quit IRC | 22:55 | |
| ianw | pabelanger / clarkb : dib 2.14.0 with pip and initrd fixes pushed | 22:57 |
| ianw | "fixes" :) | 22:58 |
| notmyname | corvus: ah, so here's the situation. we've got a feature branch that we're ready to propose to master. normally, we create a whole new feature branch for review and work on it there. in this case, we're happy with the current feature branch we've been working on and want to propose that to master as a merge commit. however, gerrit is rejecting the merge commit because it includes already-land | 23:00 |
| notmyname | ed commit ids (the stuff from the feature branch) | 23:00 |
| notmyname | is it possible to do what I want? or do I need to find a way to make a clean merge branch? | 23:02 |
| fungi | i thought if the tip being proposed was a merge commit then gerrit was supposed to be cool with any parent changes being commits from another known branch | 23:02 |
| *** tosky has quit IRC | 23:02 | |
| fungi | there was a behavior change in 2.13 which (finally, after many years of attempting to convince them of the problem) stopped allowing you to push a normal change which had parents from another branch | 23:04 |
| fungi | but that check was supposed to be skipped when proposing merge commits | 23:04 |
| *** tpsilva has quit IRC | 23:06 | |
| corvus | notmyname: can you paste the error? | 23:06 |
| fungi | in theory if all the changes which are parents of the merge commit are already merged and you're proposing a merge of branch A to the tip of branch B it should just work, i thought | 23:08 |
| *** slaweq has joined #openstack-infra | 23:09 | |
| notmyname | corvus: yeah | 23:09 |
| notmyname | https://gist.github.com/notmyname/ef56f101f4d6d302b299e1957980fd16 | 23:09 |
| *** jamesmcarthur has quit IRC | 23:12 | |
| corvus | notmyname: any chance that was proposed to feature/s3api because of the .gitreview contents? maybe you need to do 'git review -R master' ? | 23:13 |
| corvus | fungi: ^ ? | 23:13 |
| *** slaweq has quit IRC | 23:13 | |
| notmyname | yesh, that's the one that it's blockingon | 23:13 |
| clarkb | ya I read that error as you pushed to the branch that already merged this | 23:14 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 23:14 |
| corvus | notmyname: true, but that's a coincidence -- what i'm getting at is that because of that change, .gitreview tells git-review to propose the change to s3api instead of master. | 23:14 |
| notmyname | that's a good point | 23:14 |
| clarkb | the refs/publish/feature ath there being the key indicator | 23:14 |
| notmyname | I need to change the contents anyway | 23:14 |
| corvus | notmyname: yeah, you don't want to change the value of that in master; so maybe one more change to put .gitreview back to master then merge. merge commits can have contents, so you can actually make the merge commit do that. | 23:15 |
| corvus | either way | 23:15 |
| notmyname | yep | 23:15 |
| notmyname | yep. that was it | 23:16 |
| corvus | \o/ | 23:16 |
| notmyname | oh | 23:16 |
| notmyname | no | 23:16 |
| notmyname | see -swift | 23:16 |
| *** jamesmcarthur has joined #openstack-infra | 23:17 | |
| notmyname | any way to bulk abandon a bunch of patches? | 23:17 |
| *** hongbin_ has quit IRC | 23:18 | |
| corvus | notmyname: have any gertty users? | 23:18 |
| *** andreaf has quit IRC | 23:18 | |
| *** andreaf has joined #openstack-infra | 23:18 | |
| notmyname | I installed it once | 23:18 |
| notmyname | https://review.openstack.org/#/q/status:open+project:openstack/swift+branch:master+topic:s3api_merge | 23:19 |
| clarkb | I dont recall this being a problem with the zuulv3 merge | 23:20 |
| clarkb | would be interesting to sort out why that happened | 23:21 |
| corvus | notmyname: i'm not subscribed to swift, so it'll probably be faster for you to abandon them piecemeal. | 23:21 |
| fungi | gertty can use a process mark to flag a bunch of changes based on a query and then mass abandon them | 23:21 |
| fungi | but other than that or the api, no idea | 23:21 |
| notmyname | I did | 23:22 |
| notmyname | I abandoned them all in the web gui | 23:23 |
| clarkb | first parent is master second parent is not feature/s3api | 23:23 |
| clarkb | I think that is why that happened | 23:23 |
| notmyname | and new change-id created | 23:23 |
| clarkb | you might reconsider that merge commit too based on that | 23:23 |
| notmyname | oh, hmm | 23:23 |
| notmyname | interesting | 23:24 |
| corvus | yeah, based on what was said in -swift, it looks like there was a local rebase of the whole branch, so entirely new commits all around there | 23:24 |
| notmyname | yep | 23:24 |
| corvus | at least that explains why | 23:24 |
| notmyname | which I could undo. just means redoign the tedious work of the conflict resolution | 23:24 |
| clarkb | I wouldnt be surprised if gerrit refuses to merge the merge commit unless the other changes merge first | 23:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Unpause arm64 builds https://review.openstack.org/562426 | 23:26 |
| *** jamesmcarthur has quit IRC | 23:26 | |
| ianw | clarkb / pabelanger: ^ i propose we start here first, make sure we have networking on arm64 before rolling out fully | 23:26 |
| notmyname | sounds like I need to start over, then. thanks for your help | 23:26 |
| corvus | maybe there's a more git-like way to do this, but i'd probably copy the contents of the repo to some place, then reset to s3api tip, then create a master->s3api merge commit, then copy the contents back into place and mark it resolved. then make a s3->master merge. | 23:26 |
| corvus | notmyname: ^ | 23:26 |
| corvus | notmyname: i think that will save you the tedious work of redoing the resolution (assuming master now is basically the same as when you did the rebase) | 23:27 |
| corvus | or, well, i guess you don't need the master->s3api merge... | 23:28 |
| clarkb | ianw: +2 | 23:28 |
| corvus | i guess you could simplify that to just make a new s3api->master merge commit, and copy the contents of the current commit onto that as the resolution | 23:28 |
| *** eernst has joined #openstack-infra | 23:28 | |
| pabelanger | ianw: +3 | 23:32 |
| *** mriedem has quit IRC | 23:32 | |
| pabelanger | Hmm | 23:34 |
| pabelanger | ianw: both nb01 / nb02 have hung processes for diskimage-builder | 23:34 |
| pabelanger | nodepool 27018 0.0 0.0 18880 3736 ? S 08:42 0:00 git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/qinling.git/ | 23:34 |
| pabelanger | I guess our GIT_HTTP_LIMIT settings didn't work as expected | 23:35 |
| pabelanger | wonder if we need to set them up within an element | 23:35 |
| *** caphrim007_ has joined #openstack-infra | 23:35 | |
| *** felipemonteiro_ has quit IRC | 23:35 | |
| notmyname | corvus: clarkb: much better! https://review.openstack.org/#/c/562428/ | 23:35 |
| ianw | pabelanger: if you haven't killed it, we should be able to examine the environment from proc and see if it got it? | 23:35 |
| pabelanger | ianw: I was too fast on the trigger, I should have kept one online | 23:36 |
| pabelanger | ianw: it happens pretty often, as a results networking | 23:37 |
| pabelanger | I'll be sure to capture the environment for the next time | 23:37 |
| ianw | pabelanger: ok, has it left mounts behind too? | 23:37 |
| *** jamesmcarthur has joined #openstack-infra | 23:38 | |
| pabelanger | ianw: I believe so, yes | 23:38 |
| *** caphrim007 has quit IRC | 23:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Unpause arm64 builds https://review.openstack.org/562426 | 23:39 |
| pabelanger | ianw: I also see a lot of profiledir.OXvlmf in the dib_tmp folder, something else we are leaking? | 23:40 |
| *** HeOS has quit IRC | 23:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 23:42 |
| ianw | pabelanger: hrm, i feel like i've looked into that before | 23:42 |
| *** jamesmcarthur has quit IRC | 23:42 | |
| pabelanger | ianw: clarkb: https://review.openstack.org/562349/ should help fix debian images. and https://review.openstack.org/562348/ removes debian-security from reprepro. I'll see about working on a new mirror.debian-security setup for tomoorow | 23:43 |
| pabelanger | tomorrow* | 23:43 |
| *** eharney has quit IRC | 23:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 23:52 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!