| *** betherly has joined #openstack-infra | 00:01 | |
| clarkb | ianw: reading that maybe what we really want to test here is that each of the individual rules we care about are in the policy? | 00:04 |
|---|---|---|
| clarkb | ianw: we can likely ignore the -P lines, check that -N openstack-INPUT is present and that -A for ssh, snmp, and the ending reject rule are present? | 00:05 |
| clarkb | oh and icmp is important too | 00:05 |
| clarkb | order technically matters but I think we can fudge that for testing this | 00:06 |
| clarkb | then basically ignore the docker ruleset since it doesn't appear to interfere with us | 00:06 |
| *** betherly has quit IRC | 00:06 | |
| clarkb | oh there is a "conflict" | 00:07 |
| clarkb | we set -P FORWARD ACCEPT on our default policy? but docker sets it to DROP? | 00:07 |
| clarkb | we might actually want to reconcile that | 00:08 |
| clarkb | (I think we can safely set -P FORWARD DROP since the onyl place it matter is our test nodes and those are dib configured | 00:08 |
| fungi | that's for routing, correct? | 00:08 |
| clarkb | fungi: ya | 00:08 |
| clarkb | fungi: so it would matter for infracloud or test nodes running openstack | 00:08 |
| clarkb | fungi: but I don't think we route in control plane anywhere | 00:08 |
| fungi | agreed, none of our control plane servers should be acting as routers anyway | 00:08 |
| fungi | i expect that was there because we were originally using the same base ruleset for test nodes running devstack | 00:10 |
| clarkb | ianw: fungi the other thing we should test is if bouncing the network/iptables-persist services results in removal of the docker rules | 00:10 |
| clarkb | we can sort that out later though since this whole thing is a learning experience | 00:10 |
| clarkb | fungi: ya | 00:10 |
| fungi | oh, yep, iptables-persistent may clear any rules not in /etc/firewall/rules | 00:11 |
| clarkb | docker uses its own chains but we dump the whole set with iptables-persistent | 00:11 |
| clarkb | internet seems to agree this is an issue and the fix is to have iptables-persistent manage the docker rules for it | 00:12 |
| clarkb | https://blog.daknob.net/debian-firewall-docker/ | 00:12 |
| *** rkukura has quit IRC | 00:13 | |
| clarkb | ugh this could get complicated very quickly | 00:14 |
| clarkb | I wonder if restarting the docker daemon will update iptables | 00:17 |
| clarkb | a better fix here might be to have those two services depend on each other in such a way that restarting one restarts the other | 00:18 |
| clarkb | I guess its unidirectional. If we restart iptables-persistent then restart docker | 00:18 |
| clarkb | and existing connections should keep running through that since we are nice to established connections | 00:18 |
| clarkb | (except maybe nat will have a sad?) | 00:18 |
| *** betherly has joined #openstack-infra | 00:21 | |
| *** rh-jelabarre has quit IRC | 00:22 | |
| fungi | i doubt it will clear the conntrack tables so shouldn't? | 00:22 |
| clarkb | fungi: actually reading more the sad is that restarting docker daemon restarts your containers | 00:23 |
| clarkb | because bonghits | 00:23 |
| fungi | oh :/ | 00:24 |
| clarkb | but restarting docker after restarting iptables would address it | 00:24 |
| *** betherly has quit IRC | 00:25 | |
| clarkb | https://unrouted.io/2017/08/15/docker-firewall/ is another approach via the DOCKER-USER chain and explicit flushes (that don't flush existing docker rules) | 00:26 |
| clarkb | reality is that we update firwealls so rarely that restarting docker after updating iptables is probably fine for us | 00:26 |
| clarkb | and yet another option is to put docker in its own network namespace (lol) | 00:27 |
| clarkb | then you punch holes from real interface namespace to docker namespace | 00:27 |
| clarkb | I wonder how neutron addresses this because it presumeably has the same problem | 00:27 |
| fungi | DOCKER-USER looks like the easiest solution, but it's unfortunately a break from how we've been firewalling our servers in the past | 00:31 |
| fungi | and also i expect we'll have to revisit yet again when the inevitable userland transition from iptables to nftables finally happens | 00:34 |
| *** betherly has joined #openstack-infra | 00:34 | |
| clarkb | reading this I want pf | 00:36 |
| fungi | tell me about it. i was helping my brother troubleshoot a rather complex pf ruleset earlier today, and it was a breeze | 00:36 |
| fungi | it had zone-specific inclusions, tables loaded from additional files, interface and host aliases out the wazoo | 00:37 |
| clarkb | its been a while since I had to read a ruleset but I seem to recall that pf rules always applied on packets going in a single direction | 00:39 |
| clarkb | much easier to reason about because you didn't have different types of chains, you just evaluate at a given interface and direction | 00:40 |
| fungi | they certainly can if you write them relative to particular interfaces (or interface groups) | 00:40 |
| *** betherly has quit IRC | 00:40 | |
| fungi | but you can also define rules generally directionless and have them evaluated on ingress to all interfaces | 00:40 |
| fungi | coupled with state tracking and urpf it ends up being much more efficient | 00:41 |
| clarkb | pass on tl0 from $web_serv_int to any binat-to $web_serv_ext | 00:42 |
| fungi | then you let your routing tables/interface configs worry about which networks live where and let the firewall apply sensible traffic policy | 00:42 |
| clarkb | compared to iptables -t nat -A POSTROUTING -s ${IPADDR_OUTSIDE} -j MASQUERADE | 00:42 |
| clarkb | these days I let pfsense build my rules for me because elazy but once upon a time I had the dual proc pentium 3 router running pf on openbsd | 00:43 |
| fungi | also modern openbsd has automagic interface group names like "egress" which applies to any interface through which a default route lies | 00:43 |
| clarkb | fwiw I think we should probably punt on this particular concern for now. We can restart docker (and containers) if iptables rules change | 00:43 |
| clarkb | the other thing to keep in mind is that any service we don't actually want to expose may end up being fun to fireawll (think elasticsearch) | 00:44 |
| *** longkb has joined #openstack-infra | 00:47 | |
| *** longkb has quit IRC | 00:48 | |
| fungi | for a more modern example, here's the pf.conf on my home firewall (only uses two physical interfaces and doesn't have many services behind it, performs overload nat outbound for ipv4 and allows ipv6 out without translation): http://paste.openstack.org/show/735924/ | 00:48 |
| clarkb | we'd need to use DOCKER-USER chain to filter access to elasticsearch nodes I think | 00:49 |
| *** mordred has quit IRC | 00:49 | |
| fungi | inbound ipv4 port redirection for ssh/mosh to my workstation (.64) and access to my jukebox icecast stream (.65) | 00:51 |
| fungi | oh, and a udp port range on the jukebox because i occasionally ran a bittorrent client there | 00:51 |
| clarkb | suddenly though I'm reminded why people cling to their centralized firwalls | 00:53 |
| clarkb | tldr docker makes this hard on your network leaves so punt to the routers | 00:53 |
| clarkb | mordred[m]: ianw ^ its probably worth an update to our spec to outline whatever firewall design choices we've made once we've made them | 01:01 |
| clarkb | mordred[m]: ianw in particular I think what will happen is any service running in a docker container with exposed ports will be available on the internet because the forwading chain will bypass any input rules we already have | 01:02 |
| clarkb | for most of our services this is fine since we expose them to the internet, but for others like gearman and elasticsearch we don't and those should be carefully managed | 01:02 |
| *** tosky has quit IRC | 01:07 | |
| *** jamesmcarthur has quit IRC | 01:12 | |
| *** jamesdenton has quit IRC | 01:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var https://review.openstack.org/619119 | 01:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support https://review.openstack.org/619120 | 01:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix unit tests for elements https://review.openstack.org/619387 | 01:13 |
| clarkb | https://github.com/containernetworking/plugins/pull/75#issuecomment-398332210 remains an unsolved problem on podman too (in case anyone is wondering if that will solve the problem) | 01:13 |
| clarkb | apparently our existing rules would work if k8sing ebcause they expose the services directly and not via nat? | 01:14 |
| clarkb | I think the ideal we want here is to have DOCKER-USER explicitly allow traffic we want and drop anything else (so that we don't accidentally expose an elasticsaerch). Then we can add that to our existing rules and restart docker for easy mode. Or update iptable rule setting to only drop and reset the chains we control and not affect the other docker* chains | 01:16 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: run-test-command: support list in test_command variable https://review.openstack.org/610888 | 01:23 |
| imacdonn | Need validation on an assumption ... I have a change which spans glance and glance_store ... if I put the gerrit URL of the glance_store change as a Depends-On in the commit message for the glance change, it should pull in both, right ? | 01:30 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix unit tests for elements https://review.openstack.org/619387 | 01:31 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var https://review.openstack.org/619119 | 01:31 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support https://review.openstack.org/619120 | 01:31 |
| ianw | imacdonn: yes ... as long as the test is configured to install glance_store from the zuul checkout | 01:32 |
| ianw | which it most likely is | 01:32 |
| imacdonn | hmm | 01:32 |
| imacdonn | where would that be configured? | 01:32 |
| *** rkukura has joined #openstack-infra | 01:33 | |
| ianw | not so much configured, but the test is written to do that. tox based tests, devstack based test should pretty much all get this right out of the box | 01:33 |
| imacdonn | the test (module) just goes "import glance_store" | 01:34 |
| ianw | imacdonn: right, what particular test are you talking about? it's where glance_store gets installed | 01:34 |
| *** rkukura has quit IRC | 01:35 | |
| imacdonn | openstack-tox-py27 and openstack-tox-py35 both failed, but, oddly-enough, penstack-tox-py36 didn't | 01:35 |
| ianw | right, for those, you want to make sure glance_store is a required-project in your zuul config. the tox runners in zuul-jobs will then install things correctly | 01:36 |
| ianw | in particular, see tox_install_siblings in http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/tox/README.rst | 01:37 |
| imacdonn | I don't see any mention of "openstack-tox-py" in the glance repo | 01:39 |
| *** rcernin has quit IRC | 01:39 | |
| *** rcernin has joined #openstack-infra | 01:39 | |
| imacdonn | nor "py36" for that matter ... I must be looking in the wrong place | 01:41 |
| ianw | those jobs come in via the templates | 01:42 |
| ianw | imacdonn: i think what you're probably going to have to do is add glance-store to the jobs | 01:44 |
| imacdonn | do you have an example handy? ... of a project that does this properly | 01:44 |
| ianw | imacdonn: you can see a few places where it's done like http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n5374 | 01:45 |
| ianw | you'll have to add entries like | 01:45 |
| ianw | openstack-tox-py27: | 01:46 |
| ianw | required-projects: | 01:46 |
| ianw | - openstack/glance-store | 01:46 |
| ianw | this is probably right, as i see glance-store is in the requirements.txt for glance, so it's a hard dependency | 01:46 |
| ianw | ... but then again ... | 01:47 |
| ianw | you're going to create a glance master that requires glance-store master to work | 01:47 |
| imacdonn | right | 01:48 |
| ianw | ok, just pointing out that currently it tests against the pip version, where if you add it to required-projects, the tox role is going to install it from a git checkout | 01:52 |
| imacdonn | right .. hmm ... I guess I need to discuss this with the glance team | 01:58 |
| *** annp has quit IRC | 02:04 | |
| *** mrsoul has quit IRC | 02:31 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:46 | |
| *** jamesmcarthur has joined #openstack-infra | 03:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix unit tests for elements https://review.openstack.org/619387 | 03:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var https://review.openstack.org/619119 | 03:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support https://review.openstack.org/619120 | 03:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Caputure ramdisk test run logs https://review.openstack.org/619400 | 03:14 |
| *** jamesmcarthur has quit IRC | 03:17 | |
| *** psachin has joined #openstack-infra | 03:19 | |
| *** fuentess has quit IRC | 03:20 | |
| *** BOKALDO has joined #openstack-infra | 03:25 | |
| *** bhavikdbavishi has quit IRC | 03:33 | |
| *** eernst has joined #openstack-infra | 03:33 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:38 | |
| *** roman_g has quit IRC | 03:38 | |
| *** eernst has quit IRC | 03:38 | |
| *** ykarel|away has joined #openstack-infra | 03:49 | |
| *** ykarel|away is now known as ykarel | 03:56 | |
| openstackgerrit | Rui Chen proposed openstack-infra/system-config master: Add #askopenlab into meetbot channels https://review.openstack.org/619432 | 03:57 |
| *** BOKALDO has left #openstack-infra | 04:12 | |
| *** udesale has joined #openstack-infra | 04:15 | |
| *** rlandy|bbl is now known as rlandy | 04:21 | |
| *** janki has joined #openstack-infra | 04:32 | |
| *** yamamoto has quit IRC | 04:49 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add Fedora 29 testing https://review.openstack.org/618671 | 04:58 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean master: Add NetworkManager distro plugin support https://review.openstack.org/618964 | 04:59 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove pypi symlink https://review.openstack.org/619435 | 05:15 |
| ianw | infra-root: ^ if we have a sec, this will make sure the volume is unreferenced | 05:16 |
| *** armax has quit IRC | 05:17 | |
| *** ykarel has quit IRC | 05:24 | |
| *** yamamoto has joined #openstack-infra | 05:29 | |
| *** chandankumar has joined #openstack-infra | 05:33 | |
| *** chandankumar is now known as chkumar|ruck | 05:33 | |
| *** yamamoto has quit IRC | 05:39 | |
| *** ykarel has joined #openstack-infra | 05:40 | |
| *** rkukura has joined #openstack-infra | 05:40 | |
| AJaeger | gmann: could you review https://review.openstack.org/617662, please? That adds a new grenade py35 job to integrated gate | 06:01 |
| openstackgerrit | Merged openstack-infra/project-config master: Add operations-docs to storyboard https://review.openstack.org/618722 | 06:05 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Only setup zNode caches in launcher https://review.openstack.org/619440 | 06:07 |
| *** bhavikdbavishi has quit IRC | 06:07 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix print-zk tool for python3 https://review.openstack.org/619441 | 06:10 |
| *** yamamoto has joined #openstack-infra | 06:14 | |
| *** noama has joined #openstack-infra | 06:16 | |
| gmann | AJaeger: +1 on running it on integrated gate but one question inline for setting the branch. | 06:19 |
| *** ifat_afek has joined #openstack-infra | 06:20 | |
| *** ahosam has joined #openstack-infra | 06:31 | |
| *** jamesmcarthur has joined #openstack-infra | 06:34 | |
| openstackgerrit | Rui Chen proposed openstack-infra/irc-meetings master: Add OpenLab team meeting https://review.openstack.org/619446 | 06:39 |
| *** jamesmcarthur has quit IRC | 06:39 | |
| AJaeger | thanks, gmann - agree with your comment | 06:42 |
| *** bhavikdbavishi has joined #openstack-infra | 06:46 | |
| *** bhavikdbavishi has quit IRC | 06:51 | |
| *** ahosam has quit IRC | 06:57 | |
| openstackgerrit | Kartikeya Jain proposed openstack/diskimage-builder master: Adding support for SLES 15 in element 'sles' https://review.openstack.org/619186 | 06:57 |
| *** slaweq has joined #openstack-infra | 06:59 | |
| *** quiquell|off is now known as quiquell | 07:01 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider https://review.openstack.org/570667 | 07:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift Pod provider https://review.openstack.org/590335 | 07:02 |
| openstackgerrit | Rui Chen proposed openstack-infra/project-config master: Add OpenLab channel #askopenlab into accessbot list https://review.openstack.org/619450 | 07:05 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add Fedora 29 testing https://review.openstack.org/618671 | 07:08 |
| openstackgerrit | Rui Chen proposed openstack-infra/irc-meetings master: Add OpenLab team meeting https://review.openstack.org/619446 | 07:08 |
| *** aojea has joined #openstack-infra | 07:16 | |
| *** pcaruana has joined #openstack-infra | 07:22 | |
| *** ramishra has joined #openstack-infra | 07:30 | |
| *** jaosorior has joined #openstack-infra | 07:41 | |
| *** ykarel is now known as ykarel|lunch | 07:44 | |
| *** rpittau has joined #openstack-infra | 07:44 | |
| *** eumel8 has joined #openstack-infra | 07:51 | |
| *** kjackal has joined #openstack-infra | 07:56 | |
| *** xarses_ has quit IRC | 07:56 | |
| *** xarses_ has joined #openstack-infra | 07:57 | |
| *** ginopc has joined #openstack-infra | 08:11 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:12 | |
| *** rcernin has quit IRC | 08:13 | |
| *** florianf has joined #openstack-infra | 08:15 | |
| *** jbadiapa has quit IRC | 08:16 | |
| *** ralonsoh has joined #openstack-infra | 08:18 | |
| *** shardy has joined #openstack-infra | 08:26 | |
| *** jbadiapa has joined #openstack-infra | 08:32 | |
| *** ykarel|lunch is now known as ykarel | 08:36 | |
| *** jpena|off is now known as jpena | 08:37 | |
| *** jbadiapa has quit IRC | 08:37 | |
| *** imacdonn has quit IRC | 08:38 | |
| *** xek has joined #openstack-infra | 08:42 | |
| *** tosky has joined #openstack-infra | 08:49 | |
| *** slaweq has quit IRC | 08:56 | |
| *** jpich has joined #openstack-infra | 09:01 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Capture ramdisk test run logs https://review.openstack.org/619400 | 09:09 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix unit tests for elements https://review.openstack.org/619387 | 09:09 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var https://review.openstack.org/619119 | 09:09 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support https://review.openstack.org/619120 | 09:09 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean master: Add NetworkManager distro plugin support https://review.openstack.org/618964 | 09:13 |
| *** e0ne has joined #openstack-infra | 09:17 | |
| *** jbadiapa has joined #openstack-infra | 09:19 | |
| *** mrhillsman is now known as openlab | 09:19 | |
| *** openlab is now known as mrhillsman | 09:20 | |
| *** yboaron_ has joined #openstack-infra | 09:28 | |
| *** ifat_afek has quit IRC | 09:31 | |
| *** gfidente has joined #openstack-infra | 09:31 | |
| *** derekh has joined #openstack-infra | 09:34 | |
| *** Emine has joined #openstack-infra | 09:41 | |
| *** electrofelix has joined #openstack-infra | 09:52 | |
| *** dtantsur|afk is now known as dtantsur | 09:54 | |
| *** zigo has joined #openstack-infra | 10:03 | |
| *** apetrich has joined #openstack-infra | 10:06 | |
| openstackgerrit | Hemanth Nakkina proposed openstack-infra/project-config master: Create airship-spyglass repo https://review.openstack.org/619493 | 10:07 |
| *** bhavikdbavishi has quit IRC | 10:10 | |
| *** yamamoto has quit IRC | 10:14 | |
| *** ifat_afek has joined #openstack-infra | 10:15 | |
| *** ianychoi has quit IRC | 10:25 | |
| *** dpawlik has quit IRC | 10:37 | |
| *** dpawlik_ has joined #openstack-infra | 10:37 | |
| *** jpena is now known as jpena|off | 10:37 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:41 | |
| *** xarses_ has quit IRC | 10:43 | |
| *** ramishra has quit IRC | 10:43 | |
| *** xarses_ has joined #openstack-infra | 10:43 | |
| *** toabctl has joined #openstack-infra | 10:54 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:54 | |
| *** ifat_afek has quit IRC | 10:55 | |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** bhavikdbavishi has quit IRC | 11:01 | |
| *** yamamoto has quit IRC | 11:05 | |
| *** hemanth_n has joined #openstack-infra | 11:07 | |
| *** ccamacho has quit IRC | 11:08 | |
| *** ccamacho has joined #openstack-infra | 11:10 | |
| *** yamamoto has joined #openstack-infra | 11:11 | |
| hemanth_n | AJeager: Regarding https://review.openstack.org/#/c/619493 the code will be ready by next monday... once the project is created i would like to upload the codebase | 11:12 |
| hemanth_n | Ajaeger: Regarding https://review.openstack.org/#/c/619493 the code will be ready by next monday... once the project is created i would like to upload the codebase | 11:13 |
| *** udesale has quit IRC | 11:15 | |
| *** ifat_afek has joined #openstack-infra | 11:19 | |
| *** ianychoi has joined #openstack-infra | 11:31 | |
| AJaeger | hemanth_n: commented on review - I think you better WIP the change (workflow-1) and use the upstream field to import code you have... | 11:32 |
| hemanth_n | Ajaeger: Ok thanks.. | 11:33 |
| hemanth_n | AJaeger: One query on upstream field.. is it possible to just pull one branch instead of all branches | 11:35 |
| AJaeger | no, we import whole repo. But you can dleete branches... Clone the repo to another place, delete what you don't need... | 11:35 |
| hemanth_n | AJaeger: Okie.. thanks | 11:36 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/nodepool master: Add ip-pool option to the openstack provider https://review.openstack.org/619525 | 11:38 |
| *** janki has quit IRC | 11:43 | |
| *** dims has quit IRC | 11:45 | |
| *** e0ne has quit IRC | 11:46 | |
| *** yboaron_ has quit IRC | 11:50 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: Update change URL for Gerrit v2.16 https://review.openstack.org/619533 | 11:55 |
| *** ralonsoh has quit IRC | 11:58 | |
| *** hemanth_n has quit IRC | 11:58 | |
| *** e0ne has joined #openstack-infra | 11:59 | |
| *** ralonsoh has joined #openstack-infra | 11:59 | |
| *** yamamoto has quit IRC | 11:59 | |
| *** yamamoto has joined #openstack-infra | 12:00 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/openstack-zuul-jobs master: Correct MariaDB proxy suffix https://review.openstack.org/619537 | 12:04 |
| odyssey4me | mnaser ^ | 12:04 |
| *** kjackal has quit IRC | 12:07 | |
| mnaser | thanks odyssey4me | 12:07 |
| openstackgerrit | Jakub Sliva proposed openstack-infra/project-config master: Create repository for ceilometer-dashboard plugin https://review.openstack.org/619235 | 12:13 |
| *** ginopc has quit IRC | 12:15 | |
| *** ramishra has joined #openstack-infra | 12:17 | |
| *** apetrich has quit IRC | 12:18 | |
| *** kjackal has joined #openstack-infra | 12:18 | |
| *** bobh has joined #openstack-infra | 12:20 | |
| *** ginopc has joined #openstack-infra | 12:29 | |
| *** yamamoto has quit IRC | 12:30 | |
| *** kjackal has quit IRC | 12:31 | |
| *** kjackal has joined #openstack-infra | 12:31 | |
| *** apetrich has joined #openstack-infra | 12:38 | |
| frickler | infra-root: review.o.o is pretty slow for me currently. there is also a peak of traffic at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=45&rra_id=all | 12:39 |
| * prometheanfire just realized he needed to restart neomutt to see the new folder for the new list... | 12:42 | |
| AJaeger | oh, "gerrit show-queue" shows a stuck queue since 2nd of November for openstack/networking-odl - anything we need to care about? | 12:42 |
| *** witek has joined #openstack-infra | 12:51 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/elastic-recheck master: Add query for ssh timeout in tempest scenario for bug 1802971 https://review.openstack.org/617579 | 12:52 |
| openstack | bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] https://launchpad.net/bugs/1802971 | 12:52 |
| *** yamamoto has joined #openstack-infra | 13:01 | |
| mnaser | is it me or gerrit seems pretty slow | 13:05 |
| kaiokmo | mnaser: same | 13:05 |
| *** yamamoto has quit IRC | 13:06 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** yamamoto has quit IRC | 13:08 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: Made elastic-recheck py3 compatible https://review.openstack.org/616578 | 13:15 |
| *** bobh has quit IRC | 13:25 | |
| *** yamamoto has joined #openstack-infra | 13:26 | |
| frickler | infra-root: gerrit still seems to suffer from memory shortage, started swapping at 10:30 when also the load peaked http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=29&rra_id=all | 13:28 |
| frickler | I'm not sure whether I should leave it that way in the hope that it will either recover or someone will find time to debug further. the other option would be a restart which may or may not resolve the issue | 13:29 |
| frickler | and I'll be pretty much offline for a long weekend soon myself | 13:30 |
| fungi | i'm not in a position to troubleshoot it yet, but it may recover on its own if the traffic burst doesn't persist | 13:32 |
| fungi | otherwise i probably have to dig into logs and state tables to find abusers and block them with iptables rules | 13:33 |
| frickler | fungi: the traffic seems to have normalized, while memory usage hasn't. the logs didn't show anything obvious for me. there seems to be one bot at work, but I see requests from it also before the issue started | 13:34 |
| fungi | frickler: if it helps, gerrit admins can also view the javamelody monitoring interface for gerrit at https://review.openstack.org/monitoring once authenticated | 13:39 |
| *** slaweq has joined #openstack-infra | 13:41 | |
| *** yamamoto has quit IRC | 13:41 | |
| fungi | sluggishness usually transpires when there is heavily jvm garbage collection activity | 13:42 |
| fungi | (you have to expand "other charts" to see that) | 13:43 |
| fungi | looks like there was a gc burst around 12:15z | 13:43 |
| fungi | but it hasn't continued | 13:43 |
| frickler | fungi: it's not garbage collection, that shows only a single peak an hour ago | 13:43 |
| *** udesale has joined #openstack-infra | 13:44 | |
| fungi | so this isn't the sort of thing we were seeing regularly in the past | 13:44 |
| *** pcaruana has quit IRC | 13:50 | |
| *** fuentess has joined #openstack-infra | 13:52 | |
| *** ifat_afek has quit IRC | 13:58 | |
| *** chkumar|ruck has quit IRC | 14:02 | |
| *** zul has joined #openstack-infra | 14:06 | |
| *** slaweq has quit IRC | 14:07 | |
| *** mnencia has quit IRC | 14:07 | |
| *** pbourke has quit IRC | 14:09 | |
| *** mnencia has joined #openstack-infra | 14:09 | |
| *** pbourke has joined #openstack-infra | 14:11 | |
| *** dims has joined #openstack-infra | 14:14 | |
| *** rfolco has quit IRC | 14:15 | |
| *** rfolco has joined #openstack-infra | 14:16 | |
| *** pcaruana has joined #openstack-infra | 14:25 | |
| *** emine__ has joined #openstack-infra | 14:30 | |
| *** Emine has quit IRC | 14:30 | |
| *** ykarel is now known as ykarel|away | 14:34 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add second level cache of nodes https://review.openstack.org/619025 | 14:37 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add second level cache to node requests https://review.openstack.org/619069 | 14:37 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Only setup zNode caches in launcher https://review.openstack.org/619440 | 14:37 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Asynchronously update node statistics https://review.openstack.org/619589 | 14:37 |
| *** ykarel|away has quit IRC | 14:47 | |
| *** quiquell is now known as quiquell|off | 14:53 | |
| *** yamamoto has joined #openstack-infra | 14:55 | |
| *** psachin has quit IRC | 14:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 14:55 | |
| *** emine__ has quit IRC | 15:02 | |
| *** yamamoto has quit IRC | 15:03 | |
| *** quiquell|off is now known as quiquell | 15:15 | |
| mnaser | is it possible to get a bigger instance for gerrit? | 15:15 |
| dmsimard | depends what the problem is | 15:16 |
| dmsimard | it's probably not very loaded right now, a lot of people are off because of thanksgiving | 15:16 |
| dmsimard | I did get a 502 proxy error earlier and frickler mentioned issues as well | 15:16 |
| *** kjackal has quit IRC | 15:17 | |
| *** kjackal_v2 has joined #openstack-infra | 15:17 | |
| fungi | it's a 60gb ram instance already | 15:18 |
| fungi | normally it doesn't (and shouldn't) need anywhere near that amount of memory | 15:18 |
| dmsimard | java (gerrit) is at ~500% CPU and 86% memory | 15:19 |
| dmsimard | which is probably not normal | 15:19 |
| fungi | the javamelody graph says it's been hovering around 8gb of used memory in the jvm up until around 10:30z today when it started to grow to around 40gb | 15:19 |
| *** jistr is now known as jistr|call | 15:20 | |
| mnaser | <insert joke about thanksgiving and gerrit is also hungry and eating a lot of memory instead of turkey> | 15:20 |
| fungi | so a fivefold growth in memory utilization over the course of ~4 hours | 15:20 |
| fungi | bloated! ;) | 15:20 |
| dmsimard | fungi: looks like there's something going on *right now* according to cacti http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=25&rra_id=0&view_type=tree&graph_start=1542035940&graph_end=1542899940 | 15:20 |
| dmsimard | a very abnormal cpu spike | 15:20 |
| fungi | i'll see what i can spot | 15:20 |
| dmsimard | Seeing a lot of these: [Thu Nov 22 15:21:12.292112 2018] [mpm_event:error] [pid 48415:tid 140553527216000] AH00485: scoreboard is full, not at MaxRequestWorkers | 15:21 |
| *** kjackal_v2 has quit IRC | 15:21 | |
| *** kjackal has joined #openstack-infra | 15:22 | |
| fungi | yeah, that likely indicates we have a lot of additional established connections to apache than usual | 15:22 |
| fungi | er, a lot more than usual | 15:22 |
| fungi | which could also just be a symptom of gerrit being slow to respond for some other reason, clients end up connected longer waiting for responses so they pile up | 15:23 |
| dmsimard | yeah, was thinking the same thing | 15:24 |
| dmsimard | I feel like I remember something about Zuul switching to the http API for gerrit (instead of SSH) -- or was it the other way around ? | 15:26 |
| dmsimard | I ask because there's a lot of ssh login/logout from zuul although it's probably normal/expected if we're still using SSH and my memory is failing me | 15:27 |
| *** jistr|call is now known as jistr | 15:28 | |
| fungi | i think it still needs ssh for the event stream | 15:28 |
| fungi | but uses the rest api for reporting now | 15:29 |
| dmsimard | ah, makes sense | 15:30 |
| fungi | >600 established connections to the gerrit ssh api right now, though i don't know whether that's typical | 15:31 |
| *** ccamacho has quit IRC | 15:31 | |
| *** ccamacho has joined #openstack-infra | 15:32 | |
| fungi | highest offending ip address only has 34 connections. not great, but not necessarily indicative of a runaway system | 15:33 |
| dmsimard | what's the url for melody again? doesn't seem mentioned in https://docs.openstack.org/infra/system-config/gerrit.html | 15:33 |
| fungi | it's /monitoring | 15:34 |
| mnaser | i think it's completely fallen over now | 15:34 |
| fungi | highest offending account for established ssh api connections is 53, so also not near our set max limit | 15:35 |
| dmsimard | fungi: in /monitoring you see a particular request that has been running for a very long time -- should we attempt to kill that ? | 15:36 |
| dmsimard | says it's been running for 312 minutes | 15:36 |
| dmsimard | ~5 hours ago | 15:36 |
| fungi | the /changes/?q=is:watched+is:merged&n=25&O=81 one? | 15:37 |
| scas | chiming in to mention that i'm also seeing a 502 proxy error | 15:37 |
| dmsimard | fungi: there's a few, actually: https://screenshots.firefox.com/wCp17ZUj4NP5JEFg/review.openstack.org | 15:38 |
| scas | something something holidays | 15:38 |
| dmsimard | scas: we're on it :) | 15:38 |
| scas | <3 | 15:38 |
| AJaeger | dmsimard: that are 5 of these - interesting | 15:39 |
| fungi | if you expand the request details there are 5 of those, all long-running | 15:39 |
| fungi | yeah | 15:39 |
| dmsimard | fungi: may I kill them ? | 15:39 |
| AJaeger | +1 | 15:39 |
| dmsimard | it sounds like it's an odd query as well... "is watched and is merged" ? that'd return a LOT of changes | 15:40 |
| fungi | i wish it told you the account as well | 15:40 |
| scas | if someone's filters weren't set up, yeah, it would. i use that query on rare occasion, but i have filters | 15:40 |
| fungi | i'm suspecting the active stackalytics account is running those, but it's not clear | 15:41 |
| dmsimard | we can search the thread numbers in the logs | 15:41 |
| scas | it was not me issuing that, however, as i'm just a bystander | 15:41 |
| dmsimard | for example the first one turns up | 15:41 |
| dmsimard | [2018-11-22 03:14:33,810] [HTTP-4277706] WARN com.google.gerrit.server.extensions.events.EventUtil : Error in listener com.google.gerrit.server.events.StreamEventsApiListener for event com.google.gerrit.server.extensions.events.CommentAdded: null | 15:41 |
| fungi | oh, actually not stackalytics as it still queries via the ssh api | 15:42 |
| fungi | go ahead and kill those long-running rest api queries, though i have doubts it will help it's a fair test | 15:43 |
| dmsimard | eh, getting a 502 proxy error when attempting to kill | 15:44 |
| fungi | fun! | 15:44 |
| *** psachin has joined #openstack-infra | 15:44 | |
| dmsimard | I guess we're looking at a restart ? | 15:44 |
| *** mdbooth has joined #openstack-infra | 15:44 | |
| mdbooth | I'm very conscious that I only ever join here when things are going wrong :) | 15:44 |
| fungi | the kill button warns me "Killing a Java thread is not recommended and unsafe." | 15:45 |
| mdbooth | Or at least I suspect they are. Is gerrit down for you, too? | 15:45 |
| AJaeger | mdbooth: team is on it... | 15:45 |
| mdbooth | AJaeger: Awesome, thanks. | 15:45 |
| fungi | the 502 proxy error is generally just apache giving up after the java web service takes too long to return a response | 15:46 |
| AJaeger | fungi, shall I send an alert out? | 15:46 |
| dmsimard | fungi: this is what searching for the http thread IDs in the error log turns up: http://paste.openstack.org/show/735947/ | 15:46 |
| fungi | i'm managing to kill them | 15:46 |
| AJaeger | What about? #status alert review.openstack.org is currently under high load, we're investigating who works on Thanksgiving... | 15:47 |
| AJaeger | Or notice? | 15:47 |
| dmsimard | fungi: load has already significantly gone down | 15:47 |
| dmsimard | and gerrit is reachable again | 15:48 |
| fungi | i think it started to become reachable on its own before i killed those threads, but it's hard to be certain | 15:48 |
| *** psachin has quit IRC | 15:48 | |
| dmsimard | AJaeger: I think we should be good -- a notice to mention the issue, the recovery and that we're keeping an eye on things is probably appropriate | 15:49 |
| *** quiquell is now known as quiquell|off | 15:49 | |
| fungi | most of those threads get reused for different things over time, so the entries closer to when the issue started (~10:00z today) are likely more relevant | 15:49 |
| AJaeger | So, #status notice review.openstack.org was under heavy note and should recover now; infra team keeps monitoring ? | 15:50 |
| AJaeger | Or just leave it, since we hope it's fine? | 15:50 |
| scas | as a drive-by observer, a quick note making people aware that you're aware goes a long way to making sure when it happens again it won't be a (big) shock | 15:51 |
| fungi | dmsimard: looking at that subset of the error log, i don't think any of those are relevant | 15:51 |
| fungi | none are for queries | 15:52 |
| fungi | they're storyboard plugin and stream events and comment added | 15:52 |
| AJaeger | fungi, dmsimard, memory is still high, isn't it? | 15:52 |
| fungi | in theory it should get garbage-collected | 15:53 |
| AJaeger | question is when... | 15:53 |
| fungi | the gc thread wakes up periodically, or we can trigger it through javamelody on demand | 15:53 |
| dmsimard | fungi: yeah, neither do I -- I looked at the context too and there's no traces or anything.. searching for the actual error did turn up a bug in "its-base" which might be used by storyboard ? | 15:53 |
| dmsimard | https://bugs.chromium.org/p/gerrit/issues/detail?id=7435 | 15:53 |
| * AJaeger needs to step out | 15:56 | |
| fungi | its-base is the issue tracker base plugin, which the its-storyboard plugin is built on | 15:56 |
| *** tobias-urdin has joined #openstack-infra | 15:58 | |
| dmsimard | #status log We have recovered from high cpu usage on review.openstack.org by killing several requests in melody that had been running for several hours and brought gerrit to a crawl with proxy errors. Requests looked like this: "/changes/?q=is:watched+is:merged&n=25&O=81 GET" but we haven't been able to identify where these requests came from. | 15:58 |
| openstackstatus | dmsimard: finished logging | 15:58 |
| fungi | i'm going to check for those query strings in the gerrit ssh and apache logs | 16:00 |
| dmsimard | oh, good idea | 16:00 |
| dmsimard | there's matches in gerrit-ssl-access.log | 16:00 |
| *** mgutehall has quit IRC | 16:02 | |
| fungi | oh, cool, i'll leave you to that. still trying to deal with things in the kitchen here at the same time | 16:02 |
| dmsimard | There's requests from that IP with the gertty user agent but those particular hits came from a chrome agent | 16:02 |
| dmsimard | sure, I'll try and find out | 16:02 |
| *** mgutehall has joined #openstack-infra | 16:03 | |
| dmsimard | fungi: you're off today anyway right? | 16:03 |
| *** aojea has quit IRC | 16:08 | |
| *** jamesmcarthur has joined #openstack-infra | 16:09 | |
| fungi | some of today | 16:09 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Correct MariaDB proxy suffix https://review.openstack.org/619537 | 16:10 |
| fungi | it's a good excuse to get together with friends who have more rigid work schedules and are off for a holiday | 16:10 |
| *** ykarel|away has joined #openstack-infra | 16:11 | |
| fungi | not one of my preferred holidays personally what with the oppressive religious overtones and whitewashing of imperialist genocide | 16:11 |
| dmsimard | ++ | 16:11 |
| dmsimard | I've identified the username who ran the queries in question | 16:12 |
| dmsimard | But I'm not entirely positive on how to match that username to a name/email | 16:12 |
| fungi | you can start a query on the gerrit webui like reviewer:thatusername and see what it tries to autocomplete with | 16:18 |
| fungi | though there are also rest api methods to get user info, and as a last resort the external_id column of the account_external_ids table in the backend db can be queried for a match on "username:thatusername" | 16:19 |
| *** bhavikdbavishi has quit IRC | 16:19 | |
| dmsimard | I was ~75% certain of a match but an IRC whois confirmed it :p | 16:20 |
| *** bhavikdbavishi has joined #openstack-infra | 16:20 | |
| dmsimard | Sent a PM to see if we can figure out what happened | 16:20 |
| fungi | the jvm gc seems to have woken up and started to free up references too | 16:21 |
| fungi | keep in mind that we only have at best anecdotal evidence to suggest those query threads were the problem | 16:21 |
| dmsimard | yup | 16:22 |
| fungi | just because it started to recover roughly when i killed them doesn't mean they were the cause, or in any way related for that matter | 16:22 |
| dmsimard | I /could/ try to reproduce the issue heh | 16:23 |
| *** bhavikdbavishi1 has joined #openstack-infra | 16:23 | |
| fungi | we probably should try that at some point, yes | 16:23 |
| *** bhavikdbavishi has quit IRC | 16:24 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 16:24 | |
| dmsimard | searching for "is:watched is:merged" returns instantly for me, so might be a red herring | 16:28 |
| *** e0ne has quit IRC | 16:28 | |
| *** lbragstad has joined #openstack-infra | 16:28 | |
| *** roman_g has joined #openstack-infra | 16:28 | |
| clarkb | dmsimard: the is watched means watched by you I think | 16:30 |
| clarkb | so different users watch lists could have different costs to query | 16:30 |
| *** ramishra has quit IRC | 16:32 | |
| *** Emine has joined #openstack-infra | 16:35 | |
| *** lbragstad has quit IRC | 16:39 | |
| *** roman_g has quit IRC | 16:41 | |
| *** armax has joined #openstack-infra | 16:41 | |
| *** kjackal has quit IRC | 16:44 | |
| *** kjackal has joined #openstack-infra | 16:44 | |
| *** bhavikdbavishi has quit IRC | 16:48 | |
| *** udesale has quit IRC | 16:48 | |
| *** diablo_rojo has joined #openstack-infra | 16:58 | |
| *** ginopc has quit IRC | 17:01 | |
| *** shardy has quit IRC | 17:05 | |
| *** mdbooth_ has joined #openstack-infra | 17:16 | |
| *** mdbooth has quit IRC | 17:19 | |
| *** jpich has quit IRC | 17:20 | |
| *** e0ne has joined #openstack-infra | 17:25 | |
| *** mdbooth_ has quit IRC | 17:27 | |
| *** Emine has quit IRC | 17:34 | |
| *** dtantsur is now known as dtantsur|afk | 17:35 | |
| *** jamesmcarthur has quit IRC | 17:35 | |
| *** jamesmcarthur has joined #openstack-infra | 17:39 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/openstack-zuul-jobs master: use opensuse15 as generic name instead of opensuse150 https://review.openstack.org/619628 | 17:40 |
| *** rossella_s has quit IRC | 17:40 | |
| tobias-urdin | can you request op in a channel using accessbot? would like to be able to set topic in #puppet-openstack if that's allowed since it displays CI status | 17:42 |
| *** gfidente is now known as gfidente|afk | 17:44 | |
| fungi | we just add people as chanops for specific channels on request. in this case as ptl for a project where that's the preferred irc channel for discussions, i have no problem adding you to the access list there | 17:44 |
| fungi | the global access lists are managed by accessbot more for bootstrapping purposes, so we don't end up with channels where nobody can add you as an op | 17:45 |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Add config parameters for WinRM timeouts https://review.openstack.org/619630 | 17:46 |
| fungi | tobias-urdin: freenode -- ChanServ (ChanServ@services.): Flags +Aefortv were set on tobasco in #puppet-openstack. | 17:46 |
| fungi | tobias-urdin: you should now be able to `/msg chanserv #puppet-openstack topic This is our fancy new topic!` | 17:47 |
| fungi | er, that should be `/msg chanserv topic #puppet-openstack This is our fancy new topic!` | 17:47 |
| *** e0ne has quit IRC | 17:49 | |
| *** kjackal has quit IRC | 17:49 | |
| *** kjackal_v2 has joined #openstack-infra | 17:49 | |
| *** jamesmcarthur has quit IRC | 17:56 | |
| *** jamesmcarthur has joined #openstack-infra | 18:00 | |
| fungi | #status log manually triggered gerrit's jvm garbage collection from the javamelody interface, freeing some 40gb of used memory within the jvm | 18:00 |
| openstackstatus | fungi: finished logging | 18:00 |
| *** derekh has quit IRC | 18:01 | |
| *** jamesdenton has joined #openstack-infra | 18:04 | |
| *** e0ne has joined #openstack-infra | 18:07 | |
| *** jamesmcarthur has quit IRC | 18:08 | |
| *** ykarel|away has quit IRC | 18:09 | |
| AJaeger | wow, 40GB! Thanks, fungi ! | 18:15 |
| *** e0ne has quit IRC | 18:15 | |
| fungi | np | 18:18 |
| * fungi disappears for some hours | 18:18 | |
| *** armax has quit IRC | 18:21 | |
| AJaeger | fungi, I don't see that one yet on http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 18:33 |
| AJaeger | fungi, enjoy thanks giving | 18:33 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix manual dequeue of github items https://review.openstack.org/619272 | 18:39 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Retry queries for commits https://review.openstack.org/575137 | 18:40 |
| *** jamesmcarthur has joined #openstack-infra | 18:45 | |
| *** rpittau has quit IRC | 18:46 | |
| *** lpetrut has joined #openstack-infra | 18:46 | |
| *** electrofelix has quit IRC | 18:53 | |
| *** jamesmcarthur has quit IRC | 18:53 | |
| *** jamesmcarthur has joined #openstack-infra | 18:57 | |
| *** yamamoto has joined #openstack-infra | 19:00 | |
| *** jamesmcarthur has quit IRC | 19:02 | |
| *** armax has joined #openstack-infra | 19:03 | |
| *** yamamoto has quit IRC | 19:05 | |
| *** noama has quit IRC | 19:05 | |
| *** Emine has joined #openstack-infra | 19:08 | |
| *** apetrich has quit IRC | 19:16 | |
| *** auristor has quit IRC | 19:16 | |
| *** apetrich has joined #openstack-infra | 19:17 | |
| dmsimard | AJaeger: I believe the RAM is still allocated to the JVM, it's within the JVM that it has been free'd up | 19:35 |
| dmsimard | "Java memory used: 8,111 Mb / 47,378 Mb " | 19:35 |
| AJaeger | I see - thanks | 19:40 |
| *** Emine has quit IRC | 19:48 | |
| *** diablo_rojo has quit IRC | 19:59 | |
| *** kjackal_v2 has quit IRC | 20:10 | |
| *** kjackal has joined #openstack-infra | 20:11 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Move k8s install to pre playbook https://review.openstack.org/617699 | 20:12 |
| openstackgerrit | Merged openstack-infra/zuul master: Add allowed-triggers and allowed-reporters tenant settings https://review.openstack.org/554082 | 20:14 |
| openstackgerrit | Merged openstack-infra/zuul master: encrypt_secret: support self-signed certificates via --insecure argument https://review.openstack.org/617281 | 20:14 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: upload-logs-swift: Turn FileList into a context manager https://review.openstack.org/592850 | 20:41 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: upload-logs-swift: Keep the FileList in the indexer class https://review.openstack.org/592851 | 20:41 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Add Fedora 29 testing https://review.openstack.org/618671 | 20:55 |
| *** gfidente|afk has quit IRC | 21:04 | |
| *** bobh has joined #openstack-infra | 21:13 | |
| *** yamamoto has joined #openstack-infra | 21:22 | |
| *** lpetrut has quit IRC | 21:23 | |
| *** slaweq has joined #openstack-infra | 21:23 | |
| *** bobh has quit IRC | 21:26 | |
| *** yamamoto has quit IRC | 21:37 | |
| *** kjackal has quit IRC | 21:38 | |
| *** yamamoto has joined #openstack-infra | 21:40 | |
| *** ralonsoh has quit IRC | 21:47 | |
| *** lbragstad has joined #openstack-infra | 21:54 | |
| *** rcernin has joined #openstack-infra | 22:05 | |
| *** xek_ has joined #openstack-infra | 22:06 | |
| *** yamamoto has quit IRC | 22:07 | |
| *** yamamoto has joined #openstack-infra | 22:07 | |
| *** yamamoto has quit IRC | 22:08 | |
| *** xek has quit IRC | 22:09 | |
| *** slaweq has quit IRC | 22:15 | |
| *** auristor has joined #openstack-infra | 22:19 | |
| *** jtomasek has quit IRC | 22:20 | |
| *** xek__ has joined #openstack-infra | 22:25 | |
| *** xek_ has quit IRC | 22:27 | |
| *** xarses_ has quit IRC | 22:42 | |
| *** xarses_ has joined #openstack-infra | 22:42 | |
| *** yamamoto has joined #openstack-infra | 22:46 | |
| *** ahosam has joined #openstack-infra | 22:47 | |
| *** ahosam has quit IRC | 22:57 | |
| *** yamamoto has quit IRC | 22:57 | |
| *** pfallenop has quit IRC | 23:04 | |
| *** pfallenop has joined #openstack-infra | 23:05 | |
| *** yamamoto has joined #openstack-infra | 23:12 | |
| *** lbragstad has quit IRC | 23:23 | |
| *** tosky has quit IRC | 23:31 | |
| *** jamesmcarthur has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!