| openstackgerrit | Merged opendev/system-config master: Install jeepyb into the gerrit images https://review.opendev.org/682601 | 00:07 |
|---|---|---|
| *** factor has quit IRC | 00:09 | |
| *** yamamoto has joined #openstack-infra | 00:10 | |
| *** gyee has quit IRC | 00:15 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add environment switch for centos8 to use dnf https://review.opendev.org/684459 | 00:18 |
| corvus | yay the password is in place; i'll restart the executors now | 00:29 |
| fungi | lmk if you need help with it | 00:31 |
| fungi | this fudge cools at a rate of ~1c/min | 00:31 |
| corvus | fungi: would a fan help? | 00:32 |
| corvus | (do you want fudge to cool slowly or quickly?) | 00:32 |
| fungi | maybe, but doing my best not to speed it up, yeah | 00:32 |
| fungi | slowly | 00:32 |
| corvus | executors cool slowly too | 00:33 |
| fungi | well, at least per the 1963 good housekeeping recipe. the 1938 fannie farmer recipe calls for working it on a marble slab, i should have one but do not at present | 00:33 |
| fungi | this is very reminiscent of tempering chocolate | 00:34 |
| corvus | all stopped | 00:37 |
| corvus | starting up again | 00:38 |
| corvus | and i've rechecked 678875 to exercise it | 00:38 |
| fungi | grafana says we're back to 12 | 00:39 |
| fungi | and 20 mergers | 00:40 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Get ansible version at start of job https://review.opendev.org/686853 | 00:46 |
| corvus | saw that in the logs when it shut down ^ | 00:46 |
| corvus | otherwise things lgtm | 00:47 |
| fungi | yeah, that looks like a good idea | 00:49 |
| *** ricolin_ has joined #openstack-infra | 01:03 | |
| *** yamamoto has quit IRC | 01:04 | |
| *** yamamoto has joined #openstack-infra | 01:08 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** slaweq has quit IRC | 01:16 | |
| *** yamamoto has quit IRC | 01:21 | |
| *** michael-beaver has quit IRC | 01:33 | |
| *** yamamoto has joined #openstack-infra | 02:38 | |
| *** yamamoto has quit IRC | 02:48 | |
| *** yamamoto has joined #openstack-infra | 02:59 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Use $YUM instead of direct calls in more places https://review.opendev.org/684460 | 02:59 |
| openstackgerrit | Merged openstack/diskimage-builder master: yum-minimal: Don't install yum, install libcurl https://review.opendev.org/684461 | 02:59 |
| *** whoami-rajat has joined #openstack-infra | 03:12 | |
| *** yamamoto has quit IRC | 03:43 | |
| *** yamamoto has joined #openstack-infra | 03:48 | |
| *** factor has joined #openstack-infra | 03:53 | |
| *** yamamoto has quit IRC | 04:04 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** dklyle has quit IRC | 04:36 | |
| *** david-lyle has joined #openstack-infra | 04:36 | |
| *** yamamoto has joined #openstack-infra | 04:36 | |
| *** yamamoto has quit IRC | 04:47 | |
| *** david-lyle has quit IRC | 04:58 | |
| *** dklyle has joined #openstack-infra | 04:59 | |
| *** ociuhandu has joined #openstack-infra | 05:01 | |
| *** cyberpear has quit IRC | 05:04 | |
| *** ociuhandu has quit IRC | 05:05 | |
| *** yamamoto has joined #openstack-infra | 05:22 | |
| *** dklyle has quit IRC | 05:25 | |
| *** whoami-rajat has quit IRC | 05:31 | |
| openstackgerrit | Merged openstack/diskimage-builder master: pip-and-virtualenv : deprecate source for CentOS 8, new variables https://review.opendev.org/684462 | 05:53 |
| *** kjackal has joined #openstack-infra | 06:01 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** slaweq has quit IRC | 06:16 | |
| *** e0ne has joined #openstack-infra | 06:16 | |
| *** e0ne has quit IRC | 06:18 | |
| *** jbadiapa has quit IRC | 06:22 | |
| *** yamamoto has quit IRC | 06:35 | |
| *** yamamoto has joined #openstack-infra | 06:40 | |
| *** yamamoto has quit IRC | 06:40 | |
| *** yamamoto has joined #openstack-infra | 06:40 | |
| *** yamamoto has quit IRC | 06:44 | |
| *** whoami-rajat has joined #openstack-infra | 06:46 | |
| *** dpawlik has joined #openstack-infra | 07:03 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** kjackal has quit IRC | 07:13 | |
| *** slaweq has quit IRC | 07:15 | |
| *** slaweq has joined #openstack-infra | 07:18 | |
| *** yamamoto has joined #openstack-infra | 07:20 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove dependencies from promote https://review.opendev.org/686868 | 07:20 |
| mordred | corvus: https://zuul.opendev.org/t/zuul/build/0462c9784eb84765a83e346a3f0d7cd4 <-- upload image error with an http issue | 07:23 |
| *** yamamoto has quit IRC | 07:27 | |
| *** yamamoto has joined #openstack-infra | 07:50 | |
| *** prometheanfire has quit IRC | 07:53 | |
| *** prometheanfire has joined #openstack-infra | 07:54 | |
| *** yamamoto has quit IRC | 07:54 | |
| *** xenos76 has joined #openstack-infra | 08:17 | |
| *** xenos76 has quit IRC | 08:32 | |
| *** ociuhandu has joined #openstack-infra | 08:52 | |
| *** ociuhandu has quit IRC | 08:54 | |
| *** markvoelker has joined #openstack-infra | 09:00 | |
| *** markvoelker has quit IRC | 09:05 | |
| *** e0ne has joined #openstack-infra | 09:42 | |
| *** yamamoto has joined #openstack-infra | 09:53 | |
| *** xenos76 has joined #openstack-infra | 09:55 | |
| *** jistr has quit IRC | 10:23 | |
| *** jistr has joined #openstack-infra | 10:23 | |
| *** xenos76 has quit IRC | 10:31 | |
| *** e0ne has quit IRC | 10:36 | |
| *** yamamoto has quit IRC | 10:47 | |
| *** ociuhandu has joined #openstack-infra | 10:55 | |
| *** dpawlik has quit IRC | 10:57 | |
| *** ociuhandu has quit IRC | 11:00 | |
| *** xenos76 has joined #openstack-infra | 11:06 | |
| *** e0ne has joined #openstack-infra | 11:18 | |
| *** yamamoto has joined #openstack-infra | 11:24 | |
| *** markvoelker has joined #openstack-infra | 11:29 | |
| *** markvoelker has quit IRC | 11:33 | |
| *** yamamoto has quit IRC | 11:35 | |
| *** yamamoto has joined #openstack-infra | 11:44 | |
| *** rh-jelabarre has quit IRC | 11:48 | |
| *** odicha has joined #openstack-infra | 11:50 | |
| *** yamamoto has quit IRC | 12:04 | |
| *** e0ne has quit IRC | 12:04 | |
| *** factor has quit IRC | 12:07 | |
| *** factor has joined #openstack-infra | 12:07 | |
| *** yamamoto has joined #openstack-infra | 12:09 | |
| *** yamamoto has quit IRC | 12:13 | |
| *** tosky has joined #openstack-infra | 12:20 | |
| *** exsdev has quit IRC | 12:52 | |
| *** e0ne has joined #openstack-infra | 12:54 | |
| *** yamamoto has joined #openstack-infra | 12:58 | |
| *** whoami-rajat has quit IRC | 13:01 | |
| *** xenos76 has quit IRC | 13:02 | |
| *** dave-mccowan has joined #openstack-infra | 13:08 | |
| *** yamamoto has quit IRC | 13:09 | |
| fungi | yeah, 686853 is consistently hitting a POST_FAILURE on zuul-upload-image | 13:14 |
| donnyd | is there an issue with nodepool being able to reach FN | 13:16 |
| fungi | https://zuul.opendev.org/t/zuul/builds?job_name=zuul-upload-image says it's the only change to run that job since the registry switch | 13:16 |
| donnyd | It could be grafana just being a little behind, but it says its deleteing 102 test nodes and they have been gone for quite a while | 13:16 |
| fungi | donnyd: i don't think it's connectivity. the only fn errors which jump out at me in the nl02 logs are about insufficient quota | 13:31 |
| fungi | Not enough quota remaining to satisfy request 200-0005611699 | 13:32 |
| fungi | oh, here we go... | 13:33 |
| fungi | nodepool.exceptions.ServerDeleteException: Timeout waiting for server 6cdf39ad-7748-466b-8d07-5665af1e08d7 deletion | 13:33 |
| fungi | so maybe there are deletion problems? | 13:33 |
| fungi | looks like it started trying to delete that one at 12:57:00 | 13:34 |
| fungi | 2019-10-05 12:57:00,573 INFO nodepool.DeletedNodeWorker: Deleting used instance 6cdf39ad-7748-466b-8d07-5665af1e08d7 from fortnebula-regionone | 13:34 |
| donnyd | Oh those servers are deleted | 13:41 |
| donnyd | they have been for an hour | 13:41 |
| donnyd | Ok, looks like its running fine again | 13:41 |
| donnyd | weird, because it also looked like it happened a couple times http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=1570270402968&to=1570282640089 | 13:42 |
| donnyd | oh well... back to normal now | 13:42 |
| donnyd | my DC is loving this cooler weather.. down to 72 degrees in there | 13:43 |
| *** e0ne has quit IRC | 13:47 | |
| fungi | yeah, we were finally able to turn off our air conditioning and open our windows again last night for the first time in ages | 13:56 |
| *** Lucas_Gray has joined #openstack-infra | 14:01 | |
| *** goldyfruit_ has quit IRC | 14:06 | |
| fungi | so back to the zuul-upload-image failures, both seem to have broken at the same spot, in the "Push tag to intermediate registry" task, the skopeo command died with "Error writing blob: Error determining upload URL: http: no Location header in response" | 14:18 |
| AJaeger | skopeo? See https://review.opendev.org/686803 | 14:19 |
| AJaeger | fungi: do we need this one? ^ didn't review, just remembered seeing it flying by | 14:20 |
| *** whoami-rajat has joined #openstack-infra | 14:24 | |
| fungi | oh, quite possibly. it doesn't mention that error exactly, but makes sense that it could be related | 14:24 |
| AJaeger | tristanC: do we need that change? See 686853 and discussion above | 14:25 |
| *** kambiz has joined #openstack-infra | 14:28 | |
| tristanC | AJaeger: 686803 doesn't fix a missing Location header in response... so did skopeo got updated recently? | 14:32 |
| fungi | Oct 5 01:46:32 insecure-ci-registry01 bdff2d543272[23452]: INFO:registry.api:Start upload POST zuul/zuul-executor uuid 25557bf2b1fc49a79115a2567af5b1fe digest None | 14:32 |
| tristanC | is the zuul-registry already in use? | 14:33 |
| fungi | that's the log from the zuul-registry service corresponding to this i think: https://zuul.opendev.org/t/zuul/build/0462c9784eb84765a83e346a3f0d7cd4/log/job-output.txt#11060 | 14:34 |
| *** xenos76 has joined #openstack-infra | 14:34 | |
| fungi | tristanC: yeah, as of ~18:00z yesterday | 14:35 |
| tristanC | i've rechecked 686803 to see if it reproduces | 14:36 |
| *** pcaruana has joined #openstack-infra | 14:36 | |
| fungi | it did it a couple times already, but can't hurt to collect more examples | 14:37 |
| corvus | i'm looking at the logs for the registry and seeing if we need more logging | 14:37 |
| tristanC | perhaps this is because of https://github.com/containers/skopeo/commit/7922028d7c994c5a489283dbf04f32a7dca44f5e | 14:38 |
| tristanC | which touch a file that has the "Error determining upload URL" message | 14:39 |
| corvus | tristanC: did your local test with skopeo use that version? | 14:39 |
| tristanC | corvus: no, i use the fedora package which only provides released version. | 14:39 |
| tristanC | iirc the ppa is built continously and it may have picked that commit | 14:39 |
| corvus | ok, i'll grab skopeo from the ppa and test that locally | 14:40 |
| fungi | seems like that ppa for bionic hasn't been updated for weeks | 14:41 |
| *** lxkong has quit IRC | 14:41 | |
| tristanC | corvus: seems like the POST now excepts a location header | 14:42 |
| fungi | skopeo 0.1.39-1~ubuntu18.04~ppa1 | 14:42 |
| tristanC | the location should point at the PUT endpoint | 14:43 |
| corvus | tristanC: it sounds like you've found the error, but i still don't see it and can not reproduce; tell me more | 14:47 |
| corvus | locally, i've installed skopeo, ran into the bug that your skopeo patch fixed, i applied your patch locally, and a skopeo copy from a file to the registry works. i haven't seen the Location header error | 14:47 |
| tristanC | corvus: don't know yet, i'm currently building latest skopeo to try to reproduce locally | 14:47 |
| tristanC | hum, with the master tip of skopeo? | 14:48 |
| corvus | from the projectatomic ppa | 14:48 |
| corvus | i will try again after building a zuul image -- maybe that will reproduce | 14:49 |
| fungi | tristanC: https://launchpad.net/~projectatomic/+archive/ubuntu/ppa | 14:49 |
| corvus | (specifically, a zuul-executor image) | 14:50 |
| fungi | tristanC: skopeo 0.1.39-1~ubuntu18.04~ppa1 from there is what it looks like we're using | 14:50 |
| fungi | built a couple months ago, 2019-08-10 | 14:51 |
| tristanC | oh, and does that job ever worked with this skopeo and the zuul-registry? | 14:52 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add support for skopeo copy https://review.opendev.org/686803 | 14:52 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add timestamps to logs https://review.opendev.org/686895 | 14:53 |
| fungi | tristanC: no, not in the two times it's run since switching to zuul-registry, that i can tell | 14:53 |
| corvus | gah, i didn't mean to rebase tristanC's patch | 14:53 |
| corvus | looks like it's a no-change rebase | 14:53 |
| corvus | sorry about that | 14:53 |
| corvus | tristanC, fungi: can we go ahead and merge those 2 changes? ^ | 14:54 |
| fungi | yeah, already reviewing | 14:54 |
| corvus | since we don't have the access logs, it's possible we didn't log whatever is tripping up skopeo | 14:54 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Set debug logging in intermediate registry https://review.opendev.org/686896 | 14:56 |
| corvus | fungi, mordred: also ^ | 14:56 |
| fungi | corvus: as for timestamps, i correlated based on what got copied into syslog since there's already timestamping. are these other logs, or just timestamping them when recording to a different place? | 14:57 |
| corvus | fungi: oh neat i didn't think to check that :) | 14:58 |
| corvus | fungi: i was just looking at "docker logs" | 14:58 |
| corvus | fungi: should we omit timestamps then? | 14:58 |
| fungi | i was having trouble figuring out where to find the docker config file to make `docker-compose logs` work, then noticed docker was set to log via journald which is in turn copying to syslog | 14:58 |
| corvus | fungi: "docker logs" has a "-t" option which means that both ways we collect logs optionally have timestamps | 14:59 |
| fungi | i'm fine with timestamps appearing in docker logs output too | 14:59 |
| fungi | ahh | 14:59 |
| corvus | i'll rework that to omit timestamps for now, since it might be duplicative | 14:59 |
| fungi | wfm | 14:59 |
| corvus | fungi: the cherrypy access logs have an internal timestamp too (since they are standard web access log format) so that would make 3 if you viewed it in syslog. that seems gratuitous | 15:00 |
| fungi | yo dawg i heard you like timestamps so... | 15:00 |
| fungi | also interesting that we're getting urllib3 debug logging already in syslog. is that being set elsewhere? | 15:01 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add debug env variable https://review.opendev.org/686895 | 15:01 |
| fungi | ahh, i bet it's logging from openstacksdk | 15:01 |
| fungi | because i see swift client interactions being logged too | 15:02 |
| corvus | fungi: yeah, but the logger is supposed to be at info. i'm confused by that. | 15:02 |
| corvus | that's not what happens for me locally, but i'm not running it in a container | 15:02 |
| corvus | we're basically already getting debug level logging, except we're not getting the access logs, which i think we need for this. also, arguably, we should move the access logs to the standard info level anyway, but later. | 15:03 |
| fungi | yeah, i think early on having debug logs is a good idea | 15:04 |
| fungi | when we decide we're ready to turn them off we can figure out why we're still getting debug logging from openstacksdk | 15:05 |
| corvus | i reproduced | 15:05 |
| fungi | the skopeo error? | 15:06 |
| corvus | yeah, the one we saw in prod: http://paste.openstack.org/show/781181/ | 15:06 |
| corvus | i have no idea why that didn't show up in the container logs | 15:06 |
| fungi | nuts | 15:07 |
| fungi | stderr instead of stdout maybe? | 15:07 |
| openstackgerrit | Merged zuul/zuul-registry master: Add support for skopeo copy https://review.opendev.org/686803 | 15:09 |
| fungi | so the request header is too large for cherrypy? what are we stuffing in there which would bloat it? | 15:09 |
| corvus | i don't think it's headers | 15:10 |
| corvus | i think it's something about cherrypy and upload sizes | 15:13 |
| corvus | it may have a limit | 15:13 |
| fungi | ahh, yeah it's accumulating the bytes read and if the next chunk would put it over maxlen it raises | 15:14 |
| *** jamesmcarthur has joined #openstack-infra | 15:14 | |
| *** jamesmcarthur has quit IRC | 15:14 | |
| fungi | the ChunkedRFile class seems to accept maxlen as an argument so maybe it's tunable | 15:15 |
| *** jamesmcarthur has joined #openstack-infra | 15:15 | |
| tristanC | corvus: i was able to reproduce locally using a sufficiently large image, the zuul-registry yield OSError: Request Entity Too Large | 15:17 |
| *** jamesmcarthur has quit IRC | 15:17 | |
| *** markvoelker has joined #openstack-infra | 15:17 | |
| *** markvoelker has quit IRC | 15:18 | |
| *** jamesmcarthur has joined #openstack-infra | 15:18 | |
| corvus | fungi, tristanC: yep, i think this is just a safety limit for cherrypy. i believe the chunked upload is working as expected (ie, we're not filling up memory or anything). it's probably just a default that prevents abuse of "typical" web apps. and we should raise it very high. | 15:18 |
| tristanC | http://paste.openstack.org/show/781182/ | 15:18 |
| *** hwoarang has quit IRC | 15:20 | |
| fungi | yeah, their tests in cheroot have examples of setting wsgi_server.max_request_body_size | 15:20 |
| fungi | which propagates to there | 15:20 |
| corvus | we can also tell the client what size we would like | 15:20 |
| corvus | the cherrypy default is 100M | 15:20 |
| corvus | maybe we should stick with that and use the registry protocol to split further? | 15:21 |
| *** hwoarang has joined #openstack-infra | 15:21 | |
| fungi | that also seems reasonable. 100mb seems like a fairly large chunk size already | 15:21 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Increate max request body size to accept large entity https://review.opendev.org/686897 | 15:23 |
| corvus | (also, this is weird because they both use the same terms -- but the registry protocol has clients uploading "chunks" which are layers split up into multiple http requests -- each of those http requests are actually https chunked transfer encoded, so they are streamed efficiently; it's really the first part of this which is at issue) | 15:23 |
| tristanC | https://review.opendev.org/686897 fixed my local reproducer | 15:23 |
| corvus | fungi, tristanC: ^ i'm in favor of that for now, and next week working on the docker protocol side and maybe reducing it | 15:23 |
| fungi | yeah, i was using the term "chunk" in both senses | 15:24 |
| corvus | https://docs.docker.com/registry/spec/api/ "Chunked upload" and "416 Requested Range Not Satisfiable" are the bits i'm looking at for that | 15:24 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add debug env variable https://review.opendev.org/686895 | 15:28 |
| tristanC | corvus: another solution would be to use cherrypy.tools and manage the socket.read manually | 15:28 |
| tristanC | i had to do that to implement the kubectl exec protocol, and cherrypy let you take over easily the low level operation of the request socket | 15:29 |
| corvus | tristanC: i don't think we need to do that -- it looks like cherrypy is doing almost exactly what we would do anyway -- in fact, we should drop our 4096 argument to read, then we'll get whole chunks back which we can write to the backend | 15:30 |
| corvus | no, hrm, i'm wrong about that... let's also look at that next week | 15:32 |
| corvus | tristanC: maybe you're right and it would be more efficient for us to just grab the socket | 15:33 |
| corvus | tristanC: can you +3 https://review.opendev.org/686895 ? | 15:33 |
| *** Wryhder has joined #openstack-infra | 15:34 | |
| corvus | i think everything is approved, so in like an hour or 2, we should recheck that | 15:34 |
| *** Lucas_Gray has quit IRC | 15:35 | |
| corvus | meanwhile, i'm going to do go whatever it was i was about to do when i sat down here earlier :) | 15:35 |
| *** Wryhder is now known as Lucas_Gray | 15:35 | |
| fungi | yep, i need to stop procrastinating on overdue lawn care this morning, but will be around | 15:36 |
| tristanC | corvus: cherrypy doesn't seem to read the whole content in our back, but if it does then we could take over the socket | 15:36 |
| tristanC | have a good week end folks :-) | 15:36 |
| corvus | tristanC: you too, thanks! | 15:36 |
| openstackgerrit | Merged zuul/zuul-registry master: Increate max request body size to accept large entity https://review.opendev.org/686897 | 15:37 |
| *** tosky has quit IRC | 15:49 | |
| openstackgerrit | Merged zuul/zuul-registry master: Add debug env variable https://review.opendev.org/686895 | 15:49 |
| fungi | #status log trove instance Etherpad-Dev-MySQL was shut down temporarily by the provider for a proactive cold migration due to an impending host failure, trouble ticket #191005-ord-0000331 | 16:00 |
| openstackstatus | fungi: finished logging | 16:00 |
| *** jamesmcarthur has quit IRC | 16:02 | |
| *** jamesmcarthur has joined #openstack-infra | 16:04 | |
| *** icarusfactor has joined #openstack-infra | 16:06 | |
| *** jamesmcarthur has quit IRC | 16:08 | |
| openstackgerrit | Merged opendev/system-config master: Set debug logging in intermediate registry https://review.opendev.org/686896 | 16:17 |
| *** odicha has quit IRC | 16:19 | |
| *** ricolin_ has quit IRC | 16:20 | |
| *** jamesmcarthur has joined #openstack-infra | 16:43 | |
| *** dave-mccowan has quit IRC | 16:43 | |
| AJaeger | zuul-build-image is working in https://review.opendev.org/686853 now | 16:43 |
| fungi | yay! | 16:44 |
| AJaeger | but why is it not entering gate pipeline again after a recheck? Bug in Zuul? | 16:45 |
| AJaeger | or did we configure the Zuul tenant wrongly (I think I saw this already in zuul-jobs but didn't dig deeper) | 16:45 |
| AJaeger | fungi, shouldn't it enter gate without your extra +A? | 16:46 |
| fungi | not sure, but it got enqueued as soon as i added a new workflow +1 | 16:46 |
| AJaeger | something is strange with zuul tenant, with rechecks - in openstack tenant, it would enqueue automatically, seems it's not working in zuul tenant. corvus, any idea? | 16:47 |
| *** jamesmcarthur has quit IRC | 16:48 | |
| *** jamesmcarthur has joined #openstack-infra | 17:01 | |
| *** jamesmcarthur has quit IRC | 17:06 | |
| *** pcaruana has quit IRC | 17:18 | |
| *** markvoelker has joined #openstack-infra | 17:19 | |
| *** markvoelker has quit IRC | 17:24 | |
| openstackgerrit | Merged zuul/zuul master: Get ansible version at start of job https://review.opendev.org/686853 | 17:30 |
| *** e0ne has joined #openstack-infra | 17:33 | |
| *** e0ne has quit IRC | 17:36 | |
| *** slaweq has quit IRC | 17:44 | |
| *** diablo_rojo has joined #openstack-infra | 18:20 | |
| *** goldyfruit_ has joined #openstack-infra | 18:25 | |
| *** yamamoto has joined #openstack-infra | 19:07 | |
| *** yamamoto has quit IRC | 19:12 | |
| *** Lucas_Gray has quit IRC | 19:33 | |
| *** diablo_rojo has quit IRC | 20:18 | |
| *** mattmceuen has joined #openstack-infra | 20:18 | |
| mattmceuen | o/ whenever anyone gets a chance, could you please add me as a seed member of the airshipapis-core group? https://review.opendev.org/#/admin/groups/2066,members | 20:20 |
| mattmceuen | thanks! | 20:20 |
| fungi | mattmceuen: done! | 20:21 |
| mattmceuen | ty fungi! | 20:21 |
| fungi | any time | 20:21 |
| *** e0ne has joined #openstack-infra | 20:52 | |
| *** eernst_ has joined #openstack-infra | 21:00 | |
| *** xenos76 has quit IRC | 21:01 | |
| *** e0ne has quit IRC | 21:02 | |
| *** markvoelker has joined #openstack-infra | 21:22 | |
| *** markvoelker has quit IRC | 21:26 | |
| *** icarusfactor has quit IRC | 22:02 | |
| *** yamamoto has joined #openstack-infra | 22:39 | |
| *** yamamoto has quit IRC | 22:48 | |
| *** yamamoto has joined #openstack-infra | 22:51 | |
| *** yamamoto has quit IRC | 22:57 | |
| *** jamesmcarthur has joined #openstack-infra | 22:59 | |
| *** jamesmcarthur has quit IRC | 23:03 | |
| *** jamesmcarthur has joined #openstack-infra | 23:04 | |
| *** jamesmcarthur has quit IRC | 23:08 | |
| *** slittle1 has quit IRC | 23:16 | |
| *** slittle1 has joined #openstack-infra | 23:20 | |
| *** jamesmcarthur has joined #openstack-infra | 23:22 | |
| *** yamamoto has joined #openstack-infra | 23:23 | |
| *** yamamoto has quit IRC | 23:23 | |
| *** yamamoto has joined #openstack-infra | 23:24 | |
| *** yamamoto has quit IRC | 23:28 | |
| *** slittle1 has quit IRC | 23:30 | |
| *** slittle1 has joined #openstack-infra | 23:37 | |
| *** slittle1 has quit IRC | 23:40 | |
| *** jamesmcarthur has quit IRC | 23:49 | |
| *** jamesmcarthur has joined #openstack-infra | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!