| *** armax has quit IRC | 00:17 | |
| *** armax has joined #openstack-infra | 00:21 | |
| *** ricolin has quit IRC | 00:22 | |
| *** hamalq has quit IRC | 00:30 | |
| *** armax has quit IRC | 00:34 | |
| *** rfolco|ruck has joined #openstack-infra | 00:38 | |
| *** rfolco|ruck has quit IRC | 00:42 | |
| *** gyee has quit IRC | 01:03 | |
| openstackgerrit | Merged openstack/project-config master: Remove noop jobs for python-adjutantclient https://review.opendev.org/750990 | 01:05 |
|---|---|---|
| *** armax has joined #openstack-infra | 01:12 | |
| *** zbr7 has joined #openstack-infra | 01:16 | |
| *** Diabelko has joined #openstack-infra | 01:19 | |
| *** zbr has quit IRC | 01:19 | |
| *** rchanter has quit IRC | 01:19 | |
| *** systemd has quit IRC | 01:19 | |
| *** zbr7 is now known as zbr | 01:19 | |
| *** rchanter has joined #openstack-infra | 01:19 | |
| *** zzzeek has quit IRC | 01:21 | |
| *** zzzeek has joined #openstack-infra | 01:22 | |
| *** rfolco|ruck has joined #openstack-infra | 01:36 | |
| *** hamalq has joined #openstack-infra | 01:37 | |
| *** rfolco|ruck has quit IRC | 01:41 | |
| *** hamalq_ has joined #openstack-infra | 01:41 | |
| *** hamalq has quit IRC | 01:44 | |
| *** rcernin has quit IRC | 02:29 | |
| *** lbragstad has quit IRC | 02:31 | |
| *** zzzeek has quit IRC | 02:54 | |
| *** zzzeek has joined #openstack-infra | 02:54 | |
| *** ramishra has joined #openstack-infra | 02:58 | |
| *** zzzeek has quit IRC | 03:01 | |
| *** zzzeek has joined #openstack-infra | 03:04 | |
| *** rcernin has joined #openstack-infra | 03:15 | |
| *** dave-mccowan has quit IRC | 03:16 | |
| *** hamalq_ has quit IRC | 03:18 | |
| *** ricolin has joined #openstack-infra | 03:23 | |
| *** psachin has joined #openstack-infra | 03:35 | |
| *** psachin has quit IRC | 03:36 | |
| *** psachin has joined #openstack-infra | 03:37 | |
| *** ricolin_ has joined #openstack-infra | 03:42 | |
| *** ykarel|away has joined #openstack-infra | 04:01 | |
| *** ianychoi has quit IRC | 04:29 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| *** ykarel|away is now known as ykarel | 05:00 | |
| openstackgerrit | Merged openstack/project-config master: Remove openstack-python-jobs from GBP templates https://review.opendev.org/752575 | 05:05 |
| *** zzzeek has quit IRC | 05:19 | |
| *** zzzeek has joined #openstack-infra | 05:19 | |
| *** xek has joined #openstack-infra | 05:26 | |
| *** vishalmanchanda has joined #openstack-infra | 05:33 | |
| *** lmiccini has joined #openstack-infra | 05:38 | |
| *** hamalq has joined #openstack-infra | 05:44 | |
| *** matt_kosut has joined #openstack-infra | 05:49 | |
| *** xek has quit IRC | 06:04 | |
| *** ysandeep|away is now known as ysandeep | 06:07 | |
| *** Tengu has quit IRC | 06:07 | |
| *** ralonsoh has joined #openstack-infra | 06:08 | |
| *** slaweq has joined #openstack-infra | 06:15 | |
| *** eolivare has joined #openstack-infra | 06:21 | |
| *** auristor has quit IRC | 06:21 | |
| *** auristor has joined #openstack-infra | 06:28 | |
| *** hamalq has quit IRC | 06:29 | |
| *** hamalq has joined #openstack-infra | 06:32 | |
| *** ramishra has quit IRC | 06:41 | |
| *** dklyle has quit IRC | 06:41 | |
| *** hamalq has quit IRC | 06:44 | |
| *** zbr5 has joined #openstack-infra | 06:45 | |
| *** amorin has quit IRC | 06:46 | |
| *** zbr5 has quit IRC | 06:49 | |
| *** zbr has quit IRC | 06:49 | |
| *** hamalq has joined #openstack-infra | 06:52 | |
| *** samueldmq has quit IRC | 06:52 | |
| *** yonglihe has quit IRC | 06:53 | |
| *** portdirect has quit IRC | 06:53 | |
| *** seongsoocho has quit IRC | 06:53 | |
| *** coreycb has quit IRC | 06:53 | |
| *** guilhermesp has quit IRC | 06:53 | |
| *** gagehugo has quit IRC | 06:53 | |
| *** dougwig has quit IRC | 06:53 | |
| *** cyberpear has quit IRC | 06:53 | |
| *** mwhahaha has quit IRC | 06:53 | |
| *** hogepodge has quit IRC | 06:53 | |
| *** johnsom has quit IRC | 06:53 | |
| *** vdrok has quit IRC | 06:53 | |
| *** rpittau|afk has quit IRC | 06:53 | |
| *** nicolasbock has quit IRC | 06:53 | |
| *** srwilkers has quit IRC | 06:53 | |
| *** donnyd has quit IRC | 06:53 | |
| *** csatari has quit IRC | 06:53 | |
| *** rm_work has quit IRC | 06:53 | |
| *** masayukig has quit IRC | 06:53 | |
| *** TheJulia has quit IRC | 06:53 | |
| *** ildikov has quit IRC | 06:53 | |
| *** mattmceuen has quit IRC | 06:53 | |
| *** samueldmq has joined #openstack-infra | 06:55 | |
| *** gagehugo has joined #openstack-infra | 06:55 | |
| *** nicolasbock has joined #openstack-infra | 06:55 | |
| *** zbr has joined #openstack-infra | 06:55 | |
| *** mwhahaha has joined #openstack-infra | 06:55 | |
| *** dougwig has joined #openstack-infra | 06:55 | |
| *** mattmceuen has joined #openstack-infra | 06:55 | |
| *** seongsoocho has joined #openstack-infra | 06:55 | |
| *** vdrok has joined #openstack-infra | 06:56 | |
| *** yonglihe has joined #openstack-infra | 06:56 | |
| *** ildikov has joined #openstack-infra | 06:56 | |
| *** portdirect has joined #openstack-infra | 06:56 | |
| *** coreycb has joined #openstack-infra | 06:56 | |
| *** guilhermesp has joined #openstack-infra | 06:56 | |
| *** masayukig has joined #openstack-infra | 06:56 | |
| *** cyberpear has joined #openstack-infra | 06:56 | |
| *** srwilkers has joined #openstack-infra | 06:56 | |
| *** hashar has joined #openstack-infra | 06:56 | |
| *** rpittau|afk has joined #openstack-infra | 06:56 | |
| *** donnyd has joined #openstack-infra | 06:57 | |
| *** csatari has joined #openstack-infra | 06:57 | |
| *** TheJulia has joined #openstack-infra | 06:57 | |
| *** johnsom has joined #openstack-infra | 06:57 | |
| *** hogepodge has joined #openstack-infra | 06:59 | |
| *** debian has joined #openstack-infra | 06:59 | |
| *** debian is now known as Guest68711 | 06:59 | |
| *** Guest68711 has quit IRC | 07:00 | |
| *** debian1 has joined #openstack-infra | 07:00 | |
| *** andrewbonney has joined #openstack-infra | 07:01 | |
| *** debian1 has quit IRC | 07:02 | |
| *** amorin has joined #openstack-infra | 07:02 | |
| *** rchanter has quit IRC | 07:03 | |
| *** hamalq has quit IRC | 07:05 | |
| *** rm_work has joined #openstack-infra | 07:08 | |
| *** jcapitao has joined #openstack-infra | 07:10 | |
| *** derekh has joined #openstack-infra | 07:20 | |
| *** tosky has joined #openstack-infra | 07:25 | |
| *** gfidente|afk is now known as gfidente | 07:41 | |
| *** ociuhandu has joined #openstack-infra | 07:45 | |
| *** jpena|off is now known as jpena | 07:49 | |
| *** ykarel_ has joined #openstack-infra | 08:02 | |
| *** ramishra has joined #openstack-infra | 08:04 | |
| *** ykarel has quit IRC | 08:05 | |
| *** amorin has quit IRC | 08:08 | |
| *** ykarel_ is now known as ykarel | 08:10 | |
| *** lucasagomes has joined #openstack-infra | 08:14 | |
| *** ricolin_ has quit IRC | 08:22 | |
| *** dtantsur|afk is now known as dtantsur | 08:27 | |
| *** amorin has joined #openstack-infra | 08:51 | |
| *** mordred has quit IRC | 08:52 | |
| *** wolsen has quit IRC | 08:53 | |
| *** derekh has quit IRC | 08:55 | |
| *** mordred has joined #openstack-infra | 09:18 | |
| *** mordred has quit IRC | 09:19 | |
| *** derekh has joined #openstack-infra | 09:26 | |
| *** mordred has joined #openstack-infra | 09:26 | |
| *** vishalmanchanda has quit IRC | 09:43 | |
| *** Diabelko is now known as systemd | 09:53 | |
| *** vishalmanchanda has joined #openstack-infra | 09:57 | |
| *** wolsen has joined #openstack-infra | 10:08 | |
| *** psachin has quit IRC | 10:09 | |
| *** rcernin has quit IRC | 10:09 | |
| *** psachin has joined #openstack-infra | 10:10 | |
| *** jcapitao is now known as jcapitao_lunch | 10:16 | |
| *** psachin has quit IRC | 10:16 | |
| *** psachin has joined #openstack-infra | 10:17 | |
| *** xek has joined #openstack-infra | 10:33 | |
| *** rcernin has joined #openstack-infra | 10:44 | |
| *** hashar is now known as hasharAway | 10:47 | |
| *** andrewbonney has quit IRC | 10:51 | |
| *** rcernin has quit IRC | 10:54 | |
| *** ysandeep is now known as ysandeep|afk | 11:01 | |
| *** xek has quit IRC | 11:05 | |
| *** rcernin has joined #openstack-infra | 11:18 | |
| *** rcernin has quit IRC | 11:23 | |
| *** jpena is now known as jpena|lunch | 11:39 | |
| *** ysandeep|afk is now known as ysandeep | 11:42 | |
| *** hasharAway is now known as hashar | 11:51 | |
| *** rcernin has joined #openstack-infra | 11:54 | |
| *** jcapitao_lunch is now known as jcapitao | 11:59 | |
| *** rcernin has quit IRC | 11:59 | |
| *** ricolin_ has joined #openstack-infra | 12:02 | |
| *** rfolco|ruck has joined #openstack-infra | 12:07 | |
| *** rlandy has joined #openstack-infra | 12:07 | |
| *** andrewbonney has joined #openstack-infra | 12:36 | |
| *** jpena|lunch is now known as jpena | 12:37 | |
| *** Goneri has joined #openstack-infra | 12:42 | |
| *** dave-mccowan has joined #openstack-infra | 13:04 | |
| *** owalsh has quit IRC | 13:11 | |
| *** ysandeep is now known as ysandeep|session | 13:16 | |
| *** tosky_ has joined #openstack-infra | 13:18 | |
| *** tosky is now known as Guest83047 | 13:18 | |
| *** tosky_ is now known as tosky | 13:18 | |
| *** xek has joined #openstack-infra | 13:36 | |
| dulek | infra-root: The frozen VM at 198.72.124.155 can be destroyed now, thanks a lot for help! | 13:49 |
| fungi | dulek: were you able to identify the cause of your problem? curious what it was | 13:53 |
| dulek | fungi: Well, that's complicated. So apparently we were missing some local.conf option for OVN to work, yet when synchronizing them to match what's on Neutron gates I removed one that's required for Kuryr to work. | 13:55 |
| *** rcernin has joined #openstack-infra | 13:55 | |
| dulek | So changes were fixing it but I broke another thing. | 13:56 |
| dulek | Anyway - I identified it and it seems to work now. | 13:56 |
| fungi | oh, excellent | 13:56 |
| fungi | and thanks for the heads up, i'll release the node so it can be destroyed | 13:56 |
| *** artom has joined #openstack-infra | 13:56 | |
| dulek | fungi: But if we're talking already… So the OVN thing is just one of many stacked issues we face now. One is related to oom-killer that started to kill processes during our tests. | 13:57 |
| dulek | fungi: We're using Octavia and Amphora, so the mem footprint is very high, but this started to happen way more frequently last Friday. | 13:57 |
| dulek | Do you know if anything changed that could cause it? | 13:58 |
| fungi | interesting, do you have an example build result from shortly before the problem started, for comparison? | 13:58 |
| fungi | a number of jobs for master/victoria have been getting switched from ubuntu 18.04 to 20.04 in preparation for the release | 13:58 |
| dulek | We were pretty close to the limit anyway, so it's possible something just started using 200 MB RAM more and we're toasted. | 13:59 |
| fungi | possible that happened in the timeframe you're seeing | 13:59 |
| dulek | Hm, interesting, let me find you two results. | 13:59 |
| fungi | are you running dstat in your jobs? might be able to compare the dstat logs between them | 13:59 |
| dulek | fungi: We do, but dstat's has a bug so it shows one of our services is using TB of RAM. ;) | 14:00 |
| *** rcernin has quit IRC | 14:00 | |
| fungi | oof, yeah that sounds wrong :/ | 14:00 |
| dulek | So this is fresh failed run: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ad9/752233/3/check/kuryr-kubernetes-tempest/ad96334/ | 14:00 |
| dulek | If I read it correctly it's still bionic. | 14:01 |
| dulek | And that's a successful run from a while ago: https://zuul.opendev.org/t/openstack/build/7bd9bf946e774326b099e97852f0c9f8 | 14:01 |
| fungi | yes, that looks like bionic to me | 14:01 |
| fungi | (the failed one) | 14:01 |
| fungi | so it's not the platform switch at least | 14:02 |
| dulek | Not like those help to debug, but they're pretty cool - mem stats from a run without the issue and with the issue: | 14:04 |
| dulek | https://i.imgur.com/jzKe5PG.png and https://i.imgur.com/nqojzRK.png | 14:04 |
| fungi | dulek: you probably know without me having to dig in the setup logs, but are you not enabling swap? it's possible that could help in this case | 14:05 |
| fungi | at least enough to get a better picture of what's started using more ram | 14:06 |
| dulek | fungi: Uhm, honestly I didn't knew it's possible. | 14:06 |
| dulek | So we probably don't. | 14:06 |
| dulek | But that could help for sure, swapping out Amphora's should be fairly okay. | 14:06 |
| *** xek has quit IRC | 14:09 | |
| fungi | dulek: looking at the failed run, it did create a 1gb swapfile | 14:10 |
| fungi | (the configure-swap role tasks) | 14:11 |
| dulek | "Setting up swapspace version 1, size = 8 GiB" | 14:12 |
| dulek | Isn't it 8 actually? | 14:12 |
| *** hashar has quit IRC | 14:12 | |
| *** zxiiro has joined #openstack-infra | 14:12 | |
| *** hashar has joined #openstack-infra | 14:12 | |
| dulek | fungi: OH DAMN IT. | 14:16 |
| dulek | https://opendev.org/openstack/openstack-zuul-jobs/commit/45f555fdf036de786b5988213b458b3b12dcef74 | 14:16 |
| dulek | Exactly this. | 14:16 |
| dulek | fungi: How can I overwrite that in our jobs? | 14:17 |
| *** ysandeep|session is now known as ysandeep | 14:19 | |
| *** ysandeep is now known as ysandeep|away | 14:21 | |
| fungi | aha! i thought for a moment to suggest the fallocate->dd switch, but ruled it out forgetting that it also reduced the swap size significantly | 14:25 |
| fungi | dulek: i think configure_swap_size is probably plumbed through the devstack parent jobs... looking for an example now | 14:26 |
| dulek | fungi: I think there's none, but shouldn't putting it on `vars` in job config help? | 14:27 |
| fungi | maybe... the only example codesearch turns up outside the role itself is here: https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/prepare-node/tasks/main.yaml#L3 | 14:27 |
| fungi | that's passing it directly to the role | 14:27 |
| fungi | but i think ansible will inherit the vars like you're suggesting | 14:28 |
| dulek | I'll check it. Thanks a lot for help, we're struggling with that for the whole week. | 14:28 |
| fungi | easy enough to see if you propose it | 14:28 |
| fungi | it should get applied speculatively, so just look at the resulting run and check the configure-swap output | 14:29 |
| *** owalsh has joined #openstack-infra | 14:29 | |
| dulek | Yup! | 14:29 |
| fungi | though you may have to recheck a few times to hit a node with no ephemeral disk | 14:29 |
| fungi | oh, nevermind, it uses that value when sizing swap partitions too | 14:30 |
| fungi | so yeah, you ought to be able to tell regardless of where it runs | 14:30 |
| *** dmellado has quit IRC | 14:40 | |
| *** dmellado_ has joined #openstack-infra | 14:40 | |
| *** dmellado_ is now known as dmellado | 14:41 | |
| *** __ministry1 has joined #openstack-infra | 14:42 | |
| dulek | Looks good: "8589934592 bytes (8.6 GB, 8.0 GiB) copied, 56.9489 s, 151 MB/s" :) | 14:43 |
| dulek | Yay, what a way to end the work week! | 14:43 |
| fungi | dulek: awesome! | 14:43 |
| fungi | hard to beat a one-liner fix ;) | 14:43 |
| fungi | dulek: if you have a few minutes, that may also be worth posting to openstack-discuss in case any other projects are struggling with the same symptoms | 14:44 |
| dulek | fungi: Yeah, I was planning to do that. | 14:44 |
| fungi | i definitely didn't remember that we reduced the swap default when switching away from fallocate | 14:45 |
| clarkb | its because dd is slow | 14:58 |
| clarkb | that will add 5 minutes to all your jobs ir something | 14:58 |
| clarkb | unfortunately ext4 and faalocate decided to stop working for swap files :( | 14:58 |
| dulek | clarkb: Well, that's better than having broken jobs. | 14:58 |
| dulek | We weren't able to merge anything since last Friday. | 14:59 |
| *** ianychoi has joined #openstack-infra | 14:59 | |
| clarkb | dulek: I would argue that needing more than 1GB swap is broken ;) | 14:59 |
| *** ykarel is now known as ykarel|away | 14:59 | |
| dulek | clarkb: Oh, I can just make all of my jobs multinode instead to have more RAM. ;) | 14:59 |
| clarkb | its really there to keep a job that is on the cusp from failing. Not to ensure jobs can get by deep into swap | 14:59 |
| clarkb | dulek: yes thats the idea acyually | 15:00 |
| dulek | Isn't swap cheaper in that case? | 15:00 |
| *** nightmare_unreal has quit IRC | 15:00 | |
| dulek | Okay, anyway it's only because we're running Octavia and that means Amphora VMs and each uses 1 GB. | 15:01 |
| clarkb | in isolation likely. But heavy useof swap appears to be amajor contributor to our noisy neoghbor problwm | 15:01 |
| dulek | We'll be switching to use ovn-octavia more soon, which makes this irrelevant. | 15:01 |
| clarkb | you eatup all the disk iops | 15:01 |
| johnsom | No, your VMs are about 150MB per your mem tracker | 15:01 |
| clarkb | then a bunch of jobs fail in weird ways | 15:01 |
| *** dklyle has joined #openstack-infra | 15:02 | |
| dulek | johnsom: Hm. So what's chewing up all that RAM on our gates? | 15:02 |
| clarkb | the human side then spends a lot of cycles trying to understand the weird behavior | 15:02 |
| dulek | And I know, kuryr-daemon appears to use 1 TB of RAM, but that's dstat's bug. | 15:02 |
| clarkb | dulek: dstat's csv log should tell you | 15:02 |
| dulek | clarkb: Got it. We'll prioritize switch to ovn-octavia and should be able to drop that swap thing soon. | 15:03 |
| johnsom | Yeah, I downloaded the memtracker data and loaded it up in a spreadsheet. You had four load balancers, next was 500MB mysql and a fairly large etcd | 15:03 |
| dulek | And then all the Python services. Interesting. | 15:04 |
| johnsom | Then a bunch of kube stuff and the openstack parts | 15:04 |
| dulek | johnsom: Thanks for taking a look, I appreciate it! | 15:06 |
| johnsom | Sure, NP. Dropping that thread count will help save you some RAM too. | 15:07 |
| *** smarcet has joined #openstack-infra | 15:09 | |
| *** xek has joined #openstack-infra | 15:16 | |
| *** xek has quit IRC | 15:17 | |
| *** smarcet has quit IRC | 15:24 | |
| *** lmiccini has quit IRC | 15:27 | |
| *** smarcet has joined #openstack-infra | 15:42 | |
| *** d34dh0r53 has quit IRC | 15:43 | |
| *** d34dh0r53 has joined #openstack-infra | 15:44 | |
| *** tkajinam has quit IRC | 15:44 | |
| *** ykarel|away has quit IRC | 15:49 | |
| *** ricolin_ has quit IRC | 15:50 | |
| *** lucasagomes has quit IRC | 15:54 | |
| *** rcernin has joined #openstack-infra | 15:56 | |
| *** smarcet has quit IRC | 16:02 | |
| *** rcernin has quit IRC | 16:03 | |
| *** jcapitao has quit IRC | 16:08 | |
| *** eolivare has quit IRC | 16:08 | |
| *** dtantsur is now known as dtantsur|afk | 16:10 | |
| *** smarcet has joined #openstack-infra | 16:12 | |
| *** hashar has quit IRC | 16:21 | |
| *** vishalmanchanda has quit IRC | 16:23 | |
| *** ociuhandu_ has joined #openstack-infra | 16:23 | |
| *** ociuhandu has quit IRC | 16:27 | |
| *** ociuhandu_ has quit IRC | 16:28 | |
| *** ociuhandu has joined #openstack-infra | 16:33 | |
| *** ociuhandu has quit IRC | 16:37 | |
| *** __ministry1 has quit IRC | 16:41 | |
| *** psachin has quit IRC | 16:49 | |
| *** Tengu has joined #openstack-infra | 16:54 | |
| *** jpena is now known as jpena|off | 16:56 | |
| *** d34dh0r53 has quit IRC | 16:58 | |
| *** d34dh0r53 has joined #openstack-infra | 17:00 | |
| *** derekh has quit IRC | 17:01 | |
| *** hamalq has joined #openstack-infra | 17:01 | |
| *** iurygregory has quit IRC | 17:10 | |
| *** andrewbonney has quit IRC | 17:11 | |
| *** gfidente has quit IRC | 17:12 | |
| *** ralonsoh has quit IRC | 17:14 | |
| *** harlowja has joined #openstack-infra | 17:26 | |
| *** gyee has joined #openstack-infra | 17:35 | |
| openstackgerrit | Thomas Bachman proposed openstack/project-config master: Remove legacy gate jobs https://review.opendev.org/752735 | 17:46 |
| *** d34dh0r53 has quit IRC | 18:07 | |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Remove old nodepool builder configs https://review.opendev.org/752741 | 18:14 |
| *** d34dh0r53 has joined #openstack-infra | 18:15 | |
| *** irclogbot_1 has quit IRC | 18:19 | |
| *** irclogbot_1 has joined #openstack-infra | 18:23 | |
| *** irclogbot_1 has quit IRC | 18:31 | |
| *** irclogbot_2 has joined #openstack-infra | 18:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch grafana nodepool to Fedora 32 https://review.opendev.org/752745 | 18:42 |
| *** d34dh0r53 has quit IRC | 19:02 | |
| *** d34dh0r53 has joined #openstack-infra | 19:07 | |
| *** zxiiro has quit IRC | 19:17 | |
| *** zxiiro has joined #openstack-infra | 19:38 | |
| *** slaweq has quit IRC | 19:41 | |
| *** zzzeek has quit IRC | 20:06 | |
| *** zzzeek has joined #openstack-infra | 20:42 | |
| *** owalsh has quit IRC | 20:46 | |
| *** zzzeek has quit IRC | 20:48 | |
| *** zzzeek has joined #openstack-infra | 20:54 | |
| *** zzzeek has quit IRC | 20:59 | |
| *** harlowja has quit IRC | 21:01 | |
| *** ociuhandu has joined #openstack-infra | 21:02 | |
| *** zzzeek has joined #openstack-infra | 21:03 | |
| *** zzzeek has quit IRC | 21:05 | |
| *** zzzeek has joined #openstack-infra | 21:06 | |
| *** ociuhandu has quit IRC | 21:06 | |
| *** zzzeek has quit IRC | 21:13 | |
| *** matt_kosut has quit IRC | 21:14 | |
| *** owalsh has joined #openstack-infra | 21:14 | |
| *** armax has quit IRC | 21:15 | |
| *** zzzeek has joined #openstack-infra | 21:16 | |
| *** rlandy has quit IRC | 21:18 | |
| *** zzzeek has quit IRC | 21:23 | |
| *** zzzeek has joined #openstack-infra | 21:24 | |
| *** rcernin has joined #openstack-infra | 21:27 | |
| *** smarcet has quit IRC | 21:33 | |
| *** rcernin has quit IRC | 21:35 | |
| *** rcernin has joined #openstack-infra | 21:36 | |
| *** Goneri has quit IRC | 21:39 | |
| *** rcernin has quit IRC | 21:42 | |
| *** rcernin has joined #openstack-infra | 22:00 | |
| *** smarcet has joined #openstack-infra | 22:01 | |
| *** smarcet has quit IRC | 22:05 | |
| *** dave-mccowan has quit IRC | 22:22 | |
| *** zzzeek has quit IRC | 22:25 | |
| *** zzzeek has joined #openstack-infra | 22:26 | |
| *** zzzeek has quit IRC | 22:30 | |
| *** zzzeek has joined #openstack-infra | 22:31 | |
| *** eharney has quit IRC | 22:35 | |
| *** rfolco|ruck has quit IRC | 22:56 | |
| *** hamalq has quit IRC | 23:02 | |
| *** EmilienM has quit IRC | 23:06 | |
| *** EmilienM has joined #openstack-infra | 23:06 | |
| *** rcernin has quit IRC | 23:06 | |
| *** zzzeek has quit IRC | 23:08 | |
| *** zzzeek has joined #openstack-infra | 23:11 | |
| *** zzzeek has quit IRC | 23:16 | |
| *** zzzeek has joined #openstack-infra | 23:17 | |
| *** armax has joined #openstack-infra | 23:27 | |
| *** tosky has quit IRC | 23:31 | |
| *** rcernin has joined #openstack-infra | 23:38 | |
| *** Goneri has joined #openstack-infra | 23:44 | |
| *** hamalq has joined #openstack-infra | 23:48 | |
| *** armax has quit IRC | 23:54 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
