| openstackgerrit | Chris Wedgwood proposed airship/promenade master: controller_manager: Add pod/container security context https://review.opendev.org/651417 | 00:05 |
|---|---|---|
| *** michael-beaver has quit IRC | 00:23 | |
| *** kaspars__ has quit IRC | 00:26 | |
| *** Nishant_ has quit IRC | 01:15 | |
| *** mbeierl has joined #airshipit | 02:20 | |
| *** mbeierl has quit IRC | 02:25 | |
| *** mbeierl has joined #airshipit | 02:38 | |
| *** mbeierl has quit IRC | 02:43 | |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Fix chart repo urls from openstack to opendev https://review.opendev.org/654579 | 03:05 |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Uplift tiller/help to 2.13.1 and Airship/OSH charts https://review.opendev.org/653963 | 03:05 |
| *** kaspars__ has joined #airshipit | 03:12 | |
| openstackgerrit | Merged airship/treasuremap master: Update docs, clarify the requirements on DNS names registration https://review.opendev.org/654559 | 03:15 |
| openstackgerrit | Merged airship/treasuremap master: Add metadata proxy shared secret for Nova and Neutron https://review.opendev.org/639183 | 03:43 |
| *** mbeierl has joined #airshipit | 06:08 | |
| *** mbeierl has quit IRC | 06:14 | |
| *** lemko has joined #airshipit | 06:15 | |
| *** mbeierl has joined #airshipit | 06:27 | |
| *** kaspars__ has quit IRC | 06:30 | |
| *** mbeierl has quit IRC | 06:31 | |
| *** georgk has joined #airshipit | 07:20 | |
| *** pkaralis has joined #airshipit | 07:28 | |
| *** roman_g has joined #airshipit | 08:01 | |
| *** nick_kar has joined #airshipit | 08:49 | |
| *** happyhemant has joined #airshipit | 09:33 | |
| *** lemko has quit IRC | 09:44 | |
| *** mbeierl has joined #airshipit | 10:06 | |
| *** mbeierl has quit IRC | 10:10 | |
| *** mbeierl has joined #airshipit | 10:33 | |
| *** aojea has joined #airshipit | 10:35 | |
| *** mbeierl has quit IRC | 10:37 | |
| *** mbeierl has joined #airshipit | 11:01 | |
| *** mbeierl has quit IRC | 11:06 | |
| *** mbeierl has joined #airshipit | 11:19 | |
| *** mbeierl has quit IRC | 11:24 | |
| *** mbeierl has joined #airshipit | 11:37 | |
| *** mbeierl has quit IRC | 11:42 | |
| *** jemangs has joined #airshipit | 11:44 | |
| *** lemko has joined #airshipit | 11:49 | |
| *** Gpshz1_ has joined #airshipit | 12:03 | |
| *** Gpshz1 has quit IRC | 12:06 | |
| *** aaronsheffield has joined #airshipit | 12:51 | |
| *** bh526r has joined #airshipit | 12:52 | |
| *** mbeierl has joined #airshipit | 13:01 | |
| *** mbeierl has quit IRC | 13:06 | |
| *** mbeierl has joined #airshipit | 13:21 | |
| *** Nishant__ has joined #airshipit | 13:31 | |
| *** sthussey has joined #airshipit | 13:33 | |
| *** michael-beaver has joined #airshipit | 13:45 | |
| openstackgerrit | Matt McEuen proposed airship/promenade master: Add optional external_ip to promjoin https://review.opendev.org/650168 | 13:46 |
| *** kranthikirang has joined #airshipit | 13:47 | |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Update links in the docs to refer to opendev https://review.opendev.org/654602 | 13:49 |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Fix chart repo urls from openstack to opendev https://review.opendev.org/654579 | 13:49 |
| *** kaspars__ has joined #airshipit | 13:52 | |
| *** DanCrank has joined #airshipit | 13:57 | |
| openstackgerrit | Drew Walters proposed airship/treasuremap master: global: Uplift helm-toolkit https://review.opendev.org/645226 | 13:58 |
| openstackgerrit | Drew Walters proposed airship/treasuremap master: airskiff: Use global Airship charts https://review.opendev.org/635218 | 13:58 |
| openstackgerrit | Kaspars Skels proposed airship/maas master: Support for MAAS URL overrides https://review.opendev.org/653853 | 14:00 |
| openstackgerrit | Merged airship/armada master: Move to kubernetes python client 9.0.0 https://review.opendev.org/639789 | 14:01 |
| openstackgerrit | Kudaka Poorna Rajesh proposed airship/promenade master: Add Liveness/Readiness probe to calico-etcd. https://review.opendev.org/645561 | 14:19 |
| openstackgerrit | Sean Eagan proposed airship/promenade master: Use apps/v1 k8s controllers and add labels https://review.opendev.org/638276 | 14:25 |
| openstackgerrit | Kudaka Poorna Rajesh proposed airship/promenade master: Add Liveness/Readiness probe to calico-etcd. https://review.opendev.org/645561 | 14:29 |
| openstackgerrit | Matthew Carter proposed airship/shipyard master: Add validation support for deployment-version document https://review.opendev.org/653847 | 14:43 |
| openstackgerrit | Merged airship/treasuremap master: Update links in the docs to refer to opendev https://review.opendev.org/654602 | 14:52 |
| *** jemangs has quit IRC | 15:09 | |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Fix chart repo urls from openstack to opendev https://review.opendev.org/654579 | 15:30 |
| *** ian-pittwood has joined #airshipit | 15:30 | |
| openstackgerrit | Drew Walters proposed airship/treasuremap master: airskiff: Use global Airship charts https://review.opendev.org/635218 | 15:35 |
| openstackgerrit | Evgeniy L proposed airship/treasuremap master: Pipeline improvements for Airship Seaworthy https://review.opendev.org/654276 | 15:47 |
| *** aojea has quit IRC | 15:52 | |
| openstackgerrit | Ian Pittwood proposed airship/spyglass master: Revise CLI documentation https://review.opendev.org/650137 | 15:54 |
| mattmceuen | t-minus two minutes till meeting, team | 15:58 |
| openstackgerrit | Ian Pittwood proposed airship/spyglass master: Implements PBR for packaging https://review.opendev.org/653480 | 15:59 |
| mattmceuen | #startmeeting airship | 16:00 |
| openstack | Meeting started Tue Apr 23 16:00:29 2019 UTC and is due to finish in 60 minutes. The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: airship)" | 16:00 | |
| mattmceuen | #topic Rollcall | 16:00 |
| openstack | The meeting name has been set to 'airship' | 16:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: airship)" | 16:00 | |
| mattmceuen | Hello, everyone - welcome | 16:00 |
| aaronsheffield | o/ | 16:00 |
| mattmceuen | https://etherpad.openstack.org/p/airship-meeting-2019-04-23 | 16:00 |
| *** AlexanderHughes has joined #airshipit | 16:00 | |
| evgenyl | Hello! | 16:00 |
| arunkant | o/ | 16:00 |
| mattmceuen | ^ agenda above | 16:01 |
| AlexanderHughes | o/ | 16:01 |
| dwalt | hi all o/ | 16:01 |
| mattmceuen | We have a full agenda today so I'll try to keep us on the tracks :) | 16:01 |
| mattmceuen | #topic No meeting next week | 16:01 |
| *** openstack changes topic to "No meeting next week (Meeting topic: airship)" | 16:01 | |
| mattmceuen | I think this is self explanatory :) | 16:01 |
| mattmceuen | due to the summit | 16:02 |
| michael-beaver | o/ GM | 16:02 |
| mattmceuen | #topic Update on gerrit migration | 16:02 |
| *** openstack changes topic to "Update on gerrit migration (Meeting topic: airship)" | 16:02 | |
| *** levmorgan has joined #airshipit | 16:02 | |
| mattmceuen | We have our shiny new opendev infrastructure up and running now thanks to the hard work of the openstack infra team | 16:02 |
| mattmceuen | To make sure everyone's aware of the changes: | 16:03 |
| mattmceuen | https://opendev.org/airship/ | 16:03 |
| sthussey | opendev infra team* | 16:03 |
| *** StaceyF has joined #airshipit | 16:03 | |
| mattmceuen | ^ this is our git repository now, and it has https redirects set up | 16:03 |
| mattmceuen | TY Scott | 16:03 |
| mattmceuen | Note that previously we'd had an openstack git repo, which mirrored out to github | 16:03 |
| mattmceuen | Now we are using the gitea open source project to host both functions | 16:04 |
| mattmceuen | So if the link above looks like github - that's intentional | 16:04 |
| mattmceuen | Notice also we are in the /airship project namespace rather than /openstack now | 16:04 |
| mattmceuen | And therefore the project names themselves have lost the 'airship-' prefix | 16:04 |
| mattmceuen | e.g. https://opendev.org/airship/promenade | 16:05 |
| mattmceuen | So that's great | 16:05 |
| mattmceuen | Similarly, our gerrit is now https://review.opendev.org/ | 16:05 |
| mattmceuen | All patchsets from the old gerrit have been migrated over | 16:05 |
| mattmceuen | dwalt points out that: gerritbot no longer posting new changes in IRC https://review.opendev.org/654470 | 16:06 |
| dwalt | woops -- forgot to remove that, sorry | 16:06 |
| dwalt | that has merged since yesterday :) | 16:06 |
| mattmceuen | However that appears to be merged so I think we're good :) thanks dwalt | 16:06 |
| dwalt | np! | 16:06 |
| mattmceuen | There are a couple more items related to the migration, but they have their own topics later in the agend | 16:06 |
| *** jemangs has joined #airshipit | 16:06 | |
| mattmceuen | any questions / thoughts on the migration? | 16:07 |
| michael-beaver | airship/in-a-bottle probably needs to be changed right? | 16:07 |
| hogepodge | yes | 16:07 |
| mattmceuen | Yep sthussey has an item for that michael-beaver | 16:08 |
| michael-beaver | Oh sorry I missed that | 16:08 |
| hogepodge | For the airship-in-a-bottle rename, the infra team has asked that you follow this procedure. https://docs.openstack.org/infra/manual/creators.html#project-renames | 16:08 |
| mattmceuen | Let's hold off on that one till later, there are two parts to it | 16:08 |
| mattmceuen | Yup I chatted with clarkb this a.m. and he got me set up with the needful steps - I'm planning to get them in this week | 16:08 |
| mattmceuen | It'll probably take a week or two to get the update made, but I think that's probably fine thanks to | 16:09 |
| mattmceuen | 1) https:// redirects | 16:09 |
| mattmceuen | 2) let's just make sure the wiki and airshipit.org point to the correct place and keep those things up to date | 16:09 |
| kaspars__ | airship-in-a-bottle I think is really close with new airsloop type to be migrated to treasuremap.. just a thought | 16:09 |
| mattmceuen | lol that's the other 1/2 of sthussey's agenda item | 16:10 |
| *** gagehugo has joined #airshipit | 16:10 | |
| mattmceuen | We have effectively moved the item - let's finish it up | 16:10 |
| mattmceuen | Do we want to keep airship-in-a-bottle as a separate project? | 16:10 |
| mattmceuen | It's two different things -- a site definition (a couple of 'em) and some easy-to-use demo deployment tooling | 16:10 |
| evgenyl | I think Dimitrious started doing some work on that, but I'm not sure if he is here. | 16:11 |
| *** jemangs has quit IRC | 16:11 | |
| dwalt | I can't think of any benefits to maintaining it as a separate project. In fact, it seems like it diverges further from treasuremap every week. | 16:11 |
| michael-beaver | ++ | 16:11 |
| kaspars__ | https://review.opendev.org/#/c/654548/ | 16:12 |
| sthussey | So we'll need to determine path forward | 16:12 |
| sthussey | That repo was originally built as an integration repo, which we still need | 16:12 |
| kaspars__ | I also think having it in the same repo would simplify things | 16:12 |
| sthussey | There is documentation there that is now likely stale | 16:13 |
| kaspars__ | I feel that all that treasuremap is is basically integration.. | 16:13 |
| mattmceuen | If we want to move the aiab deployment tooling into treasuremap, we can put it in `/tools/deployment/airship-in-a-bottle` | 16:13 |
| dwalt | Along a similar vein, moving repositories would also be a good opportunity to clean-up the multinode "gate" and make it more visible | 16:13 |
| mattmceuen | or some such | 16:13 |
| sthussey | The site definitions and tooling going to treasuremap is fine | 16:14 |
| evgenyl | We will also need to move multinode deployment. | 16:14 |
| dwalt | I like that idea mattmceuen | 16:14 |
| mattmceuen | kaspars__, are you targeting the existing multinode virtualized testing using the sloop type, or something else? | 16:14 |
| sthussey | The developer documentation can move, but that changes the intent of treasuremap | 16:14 |
| kaspars__ | sloop may be too slim for gate-multinode - so I'm not 100% sure | 16:15 |
| mattmceuen | Ok. So we have a few things we'll need to figure out | 16:15 |
| kaspars__ | definitely we can have AIAB to use sloop | 16:15 |
| kaspars__ | that would fit perfect | 16:16 |
| kaspars__ | "A sloop (from Dutch sloep, in turn from French chaloupe) is a sailing boat with a single mast and a fore-and-aft rig. A sloop has only one head-sail; if a vessel has two or more head-sails, the term cutter is used, and its mast may be set further aft than on a sloop. " | 16:16 |
| kaspars__ | for people that wonder.. | 16:16 |
| mattmceuen | 1) figure how to best migrate gate-multinode tooling/site def into treasuremap | 16:17 |
| mattmceuen | 2) do the work of moving aiab into treasuremap | 16:17 |
| mattmceuen | 3) determine what we want to do with documentation (consolidate somewhere please) | 16:17 |
| mattmceuen | 4) retire aiab project | 16:17 |
| mattmceuen | We should hold off on #4 (retire aiab project) until after it's fully migrated into treasuremap | 16:18 |
| mattmceuen | With that in mind - I think I'll still go forward with requesting the project rename back to airship/airship-in-a-bottle for the migration period | 16:18 |
| evgenyl | Yes, it will take some time to get the migration finished, so it may be a good idea to rename it. | 16:19 |
| mattmceuen | I've added another agenda item for doc consolidation | 16:19 |
| mattmceuen | Not sure if we'll be able to round it out today | 16:19 |
| mattmceuen | Cool. Are we good with that general approach, all? | 16:20 |
| sthussey | As part of some improvements on build pipelines, we'll be updating the Airship coding standard doc there | 16:20 |
| *** jemangs has joined #airshipit | 16:20 | |
| michael-beaver | I'm already working on creating multinode developer documentation, I wouldn't mind also taking on moving the multinode setup into treasuremap. Not sure if we wanted to wait until after the summit or not to do that though | 16:20 |
| mattmceuen | Good question | 16:20 |
| mattmceuen | Let's see how the week goes michael-beaver | 16:21 |
| sthussey | First step to that is merging all the in-flight CS | 16:21 |
| mattmceuen | yes good point | 16:21 |
| michael-beaver | Sorry just reconnected and that just went through | 16:21 |
| mattmceuen | also thanks for volunteering for that michael-beaver -- evgenyl maybe you guys can sort some of that out in the chat room since it'll be a couple weeks till the next team meeting | 16:21 |
| mattmceuen | Ok! Moving on: | 16:22 |
| mattmceuen | #topic Mutli-OS image support | 16:22 |
| *** openstack changes topic to "Mutli-OS image support (Meeting topic: airship)" | 16:22 | |
| mattmceuen | I think this one is yours sthussey | 16:22 |
| mattmceuen | https://review.opendev.org/643106 | 16:22 |
| sthussey | yes | 16:23 |
| sthussey | so we have a spec out there to support Multi-OS based docker images | 16:23 |
| sthussey | I propose an amendment there to support some build pipeline rationalization | 16:23 |
| *** mrhillsman is now known as openlab | 16:23 | |
| *** openlab is now known as mrhillsman | 16:23 | |
| sthussey | basically what I propose is moving all the multi-OS stuff to an airship-base image that each airship component would build from | 16:24 |
| *** mrhillsman is now known as openlab | 16:24 | |
| sthussey | airship-base would be built from your OS-base of choice and include a helper script that basically allows the Airship components to install OS packages without having any idea what OS base is beneath them | 16:24 |
| sthussey | I looked at bindep, it doesn't seem to achieve this | 16:24 |
| *** openlab is now known as mrhillsman | 16:25 | |
| sthussey | airship-base would also include steps to allow someone to customize where the image build is sourcing external dependencies such as Python packages or OS packages | 16:25 |
| mattmceuen | gotcha | 16:26 |
| mattmceuen | Sounds reasonable to me. I am not an image build SME, hogepodge or portdirect, does that sound like a good plan to you too? | 16:27 |
| mattmceuen | I know you guys have done a lot to facilitate complex image builds in loci as well | 16:27 |
| hogepodge | The more you can abstract those layers the better | 16:28 |
| mattmceuen | roman_g, arunkant, jamesgu__ -- sound good from your perspective too? | 16:28 |
| hogepodge | things always sneak in, as every packaging system treats common things differently. | 16:28 |
| openstackgerrit | Merged airship/deckhand master: CI: Update OSH relative paths for OpenDev https://review.opendev.org/654604 | 16:28 |
| arunkant | sthussey: Right now, each airship project has image build support. How's base image different from what we have in Dockerfile FROM ? | 16:28 |
| sthussey | It isn't | 16:29 |
| sthussey | but deckhand is missing that | 16:29 |
| hogepodge | httpd is a prime example, the abstraction layer needs to provide interfaces for configuring things like httpd/apache that have totally different installation and management layouts | 16:29 |
| sthussey | I'm not really worried about a general solution | 16:30 |
| sthussey | As far as I know none of Airship does any of that today | 16:30 |
| arunkant | yes and package names are different in distro . So not sure what base image will have additionally other than what we have in FROM | 16:30 |
| sthussey | the point of the helper script | 16:31 |
| sthussey | We'll define some meta-packages that airship components can request, the helper script will sort out what underlying OS packages will fulfill that | 16:31 |
| hogepodge | I guess I'm saying it's a good idea, and it works best if you have the discipline to always push those abstractions to the base rather than build in OS specific stuff in the higher level components | 16:31 |
| mattmceuen | I like this idea, but am concerned that it would hold up the multi-os work. Can we plan for a migration path toward that? | 16:32 |
| sthussey | I prefer this path to implement the multi-os work | 16:32 |
| sthussey | because now the multi-os work impacts a single image, not 8 | 16:32 |
| sthussey | If it isn't a candidate for upstream, that's fine. We can implement it locally downstream. | 16:34 |
| arunkant | Right now we have distro specific docker file (in proposed reviews) and it abstracts the distro details ( package names, expected path, permission etc) in single place. | 16:34 |
| mattmceuen | arunkant, I know work has been progressing on the multi-os implementation. How "close" do you feel that work is | 16:35 |
| arunkant | May be I am not following what value add helper script are adding.. | 16:35 |
| mattmceuen | I think a spec will help to clarify - could we get some details down in a spec sthussey? | 16:35 |
| arunkant | mattmceuen: We are waiting for spec update and approval and reviews are already following that spec recommendation | 16:36 |
| mattmceuen | I guess it would be an alternative or patch on top of the current spec | 16:36 |
| sthussey | I can do a CS on the current spec | 16:36 |
| mattmceuen | Yes arunkant - since you've already done the work there, I would propose we | 16:37 |
| mattmceuen | 1) go ahead with the current plan, with multiple dockerfiles | 16:37 |
| mattmceuen | 2) work through the details of sthussey's idea in a spec | 16:37 |
| mattmceuen | 3) refactor all multi-os support for the outcome of #2 after that | 16:37 |
| mattmceuen | Mainly, I just don't want to hold up work in progress for a new approach if we can migrate to that new approach after it's fleshed out | 16:38 |
| arunkant | sounds like a plan +1 . | 16:38 |
| mattmceuen | Are you ok with that sthussey? | 16:38 |
| sthussey | Yeah, we can do it downstream in the meantime. Move it upstream if desired. | 16:38 |
| arunkant | mattmceun: Any idea when update on spec can be made as there are some update needed based on comments | 16:39 |
| mattmceuen | Alright, sounds like a plan. Could also do it in att-comdev or some such as a "POC" for the approach as could be incorporated into airship proper | 16:40 |
| arunkant | mattmceuen: If roman_g is not available, I can make the update in spec | 16:40 |
| mattmceuen | ah right - that update | 16:40 |
| mattmceuen | roman_g are you ok with arunkant pushing an update to the patchset? | 16:41 |
| mattmceuen | It's a small change and he has it well sorted out | 16:41 |
| mattmceuen | let's keep going - roman_g may be afk. We'll confirm with him when he's back. | 16:42 |
| mattmceuen | #topic Google Season of Docs | 16:42 |
| *** openstack changes topic to "Google Season of Docs (Meeting topic: airship)" | 16:42 | |
| mattmceuen | Just an FYI that some of the ideas from the team for Google Season of Docs have been fleshed out here: https://wiki.openstack.org/wiki/Airship/2019-SoD | 16:43 |
| mattmceuen | hogepodge and I have been working the details out and Airship should be on the application list before the 3pm central deadline :) | 16:43 |
| hogepodge | I'm filling out the application for it right now | 16:44 |
| mattmceuen | nice! Thanks hogepodge, let me know if there's anything I can do to help | 16:44 |
| mattmceuen | #topic Now that we have OpenDev, do we additionally still want github mirroring? | 16:44 |
| *** openstack changes topic to "Now that we have OpenDev, do we additionally still want github mirroring? (Meeting topic: airship)" | 16:44 | |
| evgenyl | Yes, I really like using hotkeys on GitHub... | 16:44 |
| mattmceuen | So - github mirroring is now turned off, so I think our github mirrors will quickly become stale | 16:45 |
| evgenyl | Gitea does not have any https://github.com/go-gitea/gitea/issues/5796 | 16:45 |
| mattmceuen | If we really want to maintain github mirroring, I think I saw that there's a request process we can go through | 16:45 |
| mattmceuen | Me personally, I'm ok with just opendev. But understand others may care more | 16:46 |
| dwalt | I would like to see GitHub mirroring for the additional visibility | 16:46 |
| kaspars__ | for now, I've been quite OK with looking at the code there.. so no strong view from me | 16:46 |
| evgenyl | Also I like to run searches for all repos in the organization, which does not seem to be available for gitea. | 16:46 |
| mattmceuen | #action mattmceuen figure out process for requesting github mirroring | 16:46 |
| mattmceuen | There will probably be things we need to figure out, e.g. since airship is not part of the openstack namespace anymore we might need to live in a new home on github - will see | 16:47 |
| evgenyl | But we should make sure that we don't use github links in the docs :) | 16:47 |
| dwalt | ++ | 16:47 |
| mattmceuen | ++ | 16:47 |
| ian-pittwood | +1 | 16:47 |
| mattmceuen | ok, trucking along: | 16:47 |
| mattmceuen | #topic New project creation request: spyglass plugin for XLS | 16:47 |
| *** openstack changes topic to "New project creation request: spyglass plugin for XLS (Meeting topic: airship)" | 16:48 | |
| mattmceuen | AlexanderHughes I think this is yours | 16:48 |
| ian-pittwood | This one is me actually I think | 16:48 |
| mattmceuen | oops! sorry ian-pittwood | 16:48 |
| mattmceuen | Last week we'd discussed splitting out spyglass plugins into their own projects | 16:48 |
| *** gagehugo has quit IRC | 16:48 | |
| mattmceuen | But we didn't officially agree on the 1st plugin project name, and I think it would be good to nail down a convention | 16:49 |
| ian-pittwood | I'm working on separating out the plugins from Spyglass as there was a proprietary plugin that made it's way in there. I was thinking we could make a new plugin for the open source plugin | 16:49 |
| mattmceuen | airship/spyglass-plugin-tugboat <- ian's proposal | 16:49 |
| ian-pittwood | Yeah there's the spreadsheet plugin currently called Tugboat | 16:49 |
| sthussey | why would this go in an individual repo? | 16:49 |
| sthussey | You could split the plugin out of the python package w/o proliferating repos | 16:49 |
| ian-pittwood | That seemed to be the direction we decided on last week, but I am open to other ideas | 16:50 |
| *** altlogbot_1 has quit IRC | 16:50 | |
| *** altlogbot_1 has joined #airshipit | 16:50 | |
| sthussey | Okay, I probably missed that. | 16:51 |
| sthussey | this project does love to grow the repo count | 16:51 |
| mattmceuen | I'm not quite as concerned about project proliferation as I used to be, now that we have an /airship namespace. Split out plugin projects was where some of the openstack projects have landed after going back and forth | 16:51 |
| mattmceuen | But I don't have a really strong opinion either | 16:51 |
| ian-pittwood | Creating a plugin repo just sort of matched some other repos I looked at | 16:52 |
| ian-pittwood | But there were also others that kept plugins all within the same repo so either way | 16:52 |
| openstackgerrit | Kaspars Skels proposed airship/maas master: Support for MAAS URL overrides https://review.opendev.org/653853 | 16:52 |
| ian-pittwood | I would just need to adjust my course a little bit depending on what we decide is best | 16:52 |
| sthussey | I'm a mono-repo proponent which isn't popular in openstack | 16:52 |
| openstackgerrit | Lev Morgan proposed airship/pegleg master: Added cleartext option to passphrase generation https://review.opendev.org/645017 | 16:53 |
| ian-pittwood | With Spyglass, it's likely that end users would create their own plugins for data extraction | 16:53 |
| sthussey | but wasn't aware this was already litigated | 16:53 |
| ian-pittwood | That's why I thought that maybe it would be a good idea to keep them all separate | 16:53 |
| sthussey | so I'll just remove my opinion from the conversation | 16:53 |
| mattmceuen | so tugboat is an xls-based plugin, right ian - are there other generic kinds of plugins that are expected? | 16:54 |
| *** altlogbot_1 has quit IRC | 16:54 | |
| *** altlogbot_0 has joined #airshipit | 16:54 | |
| mattmceuen | In any case, we need to be able to support external plugins, whether or not the "default" plugin is bundled with spyglass project itself | 16:55 |
| ian-pittwood | I imagine more would be made down the road. Input data could come from anywhere | 16:55 |
| ian-pittwood | Exactly | 16:55 |
| mattmceuen | ian-pittwood - if you're doing the work, what approach do you prefer :) | 16:56 |
| openstackgerrit | Arijit Bose proposed airship/in-a-bottle master: [site update] update software https://review.opendev.org/655197 | 16:56 |
| ian-pittwood | I guess for now I would try separate repos then | 16:56 |
| mattmceuen | ok | 16:56 |
| mattmceuen | for airship/spyglass-plugin-tugboat | 16:56 |
| mattmceuen | My only concern is that it's not obvious that it's an xls plugin | 16:56 |
| mattmceuen | what about airship/spyglass-plugin-xls? | 16:57 |
| ian-pittwood | A name change isn't that big a deal for me | 16:57 |
| ian-pittwood | Sure, that works | 16:57 |
| mattmceuen | For an example plugin, I think it's important to make it clear what it's doing is all | 16:57 |
| mattmceuen | Cool - then we'll go forward with submitting that project creation request unless anyone objects | 16:58 |
| mattmceuen | ian-pittwood: if you can put in the request and then add me as a reviewer that would be awesome :) | 16:58 |
| ian-pittwood | Sure, thank you | 16:58 |
| mattmceuen | #topic Roundtable | 16:59 |
| hogepodge | can I jump in for the last couple of minutes for some summit questions? | 16:59 |
| *** openstack changes topic to "Roundtable (Meeting topic: airship)" | 16:59 | |
| mattmceuen | #topic hogepodge roundtable | 16:59 |
| *** openstack changes topic to "hogepodge roundtable (Meeting topic: airship)" | 16:59 | |
| mattmceuen | go for it! | 16:59 |
| hogepodge | I need to get a list of names of folks who will be at the board meeting and joint leadership meeting representing airship | 16:59 |
| mattmceuen | board meeting: Matt McEuen, Kaspars Skels, Jay Ahn | 16:59 |
| mattmceuen | are the ones I'm aware of | 17:00 |
| hogepodge | Also, if you're a non-ATT person who wants to be identified as a leader for Airship within your company I also need your name so we can identify you on your badge. | 17:00 |
| mattmceuen | Kaspars is the leader from Ericsson, Jay is the leader from SKT | 17:00 |
| *** gagehugo has joined #airshipit | 17:00 | |
| mattmceuen | Joint Leadership Meeting - sorry, when is that one? Diff from the board update? | 17:01 |
| mattmceuen | we're out of time but can continue here in the room if that's ok hogepodge | 17:01 |
| jamesgu__ | hogepodge... what is the joint ledership meeting? From suse side, Dirk Mueller and/or myself can attend? | 17:01 |
| mattmceuen | Sorry we ran out, all - will copy the remaining agenda items to the next agenda | 17:02 |
| mattmceuen | #endmeeting | 17:02 |
| *** openstack changes topic to "airshipit.org || General Review Dashboard: https://review.openstack.org/#/q/project:%255Eopenstack/airship.*+status:open,n,z" | 17:02 | |
| openstack | Meeting ended Tue Apr 23 17:02:35 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:02 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-04-23-16.00.html | 17:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-04-23-16.00.txt | 17:02 |
| mattmceuen | have a good day! | 17:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-04-23-16.00.log.html | 17:02 |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Support latest MAAS in Airsloop https://review.opendev.org/653857 | 17:02 |
| evgenyl | ++ I also would like to learn more on what will be discussed on this meeting. | 17:03 |
| hogepodge | jamesgu__: the board meeting on Sunday | 17:03 |
| *** AlexanderHughes has quit IRC | 17:03 | |
| hogepodge | jamesgu__: https://wiki.openstack.org/wiki/Governance/Foundation/28April2019BoardMeeting | 17:04 |
| hogepodge | evgenyl: ^^ | 17:04 |
| hogepodge | Project updates mostly | 17:05 |
| evgenyl | hogepodge: I see, thanks for the link. | 17:08 |
| jamesgu__ | hogepodge: aha... I thought was airship specific meeting. Neither Dirk or myself will be in Denver on Sunday. Thanks for the link, would be interested to read the meeting notes. | 17:09 |
| evgenyl | Sooo, michael-beaver we have a failure for our baremetal pipeline, here Armada says that everything is fine http://paste.openstack.org/show/749658/ and Shipyard fails before that with http://paste.openstack.org/show/749645/ a complete set of logs is available here https://jenkins.atlantafoundry.com/job/airship-seaworthy/263/artifact/debug-cab23-r720-11.tgz | 17:11 |
| evgenyl | michael-beaver: Any ideas on what could go wrong? And if it is related to the recent changes in Armada/Shipyard? | 17:11 |
| openstackgerrit | Ian Pittwood proposed airship/spyglass master: Refactors CLI to match style of Airship CLIs https://review.opendev.org/649452 | 17:12 |
| *** MattCoachCarter has joined #airshipit | 17:12 | |
| kaspars__ | to add: new deploy is ongoing right now and armada apply is about to start | 17:13 |
| kaspars__ | https://jenkins.atlantafoundry.com/job/airship-seaworthy/ | 17:13 |
| michael-beaver | So just to preface, by default the Armada lock expires after 10 minutes and when trying to run Armada will error out after 1 minute of attempting to acquire a lock, although Armada always tries to remove the lock when it is done or errors out. | 17:16 |
| michael-beaver | https://www.irccloud.com/pastebin/5L7SJQs4/ | 17:17 |
| openstackgerrit | Sirajudeen proposed airship/in-a-bottle master: [site update] Adding gate set up https://review.opendev.org/646796 | 17:17 |
| michael-beaver | Haha woops, messed up the formatting in IRCCloud there, but you get the idea | 17:18 |
| openstackgerrit | Sirajudeen proposed airship/in-a-bottle master: [WIP] - set passphrase and salt to pegleg https://review.opendev.org/651659 | 17:20 |
| *** StaceyF has quit IRC | 17:20 | |
| kaspars__ | michael-beaver: I think here is looks like the way Shipyard interaction happens with Armada might be the issue.. | 17:23 |
| michael-beaver | Yeah, I am wondering if shipyard is kicking off another Armada action before a previous one is completed | 17:24 |
| evgenyl | michael-beaver: So, in the logs I sent you, is it the same lock that Shipyard and Aramda try to acquire? (it's hard to tell because there is no uuid/name of the lock in the logs) | 17:24 |
| michael-beaver | I don't think Shipyard acquires a lock | 17:25 |
| openstackgerrit | Arun Kant proposed airship/deckhand master: Adding opensuse image build for deckhand https://review.opendev.org/638301 | 17:26 |
| kaspars__ | https://www.irccloud.com/pastebin/MrZxwYo1/ | 17:26 |
| michael-beaver | But yes it is probably referring to the same lock | 17:26 |
| kaspars__ | here is previous log from Evgenyi | 17:26 |
| kaspars__ | this might just be return code from Armada execution | 17:26 |
| kaspars__ | (btw - osh-infra is now ongoing in armada if anyone follows logs..) | 17:26 |
| michael-beaver | Yes, when Armada is unable to acquire a lock it raises an Exception instead of staying stuck forever, and when Shipyard sees this it is returning the response from Armada | 17:28 |
| kaspars__ | wonder why Armada wouldn't wait for 10min | 17:29 |
| kaspars__ | if that's kind a the timeout | 17:29 |
| kaspars__ | but then.. wonder why the lock is in place.. so far deploy is fine | 17:29 |
| michael-beaver | We can adjust the timeout to be the same length as the expiry time if that is more useful, but I think the idea is the lock could be there for quite a while | 17:32 |
| michael-beaver | And the lock is necessary because Tiller is now in the Armada pod, so there are multiple Tillers that can be interacting with the cluster at the same time | 17:32 |
| michael-beaver | We can adjust the timeout to be the same length as the expiry time if that is more useful, but I think the idea is the lock could be there for quite a while | 17:32 |
| openstackgerrit | Arun Kant proposed airship/deckhand master: Adding opensuse image build for deckhand https://review.opendev.org/638301 | 17:34 |
| mattmceuen | michael-beaver would `tiller.enabled: true` impact any of this behavior? | 17:39 |
| michael-beaver | if tiller.enabled is set to true then Armada will put Tiller into the pod, otherwise it will just try to find the Tiller pod normally | 17:40 |
| *** Talion has joined #airshipit | 17:41 | |
| openstackgerrit | Ian Pittwood proposed airship/spyglass master: Resolves Bandit 104 errors https://review.opendev.org/651807 | 17:42 |
| michael-beaver | The locking mechanism will be in place whether or not Tiller is in the pod though | 17:43 |
| evgenyl | I'm still struggling to understand what is going on between Shipyard and Armada, Armada finished the deployment at `23:25:17.175`, shipyard had started `armada_get_releases` at `23:25:03,922` and it failed because deployment was not done at this time and lock was not available. So I have several questions, why Shipyard thinks that Armada finished and started the next task? Why does `get_release` requires a lock? Should | 17:51 |
| evgenyl | there be some retries for `get_release`? | 17:51 |
| michael-beaver | No idea on the first question | 18:04 |
| michael-beaver | For the second, I'm trying to find why it is using a lock for `get_release`, but I haven't been able to find it right away | 18:04 |
| openstackgerrit | Sean Eagan proposed airship/armada master: [v2 docs] Move `upgrade.no_hooks` to `upgrade.options.no_hooks` https://review.opendev.org/655226 | 18:05 |
| openstackgerrit | Merged airship/spyglass master: Remove unnecessary exclusion https://review.opendev.org/652203 | 18:17 |
| *** lemko has quit IRC | 18:17 | |
| *** mbologna has quit IRC | 18:41 | |
| kaspars__ | strangely it still seem to continue armada apply and finish the deploy.. | 18:44 |
| *** mbologna has joined #airshipit | 18:45 | |
| openstackgerrit | Merged airship/treasuremap master: Pipeline improvements for Airship Seaworthy https://review.opendev.org/654276 | 18:48 |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Do not rerun promjoin on every reboot https://review.opendev.org/642571 | 18:48 |
| *** happyhemant has quit IRC | 18:48 | |
| *** bh526r has quit IRC | 19:01 | |
| kaspars__ | https://www.irccloud.com/pastebin/5dJzcEyz/ | 19:02 |
| kaspars__ | so it looks like exactly at 1h | 19:02 |
| kaspars__ | armada API closed connection | 19:02 |
| kaspars__ | could it be that Armada API itself has 1h timeout? | 19:02 |
| kaspars__ | after which shipyard starts 2nd attempt - and is failing because it will never get the lock anymore.. | 19:03 |
| sthussey | Probably the uWSGI read timeout | 19:03 |
| kaspars__ | let me check if it fits 1h - thanks | 19:03 |
| kaspars__ | https://www.irccloud.com/pastebin/Vf1ZviqG/ | 19:04 |
| kaspars__ | these seem commented out but seem to fit 1h | 19:04 |
| kaspars__ | https://opendev.org/airship/armada/src/branch/master/charts/armada/values.yaml#L222 | 19:05 |
| -openstackstatus- NOTICE: the zuul scheduler is being restarted now in order to address a memory utilization problem; changes under test will be reenqueued automatically | 19:07 | |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Uplift tiller/help to 2.13.1 and Airship/OSH charts https://review.opendev.org/653963 | 19:12 |
| kaspars__ | I think this should do https://review.opendev.org/#/c/653963/11/global/software/charts/ucp/armada/armada.yaml | 19:12 |
| michael-beaver | What's the reasoning behind disabling the Tiller side-car? | 19:13 |
| kaspars__ | I can put it back if it's good to use | 19:14 |
| kaspars__ | something Matt was commenting as useful | 19:14 |
| kaspars__ | in fact let's do it as I didn't see it having any impact.. it's really the timeout | 19:15 |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Uplift tiller/help to 2.13.1 and Airship/OSH charts https://review.opendev.org/653963 | 19:15 |
| michael-beaver | Yeah that was my only point is that it looks like it is just the timeout causing issues here | 19:15 |
| openstackgerrit | Drew Walters proposed airship/armada master: CI: Update jobs for OpenDev https://review.opendev.org/655236 | 19:29 |
| *** randeepjalli has joined #airshipit | 19:37 | |
| *** canderson_ has joined #airshipit | 19:40 | |
| openstackgerrit | Drew Walters proposed airship/treasuremap master: Uplift tiller/help to 2.13.1 and Airship/OSH charts https://review.opendev.org/653963 | 19:41 |
| openstackgerrit | Drew Walters proposed airship/treasuremap master: airskiff: Use global Airship charts https://review.opendev.org/635218 | 19:41 |
| *** levmorgan has quit IRC | 19:41 | |
| openstackgerrit | Drew Walters proposed airship/treasuremap master: Uplift tiller/help to 2.13.1 and Airship/OSH charts https://review.opendev.org/653963 | 19:43 |
| openstackgerrit | Ian Pittwood proposed airship/spyglass master: Refactors CLI to match style of Airship CLIs https://review.opendev.org/649452 | 19:51 |
| *** randeepjalli has quit IRC | 20:25 | |
| *** jemangs has quit IRC | 20:30 | |
| *** jemangs has joined #airshipit | 20:30 | |
| *** jemangs has quit IRC | 20:42 | |
| *** jemangs has joined #airshipit | 20:42 | |
| openstackgerrit | Kaspars Skels proposed airship/maas master: Support for MAAS URL overrides https://review.opendev.org/653853 | 20:42 |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Support latest MAAS in Airsloop https://review.opendev.org/653857 | 20:44 |
| openstackgerrit | Merged airship/in-a-bottle master: Update links in the docs to refer to opendev https://review.opendev.org/654605 | 20:46 |
| *** jemangs has quit IRC | 20:47 | |
| *** jemangs has joined #airshipit | 20:51 | |
| openstackgerrit | Dimitrios Markou proposed airship/treasuremap master: [WIP] Migrate single node manifests to treasuremap https://review.opendev.org/654548 | 21:12 |
| openstackgerrit | Dimitrios Markou proposed airship/treasuremap master: [WIP] Migrate single node manifests to treasuremap https://review.opendev.org/654548 | 21:15 |
| *** Talion has quit IRC | 21:20 | |
| *** MattCoachCarter has quit IRC | 21:22 | |
| openstackgerrit | Dan Crank proposed airship/shipyard master: [DEFECT-4328] Don't log OS_PASSWORD in auth.log https://review.opendev.org/655248 | 21:22 |
| kranthikirang | Kaspars__: I redeployed today; | 21:35 |
| kranthikirang | cluster: | 21:35 |
| kranthikirang | id: 29d8953d-0bb6-4ba1-a48a-f9be1c0937a9 | 21:35 |
| kranthikirang | health: HEALTH_OK | 21:35 |
| kranthikirang | Everything looks good; What is the name of the public flat network I should create? Is it public or external? | 21:36 |
| kaspars__ | hey congrats!! here is a sample heat template you may use to create the floating network | 21:39 |
| kaspars__ | https://opendev.org/airship/treasuremap/src/branch/master/tools/files/heat-public-net-deployment.yaml | 21:39 |
| kaspars__ | the neutron is configured towards the bond0 interface - and the creation of network then would need to use a VLAN for your public network | 21:39 |
| kaspars__ | the network can be really any provider network on a VLAN.. as it will be floating IP network you want to make sure you can access those IPs/network from your laptop or whereever you would access the VMs.. | 21:40 |
| kaspars__ | oh network name itself can be anything - our pipeline is using 'public' but it doesn't really matter | 21:41 |
| kranthikirang | I thought for flat network using Physical Network name matters coming to OVS | 21:42 |
| kranthikirang | Also I directly use public IPs to VM instead of private and then floating | 21:43 |
| kranthikirang | I will create with public name and see | 21:43 |
| kranthikirang | kaspars__: One more question, public network VLAN isn't part of bond0 right? | 21:44 |
| kaspars__ | oh I think you are right - the physical network keyword that is mapped to bond0 I think is 'external' - have a try | 21:45 |
| kaspars__ | + tools/openstack stack create --wait --parameter network_name=public --parameter physical_network_name=public --parameter physical_network_vlan=27 --parameter subnet_name=public-subnet --parameter subnet_cidr=10.23.27.0/24 --parameter subnet_gateway=10.23.27.1 --parameter subnet_pool_start=10.23.27.11 --parameter subnet_pool_end=10.23.27.99 -t /target/tools/files/heat-public-net-deployment.yaml heat-public-net-deployment | 21:45 |
| kaspars__ | the parameter that is not overriden is the physical network 'external' if you see the template - so yes - you need to use it | 21:46 |
| kaspars__ | but the network name itself can be anything | 21:46 |
| kaspars__ | it though not a flat network as usually for br-ex | 21:46 |
| kaspars__ | it is using bond0 at the end if you see common-addresses.yaml (although it's cofnigureable if needed) | 21:46 |
| kranthikirang | OK; I don't see public in specs; So will use external name | 21:47 |
| kranthikirang | yeah, | 21:47 |
| kaspars__ | so you will need to use VLAN enabled provider network to point out which VLAN on your bond0 at the end will be used for the network | 21:47 |
| kranthikirang | it using bond0; but public VLAN isn't part of bond0 as per documentation; | 21:47 |
| kaspars__ | I think we haven't documented aything for public/floating network.. | 21:48 |
| kranthikirang | Sorry, I didn't understand how this is going to work; | 21:48 |
| kaspars__ | some gap we need to fill | 21:48 |
| kaspars__ | ok - so the way reference site is setup and what would also be my recommendation is following | 21:48 |
| kranthikirang | so, currently my OVS using bond0; | 21:48 |
| kaspars__ | yes | 21:48 |
| kranthikirang | OK; What should I do in switches in order to make external access | 21:49 |
| kaspars__ | so what you want is to configure your switch to have additional VLAN - e.g. | 21:49 |
| kaspars__ | that will contain a subnet of IPs that will be assigned to your VMs | 21:49 |
| kaspars__ | these are meant for external access to VMs.. | 21:49 |
| kaspars__ | it can be simply regular routed CIDR/VLAN within your lab - as long as you can access them | 21:50 |
| kranthikirang | OK; | 21:50 |
| kaspars__ | you will configure additional VLAN much like for calico/oam, etc, etc | 21:50 |
| kaspars__ | then you need to match this VLAN on the openstack/neutron side | 21:50 |
| kaspars__ | if you look at the heat tempaltes - you will see an example there | 21:51 |
| kranthikirang | I have a VLAN on bond0 which is unused at switches; I gave a range 41-45 | 21:51 |
| kranthikirang | oh; you create network type vlan; not flat; | 21:51 |
| kaspars__ | exactly | 21:51 |
| kaspars__ | this gives you more flexibility, etc | 21:52 |
| kaspars__ | flat usually resolves directly to a network interface, etc | 21:52 |
| kaspars__ | https://opendev.org/airship/treasuremap/src/branch/master/site/airship-seaworthy/networks/common-addresses.yaml#L150 | 21:52 |
| kranthikirang | I used to do following earlier in another setup; Deploy nodes with MAAS; Deploy k8s with kubespray and then deploy openstack-helm manually or scripts, use flat network by adding a vlan interface to bond0 and giving that to ovs-bridge; Also ceph-osd failure domain as osd | 21:53 |
| kaspars__ | this is where it is configured - so technically you may use dedicated interface, etc if you wanted to | 21:53 |
| kaspars__ | yes - that is similar way - only diff you don't need to create that interfaces on your hsot server | 21:53 |
| kaspars__ | just use VLAN directly from neutron/openstack | 21:53 |
| kranthikirang | OK; cool; Never tried that; Will do that now | 21:53 |
| kaspars__ | this allows to create multiple provider network if you need to, etc | 21:54 |
| kaspars__ | as you can configure multiple VLANs, and multiple provider networks | 21:54 |
| kaspars__ | on the same bond0 | 21:54 |
| kranthikirang | oh, now I get it | 21:55 |
| kranthikirang | Its been a long time using neutron :) | 21:55 |
| kaspars__ | yeah - in your old way you could configure something like thig "external_iface: 'bond0.42'" | 21:55 |
| kranthikirang | thanks for the help; I will update the status | 21:56 |
| kaspars__ | and then create local config | 21:56 |
| kranthikirang | yeah, I used to do the same | 21:56 |
| kaspars__ | but I think this is more clean/flexible way | 21:56 |
| kranthikirang | yeah, network as flat | 21:56 |
| kranthikirang | make sense | 21:56 |
| kaspars__ | so I would still recommend using bond0 | 21:56 |
| kaspars__ | and VLAN in neutron | 21:56 |
| kranthikirang | yeah, Will do | 21:56 |
| kranthikirang | Only thing is I need for make that VLAN to spine and then to my gw device | 21:56 |
| *** jemangs has quit IRC | 22:00 | |
| *** jemangs has joined #airshipit | 22:00 | |
| *** jemangs has quit IRC | 22:05 | |
| kaspars__ | face | 22:08 |
| kaspars__ | f | 22:17 |
| openstackgerrit | Merged airship/treasuremap master: Fix chart repo urls from openstack to opendev https://review.opendev.org/654579 | 22:17 |
| openstackgerrit | Lev Morgan proposed airship/spyglass master: [WIP] NARAD Plugin for Spyglass https://review.opendev.org/624052 | 22:21 |
| *** DanCrank has quit IRC | 22:21 | |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Uplift tiller/help to 2.13.1 and Airship/OSH charts https://review.opendev.org/653963 | 22:22 |
| *** jemangs has joined #airshipit | 22:22 | |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Do not rerun promjoin on every reboot https://review.opendev.org/642571 | 22:28 |
| *** kranthikirang has quit IRC | 22:56 | |
| *** jemangs has quit IRC | 22:58 | |
| *** jemangs has joined #airshipit | 22:59 | |
| *** jemangs has quit IRC | 23:02 | |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Uplift tiller/help to 2.13.1 and Airship/OSH charts https://review.opendev.org/653963 | 23:06 |
| openstackgerrit | Kaspars Skels proposed airship/treasuremap master: Support latest MAAS in Airsloop https://review.opendev.org/653857 | 23:06 |
| *** aaronsheffield has quit IRC | 23:11 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!