opendevreview | Jacob Anders proposed openstack/ironic master: Prevent MissingAttribute error when supportedApplyTime missing https://review.opendev.org/c/openstack/ironic/+/885739 | 02:41 |
---|---|---|
janders | ^ reported tested this fix for me today, seems to work (in conjunction with BIOS/BMC firmware upgrade that fixes another missing attribute problem), reviews welcome (quick/easy review) | 02:42 |
janders | s/reported/reporter | 02:42 |
TheJulia | janders: reviewed, you'll want to take a look | 03:06 |
TheJulia | since sushy can return the property as None afaik | 03:06 |
janders | thank you Julia, will look into this more | 03:07 |
janders | thank you TheJulia, I somehow got the regression from the first revision back in (!?!). Second revision was good and I broke it again somehow while adding the release note. Still no idea how but I clearly did! | 03:10 |
janders | fixing it now | 03:10 |
TheJulia | always pull the latest change down :) | 03:11 |
TheJulia | anyway, going back to the evening | 03:11 |
TheJulia | Have a wonderful day | 03:11 |
janders | thank you TheJulia, have a great evening! | 03:11 |
opendevreview | Jacob Anders proposed openstack/ironic master: Prevent MissingAttribute error when supportedApplyTime missing https://review.opendev.org/c/openstack/ironic/+/885739 | 03:16 |
opendevreview | Jacob Anders proposed openstack/ironic master: Prevent MissingAttribute error when supportedApplyTime missing https://review.opendev.org/c/openstack/ironic/+/885739 | 03:18 |
opendevreview | Takashi Kajinami proposed openstack/ironic master: Fix missing oslo.versionedobjects library option https://review.opendev.org/c/openstack/ironic/+/890715 | 06:05 |
rpittau | good morning ironic! o/ | 06:39 |
kubajj | Good morning rpittau and everyone! o/ | 07:25 |
rpittau | hey kubajj :) | 07:25 |
opendevreview | wangjiaqi proposed openstack/ironic-tempest-plugin master: Cleanup py27 support https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/890732 | 07:42 |
opendevreview | Merged openstack/ironic master: Prevent MissingAttribute error when supportedApplyTime missing https://review.opendev.org/c/openstack/ironic/+/885739 | 07:59 |
iurygregory | morning ironic | 11:45 |
kubajj | good morning iurygregory | 11:53 |
iurygregory | kubajj, o/ | 11:56 |
dking | Good morning. | 12:17 |
iurygregory | morning dking o/ | 12:18 |
rpittau | iurygregory dking o/ | 12:23 |
rpittau | anyone free for a quick approval? :) https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/887754 | 12:23 |
iurygregory | let me look | 12:23 |
opendevreview | Riccardo Pittau proposed openstack/bifrost master: Collect lshw output in json format https://review.opendev.org/c/openstack/bifrost/+/890408 | 12:24 |
rpittau | bifrost cs9 jbos hsould be fixed, or well at least the new systemd pkg is available | 12:25 |
dking | I know this is off topic, but as I'm having trouble finding much help in other places, would anybody here be familiar with pynetbox having a problem with https_proxy? When I try to set mine (in my collector), I get some kind of SSL recursion. | 12:25 |
iurygregory | I'm not =( sorry | 12:30 |
opendevreview | Merged openstack/ironic master: Fix missing oslo.versionedobjects library option https://review.opendev.org/c/openstack/ironic/+/890715 | 12:57 |
opendevreview | Maksim Malchuk proposed openstack/bifrost stable/2023.1: Remove extra symbols accidentally added https://review.opendev.org/c/openstack/bifrost/+/890789 | 13:06 |
JayF | dking: DM me the broken output and some basic details and I'll give it a cursory look. I don't have a lot of time to look at it, but maybe I can point you in a direction | 13:29 |
opendevreview | Merged openstack/ironic-python-agent-builder master: Build tinyipa with tinycore 14.x https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/887754 | 14:14 |
opendevreview | Verification of a change to openstack/bifrost master failed: Create the log file for the disk-image-create command https://review.opendev.org/c/openstack/bifrost/+/822895 | 15:11 |
rpittau | bifrost CI still not happy, probably the new systemd pkg is not in the repo yet | 15:38 |
JayF | CentOS Stream 9 -- is it infra mirrored? | 15:43 |
rpittau | I believe it is | 15:54 |
JayF | if we know it's in upstream mirror but not in infra mirror, I wonder if fungi or one of the infra folks can help us close that gap | 15:59 |
fungi | https://static.opendev.org/mirror/centos-stream/timestamp.txt says last mirror update was about 4 hours ago | 16:02 |
fungi | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-stream-mirror-update#L44 says we're currently pulling updates from mirror.rackspace.com | 16:05 |
JayF | that can be ... significantly out of date | 16:05 |
JayF | unless they've improved management of it in the last few years | 16:05 |
rpittau | fungi: thanks, what's the link to the repo content? | 16:05 |
JayF | When working there; the mirrors could be up to 2-3 days old | 16:06 |
fungi | every miror we've ever tried can be severely out of date. several times a year, whoever we're mirroring from stops updating and somebody proposes a change to another one | 16:06 |
JayF | makes sense | 16:07 |
fungi | rpittau: give me a few, it's lunch time here so i'm trying to type one-handed between bites of leftovers | 16:07 |
fungi | if somebody can clarify what i'm looking for, that might help | 16:07 |
rpittau | fungi: ah no worries! no rush :) | 16:08 |
rpittau | fungi: we'd like to see if systemd package has been updated for cs9 | 16:08 |
JayF | do you know the fixed version rpittau ? | 16:09 |
fungi | how do i know if the systemd package i find is an updated one? | 16:09 |
fungi | make that packageS. i see quite a few systemd-252-*.rpm packages at https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ | 16:09 |
rpittau | pkg with fix is systemd-252-17 | 16:10 |
rpittau | fungi: thanks again, it's not there yet | 16:10 |
fungi | highest i see there is systemd-252-16.el9.x86_64.rpm | 16:10 |
rpittau | yeah | 16:10 |
rpittau | that's the broken one | 16:10 |
fungi | do other mirrors have a newer version? how long as the fixed package been in circulation? | 16:10 |
fungi | any idea how out of date this makes the rackspace mirror? | 16:11 |
fungi | if it's way stale, we can try switching to pull from somewhere else | 16:11 |
JayF | fixed package landed today | 16:11 |
rpittau | fungi: I'm not sure the pkg is already out yet, I believe they started syncing it today | 16:11 |
fungi | okay, yeah i wouldn't expect anything fast based on prior experiences with lack of urgency around stream | 16:12 |
rpittau | yep | 16:14 |
JayF | rpittau: gonna be honest, this experience makes me wish we were testing against rocky :/ | 16:14 |
JayF | I keep hearing people say that stream is stable, but like, if this bug landed in a point-release that an enterprise ran ... all hell would break loose :( | 16:15 |
rpittau | JayF: probably worth discussing the "rocky" testing in one of the next meetigins or even at PTG | 16:16 |
rpittau | I'm also very disappointed as it's not the first this s...tuff happens | 16:17 |
rpittau | anyway, time to rest the eyes, see ya tomorrow! o/ | 16:20 |
JayF | o/ | 16:20 |
fungi | we do have current rockylinux-9 images available for jobs to use, if you want to try it out | 16:26 |
opendevreview | Mahnoor Asghar proposed openstack/ironic master: [WIP]: Add non-default inspection hooks (1-4 of 11) https://review.opendev.org/c/openstack/ironic/+/890817 | 17:17 |
opendevreview | Jakub Jelinek proposed openstack/ironic master: WIP: Introduce default kernel/ramdisks by arch https://review.opendev.org/c/openstack/ironic/+/890819 | 17:47 |
JayF | I'm going to chuck something at a wall :( the devstack I thought was setup and sane | 20:17 |
JayF | I took for granted that nova was setup in it. It's not. | 20:17 |
TheJulia | doh | 20:18 |
JayF | I'm outsized levels of upset | 20:18 |
JayF | because I've been trying to get a working devstack for literally a week+ at this point | 20:18 |
jelabarre-rh | yes, online here | 20:19 |
JayF | I've been using exclusively bifrost dev environments for so long :/ | 20:19 |
TheJulia | jelabarre-rh: https://github.com/openstack/ironic-tempest-plugin/blob/master/ironic_tempest_plugin/tests/scenario/ironic_standalone/test_ramdisk_iso.py <-- to answer your question | 20:19 |
JayF | welcome jelabarre-rh | 20:19 |
JayF | I wonder where you work ;) | 20:19 |
jelabarre-rh | added the extension since I've had a similar nick on various IRC systems | 20:20 |
JayF | oh I'm just making jokes, don't sweat it | 20:20 |
JayF | welcome o/ | 20:20 |
jelabarre-rh | I think in the late 90's I was using Gary7 or something like that (Star Trek reference) | 20:21 |
JayF | I was unfamiliar with that reference, but I'm more of a DS9 guy (and grew up on TNG) :D | 20:22 |
jelabarre-rh | 'Assignment Earth' | 20:24 |
TheJulia | .... we need more Supervisors of the Those Old Scientists vain on Earth. | 20:24 |
TheJulia | ... and Picard too... | 20:24 |
* TheJulia swears a UFP flag did not suddenly appear | 20:25 | |
* JayF storms the vulcan ship and steals all the tech | 20:26 | |
* TheJulia fires up etsy to look for a Terran Empire flag | 20:26 | |
JayF | real talk: most underrated show and episodes in all of trek: Alternate universe episodes in ST: Enterprise | 20:27 |
TheJulia | +100000000000 | 20:27 |
TheJulia | Interestingly... there is a terran emprie flag on amazon but not etsy | 20:31 |
* JayF tried hot swapping his local.conf and seeing if it gets him a nova | 20:31 | |
JayF | looks promising so far | 20:31 |
JayF | Our devstack docs are not great, but I've failed at coming up with better methods | 20:32 |
JayF | I wanna find a way, for testing Nova<>Ironic interactions, to configure devstack to create a bunch of fake ironic nodes and register them in ironic | 20:32 |
JayF | basically the default tempest config, just with no actual hydrated nodes behind the API of Ironic | 20:32 |
JayF | that would be sufficient to test sharding and to test my lessee change | 20:33 |
JayF | oh, btw TheJulia: https://review.opendev.org/c/openstack/ironic/+/888277/2#message-1313fccebe0052159ed30ab6fee0cdc80449f8ad | 20:33 |
JayF | sharding passed our ci | 20:33 |
TheJulia | it is possible, but the stock example devstack config might not be the palce to start with that | 20:33 |
TheJulia | \o/ | 20:34 |
JayF | once I get this damn thing working again, I'm going to load up sharding and lessee and play for a bit | 20:34 |
TheJulia | okay | 20:34 |
JayF | my only real goal this week is to get lesse change in nova mergable and sharding fully tested | 20:34 |
JayF | I do think there's value in you still looking at sharding testing if you can spare time and get a stack up for something else and can jfdi | 20:34 |
JayF | just because this is a big change and more testing is better | 20:34 |
TheJulia | Okay, hit me up tomorrow morning, so I can do so with a fresh mind | 20:35 |
TheJulia | I'm close on service steps, fwiw | 20:35 |
JayF | nice | 20:35 |
JayF | client for that is my backup plan if I get frustrated doing deeper work lol | 20:35 |
JayF | ++lib/neutron_plugins/services/l3:create_neutron_initial_network:196 oscwrap --os-cloud devstack --os-region RegionOne network create private -f value -c id | 20:36 |
JayF | Error while executing command: HttpException: 503, Unable to create the network. No tenant network is available for allocation. | 20:36 |
JayF | this is what happens EVERY TIME | 20:36 |
JayF | when I use the ironic-tempest-plugin config from https://docs.openstack.org/ironic/latest/contributor/dev-quickstart.html | 20:36 |
TheJulia | hmmmm | 20:37 |
TheJulia | ummmmm | 20:37 |
JayF | dbcounter has "mysql server has gone away" errors in proximity to that in the logs | 20:38 |
JayF | about 10 seconds later though | 20:38 |
TheJulia | disable dbcounter | 20:38 |
JayF | you know the devstack slug for that off the top? | 20:38 |
TheJulia | it is a devstack_service entry afaik | 20:39 |
JayF | dstat? | 20:40 |
JayF | looking at https://github.com/openstack/ironic/blob/master/zuul.d/ironic-jobs.yaml#L81 | 20:40 |
TheJulia | maybe | 20:40 |
JayF | nah I don't think that's it looking at the bug | 20:40 |
JayF | https://github.com/openstack/ironic/commit/8b98dfafd848bc1f1bd31c073bcdd37606081045 | 20:41 |
JayF | we only disable for grenade | 20:41 |
TheJulia | nah, not dstat | 20:41 |
TheJulia | yeah, that is the way that you just found | 20:41 |
JayF | admittedly, I have little faith that is going to make a difference | 20:41 |
JayF | primarily because I'm on beefier gear than our gates right now | 20:41 |
JayF | and so it's hard for me to believe dbcounter would break locally but work in gate | 20:42 |
TheJulia | i think your issue might be that your defining local network config and devstack then might be taking a different path on the create networks code in the neutron plugin | 20:43 |
TheJulia | well, your networking issue | 20:44 |
JayF | that's identical to what I get with that local.conf on a clean install | 20:44 |
JayF | and the network this is on is 172.16.254.0/24 so it shouldn't conflict with anything devstacky | 20:44 |
JayF | (I pick that for exactly that reason) | 20:44 |
TheJulia | the doc has network_gateway, fixed_network_size, etc.. we have none of that by default in our job definitions unless it is in devstack-base | 20:44 |
JayF | I've reproduced this error with both those set and unset | 20:45 |
JayF | I've literally been fighting this for almost the whole week you were gone | 20:45 |
JayF | which is why I feel like every door is already closed on troubleshooting it | 20:45 |
TheJulia | interesting | 20:45 |
JayF | except "lol you need OBSCURE_BS=lolcats" | 20:45 |
JayF | or something like that | 20:45 |
JayF | I'm hunting a known unknown | 20:45 |
JayF | I started out with stealing LOCALRC from the gate and trying to thin it out and use it | 20:46 |
JayF | that didn't work | 20:46 |
JayF | so I went to the config | 20:46 |
JayF | that didn't work, except for apparently the last time when I copy+pasted the !nova version which did work | 20:46 |
TheJulia | do you get that error wen ironic is trying to run setup? | 20:46 |
TheJulia | NEUTRON_CREATE_INITIAL_NETWORKS comes to mind from neutron rooted at https://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L1389C9-L1389C40 | 20:47 |
TheJulia | but only so it doesn't do it again | 20:47 |
JayF | let me see, I didn't save full logs am waiting for reproduction (or not) to happen | 20:48 |
JayF | it's been hung a while on init_neutron, I think I may have finally worn this machine out lol | 20:49 |
JayF | oh it broke free, woo | 20:51 |
JayF | yeah it's not far enough to break yet, I think | 20:51 |
JayF | okay, lib/ironic ran, now neutron is running | 20:55 |
JayF | and now it breaks | 20:55 |
TheJulia | did it restart neutron? | 20:57 |
JayF | https://home.jvf.cc/~jay/devstacklog.txt | 20:58 |
JayF | create_neutron_initial_network got called in stack.sh | 20:58 |
JayF | in that method is where it goes boom | 20:58 |
JayF | do I need to set NEUTRON_CREATE_INITIAL_NETWORKS=False ? | 20:59 |
JayF | that can't be right | 20:59 |
TheJulia | eh, it has to be done once | 20:59 |
TheJulia | but it can't be done twice | 20:59 |
JayF | but it is doing twice | 20:59 |
JayF | but only for me, and not for gate | 21:00 |
JayF | seemingly | 21:00 |
JayF | I'm going to rebuild this from scratch and see if it makes a diff | 21:03 |
JayF | well, maybe just set that and I get a working one | 21:03 |
JayF | I'm going to go that route | 21:03 |
JayF | hmm | 21:04 |
JayF | oh, I think I have it, maybe? | 21:04 |
TheJulia | so no IP_VERSION... in your log | 21:04 |
JayF | ++lib/neutron_plugins/services/l3:create_neutron_initial_network:196 oscwrap --os-cloud devstack --os-region RegionOne network create private -f value -c id | 21:04 |
JayF | note that's coming from the l3 plugin | 21:04 |
JayF | which would not be skipped on second run, because that's a separate flag | 21:05 |
JayF | nope, that's wrong | 21:05 |
JayF | they are managed by the same flag | 21:05 |
JayF | and l3 is the default | 21:05 |
JayF | so this is just default | 21:05 |
JayF | TheJulia: what does that mean? | 21:05 |
JayF | TheJulia: this is a v4-only box | 21:05 |
JayF | TheJulia: does that require a specific config? | 21:05 |
TheJulia | search your log, no IP_VERSION, we should have a variable there | 21:06 |
TheJulia | I'd explicitly set it to IP_VERSION=4 then | 21:06 |
JayF | so TheJulia interesting thing | 21:06 |
JayF | there are no networks, at least not for network list | 21:06 |
JayF | even as admin | 21:07 |
JayF | oh this makes sense | 21:07 |
JayF | I should go read the neutron code | 21:07 |
JayF | I bet it's possible we: 1) assume v6 works and 2) blow up with bad values when it doesn't | 21:07 |
mmalchuk | folks, what's wrong with bifrost CI ? too many timeouts | 21:08 |
JayF | mmalchuk: there is a known issue right now, we're waiting for a fixed systemd package to come into centos stream 9 | 21:08 |
mmalchuk | TheJulia, JayF hi | 21:08 |
JayF | mmalchuk: rpittau is tracking it, I'd look for comments from him tomorrow as to if it's filtered down to us | 21:08 |
mmalchuk | JayF oh... I need to stop recheck and go to bed) | 21:08 |
TheJulia | sleep++ | 21:09 |
JayF | mmalchuk: generally speaking, there should never be a need to recheck repeatedly -- if you see a problem come ask | 21:09 |
mmalchuk | sometimes it help, but youre right) nevermind | 21:10 |
JayF | I mean, you're right too :D | 21:10 |
JayF | it can help sometimes | 21:10 |
JayF | but we have limited CI resources in the cloud so it's a good idea to only recheck if you know something is fixed or the reason it broke | 21:10 |
JayF | also I think we all wanna know anyway if something is busted :D | 21:11 |
JayF | TheJulia: I think I'm right | 21:11 |
JayF | TheJulia: default IP_VERSION=4+6 | 21:11 |
JayF | and I never make my VMs v6 compatible | 21:11 |
TheJulia | hmm | 21:12 |
* JayF cries in anguish | 21:12 | |
mmalchuk | irc is lack some kind of topic... for example CI status) | 21:12 |
TheJulia | interesting my browser didn't find IP_VERSION in the log | 21:12 |
JayF | mmalchuk: That's usually in the status whiteboard there | 21:12 |
JayF | mmalchuk: but honestly we don't do a good job of it | 21:12 |
JayF | TheJulia: I grepped for the default since I knew I didn't set it | 21:12 |
TheJulia | mmalchuk: you say the current topic odesn't appear? | 21:12 |
JayF | TheJulia: there's no logic to set it properly | 21:12 |
TheJulia | mmalchuk: to your client? | 21:12 |
mmalchuk | i see. | 21:12 |
TheJulia | okay, because when there is not, that is a solid sign of the irc servers being out of sync | 21:13 |
mmalchuk | TheJulia yep | 21:13 |
* TheJulia once had a global o line on an irc network many moons ago | 21:13 | |
JayF | mmalchuk: the whiteboard is updated about bifrost CI, but I won't pretend it was set before :D | 21:13 |
mmalchuk | TheJulia how r u? leave RH? | 21:14 |
TheJulia | still at RH | 21:14 |
mmalchuk | wow | 21:15 |
mmalchuk | seen email | 21:15 |
TheJulia | ... what email? | 21:15 |
JayF | that's not ominious at all | 21:15 |
mmalchuk | that you leave | 21:15 |
mmalchuk | am I wrong | 21:15 |
TheJulia | I think so, I've not gone anywhere | 21:16 |
JayF | if Julia is anything like me, she's DM'ing her boss to make sure she still works there | 21:16 |
JayF | LOL | 21:16 |
TheJulia | lol | 21:16 |
mmalchuk | cant remember | 21:16 |
TheJulia | nah, I talked to him earlier and put more work on his plate | 21:16 |
mmalchuk | lol | 21:16 |
mmalchuk | nevermind, glad to see you here | 21:17 |
mmalchuk | Pierre will update the whiteboard when CI fixed? | 21:20 |
mmalchuk | JayF ? | 21:20 |
mmalchuk | or you will? | 21:20 |
JayF | rpittau: can you make sure if/when bifrost gate is happy you let mmalchuk know and update the whitebaord | 21:20 |
JayF | that's all the power I have ;) | 21:20 |
mmalchuk | cool | 21:21 |
mmalchuk | JayF he is GMT+3 afaik, along as mee sleep alreay) | 21:23 |
JayF | of course, but the mirrors move slowly | 21:23 |
JayF | and he's owned the fix the whole time :) | 21:23 |
JayF | openstack is like the tortoise from the fable, we move slow and methodical but get there :D | 21:23 |
mmalchuk | nice fable) | 21:24 |
mmalchuk | openstack fable)) | 21:24 |
JayF | TheJulia: IP_VERSION did not change anything | 21:25 |
JayF | TheJulia: I'm going to assume that's still the problem and go from scratch though | 21:26 |
JayF | https://home.jvf.cc/~jay/local_conf.txt is what I'm going to try | 21:26 |
TheJulia | ok :( | 21:27 |
JayF | aight, trying again, this time in script so I have the output if something goes upside down again | 21:28 |
* mmalchuk sad because Bifrost CI not happy | 21:29 | |
JayF | there should be no rush to get your change in friend, it'll be fixed long before release :) | 21:29 |
mmalchuk | see you tomorrow folks | 21:29 |
JayF | o/ | 21:29 |
mmalchuk | \o | 21:30 |
JayF | oh hell I didn't update the clock on this snapshot lol | 21:30 |
JayF | aight, that should be happier now lol | 21:31 |
JayF | TheJulia: thinking more about what we passed over earlier during our chat: do you think there's a world where we could eat less CI resources + get similar coverage with less infra by using not-real-nodes behind Ironic for some of the scenarios? | 21:45 |
JayF | I guess most of what we're testing *in ironic* is node related, so faking out the nodes sorta just makes it more of a basic ironic<>nova smoke test | 21:46 |
TheJulia | the tempest suite uses fake nodes heavily for api integration | 21:48 |
TheJulia | I think the key is to elongate and consolidate | 21:48 |
TheJulia | we did that with some of the opportunistic changes, let me see if I can find an example in little bit | 21:49 |
TheJulia | ... hmmm | 21:51 |
TheJulia | we've got a bunch of different scenarios, but we do lots of setup for them too | 21:52 |
JayF | +++functions-common:test_with_retry:2333 timeout 20 sh -c 'while ! sudo ip netns exec qdhcp-effb6a1d-07cd-4fb7-9777-c7c370e4df4f ip link list | grep '\'' tap'\'' | cut -d'\'':'\'' -f2 | cut -d'\''@'\'' -f1 | cut -b2- | grep '\''^tap'\''; do sleep 1; done' | 21:57 |
JayF | Cannot open network namespace "qdhcp-effb6a1d-07cd-4fb7-9777-c7c370e4df4f": No such file or directory | 21:57 |
JayF | TheJulia: new errs! | 21:57 |
TheJulia | so, are you using OVN? | 21:57 |
JayF | https://home.jvf.cc/~jay/local_conf.txt | 21:58 |
JayF | that is our doc'd local.conf with additions as discussed at the bottom | 21:59 |
JayF | there are no settings around ovs or ovn in local.conf | 21:59 |
TheJulia | JayF: I think you need the following: | 22:07 |
TheJulia | https://www.irccloud.com/pastebin/WN1Q9gus/ | 22:07 |
JayF | I created https://etherpad.opendev.org/p/ironic-ptg-october-2023 to allow us to brainstorm topics for the upcoming PTG | 22:53 |
iurygregory | Remote Baremetal Deployments using Bifrost https://www.meetup.com/triangle-openstack-meetup/events/295149782/ Tue Aug 22 =) | 23:13 |
iurygregory | good to see people talking about bifrost <3 | 23:14 |
opendevreview | Julia Kreger proposed openstack/ironic master: Adds service steps https://review.opendev.org/c/openstack/ironic/+/884783 | 23:49 |
TheJulia | Okay, review away! | 23:49 |
TheJulia | I'll work on the other tests/decorating more tomorrow, I've been at it all day | 23:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!