*** weshay_xchat has quit IRC | 00:00 | |
*** morazi has quit IRC | 00:02 | |
*** panda_ has quit IRC | 00:06 | |
*** panda_ has joined #tripleo | 00:06 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 00:21 |
---|---|---|
*** morazi has joined #tripleo | 00:38 | |
*** eil397 has quit IRC | 00:52 | |
*** jcoufal has joined #tripleo | 00:57 | |
*** shivrao has quit IRC | 01:01 | |
*** dmacpher-afk has quit IRC | 01:15 | |
*** trozet has joined #tripleo | 01:29 | |
*** david-lyle has quit IRC | 01:31 | |
*** alop has quit IRC | 01:57 | |
dsneddon | I'm getting MongoDB errors in CI on a change that didn't touch MongoDB: http://logs.openstack.org/72/269872/2/check-tripleo/gate-tripleo-ci-f22-ceph/4de7701/console.html | 02:02 |
*** dmacpher has joined #tripleo | 02:09 | |
*** sthillma has quit IRC | 02:09 | |
*** tiswanso has quit IRC | 02:17 | |
*** tiswanso has joined #tripleo | 02:17 | |
*** jcoufal has quit IRC | 02:28 | |
*** penick has quit IRC | 02:39 | |
*** trozet has quit IRC | 02:54 | |
*** shivrao has joined #tripleo | 03:02 | |
*** rlandy has quit IRC | 03:14 | |
*** yuanying_ has joined #tripleo | 03:18 | |
*** dmacpher has quit IRC | 03:18 | |
*** yuanying has quit IRC | 03:19 | |
*** coolsvap|away is now known as coolsvap | 03:20 | |
*** yamahata has quit IRC | 03:20 | |
*** shivrao_ has joined #tripleo | 03:20 | |
*** shivrao has quit IRC | 03:23 | |
*** shivrao_ is now known as shivrao | 03:23 | |
*** dmacpher has joined #tripleo | 03:31 | |
*** shivrao has quit IRC | 03:41 | |
*** dmacpher has quit IRC | 03:43 | |
*** Marga_ has quit IRC | 03:44 | |
*** thrash is now known as thrash|g0ne | 03:46 | |
*** Marga_ has joined #tripleo | 03:47 | |
*** andreaf has quit IRC | 03:50 | |
*** Marga_ has quit IRC | 03:51 | |
*** rhallisey has quit IRC | 03:52 | |
*** andreaf has joined #tripleo | 03:52 | |
*** dmacpher has joined #tripleo | 03:55 | |
*** shivrao has joined #tripleo | 04:01 | |
*** yuanying_ has quit IRC | 04:05 | |
*** links has joined #tripleo | 04:06 | |
*** panda_ has quit IRC | 04:06 | |
*** panda_ has joined #tripleo | 04:06 | |
*** yuanying_ has joined #tripleo | 04:07 | |
*** yuanying has joined #tripleo | 04:20 | |
*** yuanying_ has quit IRC | 04:22 | |
*** Marga_ has joined #tripleo | 04:32 | |
*** stendulker has joined #tripleo | 04:36 | |
*** Marga_ has quit IRC | 04:36 | |
*** yuanying has quit IRC | 04:37 | |
*** yuanying has joined #tripleo | 04:39 | |
*** yuanying has quit IRC | 04:40 | |
*** mandre has quit IRC | 04:42 | |
*** mandre has joined #tripleo | 04:44 | |
*** yuanying has joined #tripleo | 04:44 | |
*** david-lyle has joined #tripleo | 04:44 | |
*** dmacpher has quit IRC | 04:44 | |
*** Marga_ has joined #tripleo | 04:46 | |
*** yuanying_ has joined #tripleo | 04:49 | |
*** yuanying has quit IRC | 04:50 | |
*** tiswanso has quit IRC | 04:50 | |
*** tzumainn has quit IRC | 04:51 | |
*** masco has joined #tripleo | 05:00 | |
*** masco has quit IRC | 05:00 | |
*** jaosorior has joined #tripleo | 05:06 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Properly fail/trap in eval_run_d https://review.openstack.org/259734 | 05:11 |
*** dmacpher has joined #tripleo | 05:21 | |
*** morazi has quit IRC | 05:21 | |
*** masco has joined #tripleo | 05:29 | |
*** teju has joined #tripleo | 05:31 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Use pip 7 for ironic https://review.openstack.org/270010 | 05:39 |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Use pip 7 for ironic https://review.openstack.org/270010 | 05:39 |
*** yamahata has joined #tripleo | 05:53 | |
*** xinwu has quit IRC | 05:54 | |
*** xinwu has joined #tripleo | 05:54 | |
*** xinwu has quit IRC | 05:55 | |
*** stendulker_ has joined #tripleo | 05:55 | |
*** anande has joined #tripleo | 05:56 | |
*** stendulker_ has quit IRC | 05:56 | |
*** stendulker has quit IRC | 05:59 | |
*** stendulker has joined #tripleo | 06:02 | |
*** jtomasek has joined #tripleo | 06:14 | |
*** regebro has joined #tripleo | 06:14 | |
*** absubram has joined #tripleo | 06:25 | |
*** shivrao_ has joined #tripleo | 06:31 | |
*** shivrao has quit IRC | 06:33 | |
*** shivrao_ is now known as shivrao | 06:33 | |
*** stendulker has quit IRC | 06:42 | |
*** stendulker has joined #tripleo | 06:44 | |
*** liverpooler has quit IRC | 06:45 | |
*** aufi has joined #tripleo | 06:48 | |
*** dmacpher has quit IRC | 06:54 | |
*** jaosorior has quit IRC | 06:54 | |
*** shardy has joined #tripleo | 06:58 | |
*** shivrao has quit IRC | 06:59 | |
*** rcernin has joined #tripleo | 07:02 | |
*** spredzy has quit IRC | 07:03 | |
*** spredzy has joined #tripleo | 07:03 | |
*** paramite has joined #tripleo | 07:05 | |
*** xinwu has joined #tripleo | 07:18 | |
*** aufi has quit IRC | 07:20 | |
*** aufi has joined #tripleo | 07:23 | |
*** dsneddon has quit IRC | 07:42 | |
*** dsneddon has joined #tripleo | 07:44 | |
*** cwolferh has quit IRC | 07:47 | |
*** dsneddon has quit IRC | 07:49 | |
*** 18VAAPK0U has joined #tripleo | 07:50 | |
*** dsneddon has joined #tripleo | 07:51 | |
*** dsneddon has quit IRC | 07:59 | |
*** dsneddon has joined #tripleo | 07:59 | |
*** dshulyak_ has joined #tripleo | 08:00 | |
*** jprovazn has joined #tripleo | 08:02 | |
*** panda_ has quit IRC | 08:06 | |
*** fgimenez has joined #tripleo | 08:06 | |
*** fgimenez has quit IRC | 08:06 | |
*** fgimenez has joined #tripleo | 08:06 | |
*** panda_ has joined #tripleo | 08:06 | |
*** dtantsur|afk is now known as dtantsur | 08:27 | |
*** jcoufal has joined #tripleo | 08:27 | |
*** liverpooler has joined #tripleo | 08:30 | |
*** liverpooler has quit IRC | 08:34 | |
*** liverpooler has joined #tripleo | 08:35 | |
*** devvesa has joined #tripleo | 08:39 | |
*** mcornea has joined #tripleo | 08:42 | |
*** coolsvap is now known as coolsvap|away | 08:43 | |
*** derekh has joined #tripleo | 08:47 | |
*** devvesa has quit IRC | 08:48 | |
*** bvandenh has joined #tripleo | 08:56 | |
derekh | CI is throwing this up this morning "/home/stack/overcloudrc: No such file or directory" | 08:57 |
derekh | anybody know if its moved? | 08:57 |
derekh | nevermind the error is up further, 2016-01-20 04:11:41.163 | 2016-01-20 04:07:12 [0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 | 08:58 |
* greghaynes bets $20 on pip 8 release | 09:00 | |
*** yamahata has quit IRC | 09:01 | |
marios | greghaynes: hey man, saw your ping. sorry about that. there was no-one else about and dtanstur had been waiting for >1 day and it was starting to block them | 09:03 |
greghaynes | marios: Its no problem :)_ | 09:03 |
marios | greghaynes: in future i'll refrain without asking about the v (I have tried to before but hellabusy last few days and just quickly did the release for them yesterday) | 09:04 |
marios | greghaynes: thx | 09:04 |
*** devvesa has joined #tripleo | 09:04 | |
greghaynes | marios: eh, feel free to release whenever, we might want to document how we release better... | 09:05 |
*** nico_auv has joined #tripleo | 09:11 | |
*** dtantsur is now known as dtantsur|brb | 09:14 | |
*** jistr has joined #tripleo | 09:16 | |
*** mbound has joined #tripleo | 09:24 | |
openstackgerrit | Steven Hardy proposed openstack/instack-undercloud: Fix puppet-lint failure https://review.openstack.org/270080 | 09:31 |
*** gfidente has joined #tripleo | 09:31 | |
*** gfidente has quit IRC | 09:31 | |
*** gfidente has joined #tripleo | 09:31 | |
openstackgerrit | Merged openstack/instack-undercloud: Enable loadbalancer with ssl on undercloud https://review.openstack.org/265376 | 09:35 |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup swift device string to delimit the ipv6 address with [] https://review.openstack.org/267523 | 09:36 |
*** akrivoka has joined #tripleo | 09:37 | |
*** devvesa has quit IRC | 09:37 | |
*** xinwu has quit IRC | 09:46 | |
*** xinwu has joined #tripleo | 09:48 | |
*** devvesa has joined #tripleo | 09:52 | |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Adds v6 capability to the deploy validation test (pings) https://review.openstack.org/268103 | 09:52 |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Adds v6 capability to the deploy validation test (pings) https://review.openstack.org/268103 | 09:54 |
*** mkovacik has quit IRC | 09:54 | |
openstackgerrit | Derek Higgins proposed openstack/tripleo-common: Pin the puppetlabs mongodb puppet module https://review.openstack.org/270095 | 09:55 |
*** xinwu has quit IRC | 09:56 | |
*** shivrao has joined #tripleo | 10:02 | |
*** jaosorior has joined #tripleo | 10:02 | |
*** jaosorior has quit IRC | 10:04 | |
*** jaosorior has joined #tripleo | 10:04 | |
*** xinwu has joined #tripleo | 10:05 | |
shardy | Hi all, could someone approve https://review.openstack.org/#/c/265267/ (2 line stable backport) please? | 10:05 |
marios | done | 10:06 |
shardy | thanks marios! :) | 10:07 |
marios | np, i clicked a button ;) | 10:07 |
*** shivrao has quit IRC | 10:07 | |
*** xinwu has quit IRC | 10:07 | |
*** paramite is now known as paramite|afk | 10:07 | |
*** jaosorior_ has joined #tripleo | 10:09 | |
openstackgerrit | Merged openstack/tripleo-puppet-elements: Use a stable version of MidoNet Neutron Plugin https://review.openstack.org/265267 | 10:10 |
openstackgerrit | Cyril Lopez proposed openstack/tripleo-heat-templates: Allow to allow_resize_to_same_host to true in nova https://review.openstack.org/269562 | 10:10 |
*** trellooobot has joined #tripleo | 10:10 | |
trellooobot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
trellooobot | +-------------------------------------+-------------------------------+-------------------+-------------+ | 10:10 |
trellooobot | | Title | URL | Members | Last Active | | 10:10 |
trellooobot | +-------------------------------------+-------------------------------+-------------------+-------------+ | 10:10 |
trellooobot | | Could not evalute MongoDB printjson | https://trello.com/c/GrdUk6RM | **NEEDS MEMBERS** | 17 min | | 10:10 |
trellooobot | +-------------------------------------+-------------------------------+-------------------+-------------+ | 10:10 |
*** trellooobot has quit IRC | 10:10 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Wait for cluster to settle in yum_update.sh https://review.openstack.org/263673 | 10:11 |
*** jaosorior has quit IRC | 10:11 | |
openstackgerrit | Merged openstack/python-tripleoclient: Allow adding new overcloud passwords https://review.openstack.org/268127 | 10:12 |
*** mgould has joined #tripleo | 10:15 | |
*** dtantsur|brb is now known as dtantsur | 10:15 | |
*** electrofelix has joined #tripleo | 10:16 | |
gfidente | marios, https://review.openstack.org/#/c/235423/ | 10:17 |
gfidente | dsneddon, will be happy too :) | 10:17 |
gfidente | and good morning | 10:17 |
marios | gfidente: yeah o/ hey will look in a bit on sthing else right now | 10:18 |
*** mkovacik has joined #tripleo | 10:28 | |
*** links has quit IRC | 10:28 | |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6 https://review.openstack.org/270110 | 10:36 |
jaosorior_ | Hey guys, this patch has already been merged in master https://review.openstack.org/#/c/267945/ any workflows for it? :D | 10:38 |
jaosorior_ | (it's a backport ^^) | 10:39 |
*** jaosorior_ is now known as jaosorior | 10:39 | |
jaosorior | jistr: You had checked this patch before, now it's simpler than before since we figured out some parts were not needed, can you check it out again? https://review.openstack.org/#/c/268647/ | 10:40 |
jaosorior | marios: thanks for the +A :D | 10:45 |
*** tosky has joined #tripleo | 10:46 | |
openstackgerrit | Merged openstack/puppet-tripleo: Enable X-Forwarded-Proto header for cinder https://review.openstack.org/267945 | 10:47 |
*** links has joined #tripleo | 10:51 | |
*** stendulker_ has joined #tripleo | 11:01 | |
*** stendulker has quit IRC | 11:04 | |
*** rcernin has quit IRC | 11:05 | |
*** links has quit IRC | 11:05 | |
*** paramite|afk is now known as paramite | 11:05 | |
jistr | jaosorior: looks alright :) +2 | 11:06 |
shardy | derekh: Hey, how much memory do we have now for undercloud VMs? | 11:07 |
*** rcernin has joined #tripleo | 11:07 | |
jaosorior | jistr: Thanks dude :D | 11:07 |
shardy | I suspect it's not enough, as I've seen heat-engine processes getting OOM killed here: | 11:07 |
shardy | https://review.openstack.org/#/c/260413/7 | 11:07 |
jaosorior | jistr: Is the bottle still at your team area? | 11:07 |
jaosorior | ' | 11:08 |
*** mgould has quit IRC | 11:08 | |
*** stendulker_ has quit IRC | 11:08 | |
jistr | jaosorior: lol yes, still almost untouched :) | 11:09 |
jistr | we have it as a decoration at this point | 11:09 |
jaosorior | jistr: uhm... was thinking of bringing another one, but that doesn't sound promising | 11:09 |
jaosorior | jistr: Offer hewbrocca some if he's around :P | 11:10 |
derekh | shardy: it was upped from 4G to 5G on sunday (actually looking at the logs free is reporting it as 4.8G) | 11:10 |
hewbrocca | jaosorior: WAT I had no idea what that bottle was | 11:11 |
hewbrocca | consider it comsumed | 11:11 |
jistr | :D | 11:11 |
shardy | derekh: ack - I suspect that's not enough, as the increase in vCPU count is spawning more processes | 11:11 |
hewbrocca | jaosorior: if you want something drunk at this office, you have to engage the managers | 11:12 |
* shardy looks around for stuff to turn off | 11:12 | |
derekh | feck it anyways ;-) | 11:12 |
jaosorior | hewbrocca: haha now I know :P Anyway, that better be done before DevConf, cause I'm bringing more bottles | 11:13 |
*** rcernin is now known as rcernin|lunch | 11:13 | |
jaosorior | right now the count of bottles is: 2 for two guys from the ID team, one for the tripleo midcycle and another one for DevConf | 11:14 |
derekh | shardy: ack, if you can't find something to turn off, we can either up it a little more or add a little swap somehow. | 11:14 |
shardy | derekh: Yeah, I think having the option to enable swap would be good, then we can figure out a way to scrape the logs and ensure it's not routinely getting used to excess | 11:15 |
shardy | I guess perhaps we can add it as an option to the undercloud install | 11:15 |
shardy | or at least to tripleo.sh | 11:16 |
gfidente | shardy, derekh I while ago I did some investigation on which processes were scaling with the number of cores | 11:17 |
gfidente | https://bugzilla.redhat.com/show_bug.cgi?id=1247924#c3 | 11:17 |
openstack | bugzilla.redhat.com bug 1247924 in rhel-osp-director "MariaDB max_connections must be based on controller count * core count" [High,New] - Assigned to jstransk | 11:17 |
jistr | shardy, derekh: yeah that's the other thing we did. We upped the vCPUs too, so i reckon the mem usage went up as well. | 11:18 |
derekh | gfidente: not as many as I thought there would be, but obviously to many.... | 11:18 |
gfidente | shardy, derekh also I have a running undercloud with 2 vcores | 11:18 |
shardy | gfidente: AFAIK nearly all projects use oslo processutils, which defaults to a worker process per core | 11:19 |
gfidente | $ free -m | 11:19 |
gfidente | total used free shared buff/cache available | 11:19 |
gfidente | Mem: 7823 4072 128 374 3622 3134 | 11:19 |
*** Marga_ has quit IRC | 11:19 | |
jistr | mine is | 11:20 |
jistr | [stack@instack ~]$ free -m | 11:20 |
jistr | total used free shared buff/cache available | 11:20 |
jistr | Mem: 5826 5342 132 0 351 245 | 11:20 |
gfidente | jistr, interestingly the difference in caches is a lot | 11:20 |
*** fgimenez has quit IRC | 11:20 | |
*** mgould has joined #tripleo | 11:20 | |
jistr | gfidente: i think the cache is so small in my case because i'm almost OOM | 11:20 |
gfidente | isn't cache part of used? | 11:21 |
derekh | the difference in cache could be explained by how long each undercloud has been running and how much data been accessed | 11:21 |
derekh | gfidente: the cache number if the OS cahcing file access, it gets released as memory is required | 11:22 |
derekh | then file reads get slower because less data is cached | 11:22 |
gfidente | derekh, yeah so I was noticing on mine seems to be using 3g on caches and a total of 4g | 11:23 |
derekh | so its not part of used | 11:23 |
gfidente | derekh, and on jistr the opposite | 11:23 |
gfidente | derekh, I think it is included in used | 11:24 |
*** fgimenez has joined #tripleo | 11:24 | |
*** fgimenez has quit IRC | 11:24 | |
*** fgimenez has joined #tripleo | 11:24 | |
derekh | gfidente: yup, you OS is caching 3.6G of data as that whats available, jistr doesn't have spare ram to cache as much | 11:24 |
jistr | this is the most hungry processes on my system http://fpaste.org/312720/28895214/ | 11:24 |
derekh | nope, don't think it is | 11:24 |
derekh | my laptop at the moment | 11:25 |
shardy | I raised bug #1536136 where we can decide the best solution | 11:25 |
openstack | bug 1536136 in tripleo "Undercloud sometimes OOM kills things in CI" [Undecided,New] https://launchpad.net/bugs/1536136 | 11:25 |
derekh | total used free shared buff/cache available | 11:25 |
derekh | Mem: 15737 3038 6592 649 6106 11727 | 11:25 |
derekh | cached is begger then used | 11:25 |
derekh | *bigger | 11:25 |
shardy | jistr: interesting, is that during a deploy? | 11:26 |
shardy | we can perhaps force the number of workers to 2 for many things - in particular heat-engine workers now default to 4, but 2 may be sufficient for simple CI tests | 11:26 |
jistr | shardy: nope, but it's been running for about 2 days | 11:26 |
jistr | shardy: yeah perhaps... and also i don't think we need 5 nova-api processes probably | 11:28 |
shardy | jistr: Hmm, in that case the heat-engine memory usage seems high | 11:28 |
shardy | yeah, probably some tuning we can do specific to CI testing | 11:28 |
jistr | i'm probably using newer heat engine then others | 11:29 |
jistr | [stack@instack ~]$ rpm -q openstack-heat-engine | 11:29 |
jistr | openstack-heat-engine-6.0.0.0b2-dev183.el7.centos.noarch | 11:29 |
jistr | i needed to bump heat manually to overcome the UPDATE_IN_PROGRESS getting stuck | 11:29 |
shardy | Yeah so you've picked up https://review.openstack.org/#/c/259172/ | 11:30 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add output of top to hostinfo https://review.openstack.org/270139 | 11:32 |
derekh | that should give a little info about hich processes are using most memory ^^ | 11:32 |
*** links has joined #tripleo | 11:32 | |
gfidente | jistr, shardy I have openstack-heat-engine-2015.1.2-7.el7ost.noarch | 11:32 |
gfidente | jistr, shardy running from a couple of days as well | 11:33 |
derekh | Hi all, master is broken because of a commit to the mongodb puppet module, the pin fixes it https://review.openstack.org/#/c/270095/1 | 11:34 |
gfidente | jistr, shardy but virtual memory was very similar, I see in your paste 790456 | 11:34 |
gfidente | mine is more or less same size | 11:34 |
gfidente | 8308 heat 20 0 676116 285624 2988 S 0.0 3.6 4:19.41 heat-engine | 11:34 |
gfidente | 8309 heat 20 0 663648 272676 2988 S 0.0 3.4 3:24.60 heat-engine | 11:34 |
derekh | I'm looking at the puppet module at the moment to see if I can fix it, wanna merge the pin in the meantime | 11:34 |
shardy | gfidente, jistr: I'm seeing much lower heat-engine memory usage on a freshly started undercloud | 11:35 |
shardy | does the usage go down a lot if you restart heat-engine via systemctl? | 11:36 |
jistr | yes | 11:36 |
jistr | 70 - 140 MB now | 11:36 |
gfidente | for me too | 11:36 |
jistr | per process | 11:36 |
shardy | Ok, so we probably have a memory leak then :( | 11:36 |
gfidente | but it would't explain why in ci 5 are not enough though | 11:37 |
openstackgerrit | Merged openstack/tripleo-common: Pin the puppetlabs mongodb puppet module https://review.openstack.org/270095 | 11:37 |
jistr | one thing i don't understand in my env is where does 5 nova-api proceses come from when i have osapi_compute_workers=2 | 11:38 |
shardy | gfidente: the peak memory usage may be higher than the idle usage, even with the leak | 11:38 |
*** links has quit IRC | 11:38 | |
jistr | i changed the numper of nova api workers to 1 and now i have 4 processes... i guess nova-api might fork for some other reason than just serving web requests then | 11:41 |
hewbrocca | general principles | 11:41 |
gfidente | shardy, indeed but I mean in my env it seems stable doing deploy/delete with less than 4G | 11:42 |
gfidente | shardy, or about 4G | 11:42 |
gfidente | from a couple of days | 11:42 |
gfidente | though I run RHEL fwiw | 11:42 |
*** openstackgerrit has quit IRC | 11:43 | |
gfidente | not sure if kernel settings and python deps are identical to centos | 11:43 |
*** openstackgerrit has joined #tripleo | 11:44 | |
jistr | (ahh i guess i got the nova-api thing, the other ones are metadata workers, and 1 main process. makes sense now) | 11:45 |
-openstackstatus- NOTICE: review.openstack.org is being restarted to apply patches | 11:45 | |
*** ChanServ changes topic to "review.openstack.org is being restarted to apply patches" | 11:45 | |
*** ChanServ changes topic to "TripleO | CI status: http://tripleo.org/cistatus.html | Docs http://docs.openstack.org/developer/tripleo-docs/" | 11:56 | |
-openstackstatus- NOTICE: Restart done, review.openstack.org is available | 11:56 | |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6 https://review.openstack.org/270154 | 12:02 |
gfidente | derekh, caches accounts indeed in the available (and non-free) memory | 12:04 |
*** panda_ has quit IRC | 12:06 | |
derekh | gfidente: is this related to todays breakage ? https://review.openstack.org/#/c/270154/1 | 12:06 |
*** panda_ has joined #tripleo | 12:06 | |
gfidente | derekh, nah it's for the ipv6 deployments | 12:07 |
derekh | gfidente: ack, just checking | 12:07 |
gfidente | what is today issue? | 12:07 |
derekh | gfidente: https://bugs.launchpad.net/tripleo/+bug/1536083 | 12:08 |
openstack | Launchpad bug 1536083 in tripleo "Could not evalute MongoDB shell command: printjson(rs.conf())" [Critical,Triaged] - Assigned to Derek Higgins (derekh) | 12:08 |
derekh | gfidente: pinning puppetlbs-mondb unblocked things, looks like this commit is when it started happening https://github.com/puppetlabs/puppetlabs-mongodb/commit/a77d1a191419a3f1f468707aefa69abd689c3e4b | 12:09 |
derekh | EmilienM: ^ | 12:09 |
derekh | EmilienM: any ideas? | 12:09 |
*** thrash|g0ne is now known as thrash | 12:09 | |
EmilienM | derekh: yes | 12:10 |
EmilienM | derekh: I recently patched mongo module to support IPv6 | 12:10 |
EmilienM | derekh: is it affecting master CI? | 12:11 |
EmilienM | or stable? | 12:11 |
derekh | EmilienM: master | 12:11 |
EmilienM | argh | 12:11 |
derekh | EmilienM: looks to me like the output of this command can't be parsed as json | 12:12 |
EmilienM | derekh: can you pin ? I need time to debug, lot of issues in parallel | 12:12 |
derekh | EmilienM: http://paste.openstack.org/show/484392/ | 12:12 |
derekh | EmilienM: I hvae already pinned | 12:12 |
EmilienM | derekh: I'm not sure that's my patch | 12:13 |
EmilienM | I need time | 12:13 |
derekh | the various date types in the json seem to be the problem to me | 12:13 |
derekh | EmilienM: no problem, no hurry the pin keeps us moving | 12:13 |
shardy | Does anyone know how we can make the various puppet check jobs voting? | 12:14 |
shardy | they're all working modulo https://review.openstack.org/#/c/270080/ getting backported | 12:14 |
shardy | it'd be good to stop them constantly getting broken, as it's clear folks are sometimes ignoring failures | 12:15 |
* shardy can't see where to change them in project-config atm | 12:15 | |
derekh | shardy: sec, I'll find it | 12:15 |
*** thrash has quit IRC | 12:17 | |
derekh | shardy: probably here, but it might be matching more then just the lint jobs http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n1400 | 12:19 |
*** thrash has joined #tripleo | 12:19 | |
*** thrash has quit IRC | 12:19 | |
*** thrash has joined #tripleo | 12:19 | |
shardy | derekh: ah, thanks, yeah that looks like those two will do it | 12:20 |
shardy | derekh: I sent https://review.openstack.org/270165, thanks - I'll also drop a list to the ML so folks are aware | 12:25 |
derekh | shardy: +1 | 12:27 |
*** rhallisey has joined #tripleo | 12:35 | |
*** dmacpher has joined #tripleo | 12:39 | |
openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Allow container template to recognize an update https://review.openstack.org/265368 | 12:39 |
*** olap_ has joined #tripleo | 12:40 | |
*** olap_ has quit IRC | 12:40 | |
*** olap_ has joined #tripleo | 12:41 | |
*** weshay_xchat has joined #tripleo | 12:42 | |
derekh | EmilienM: found the problem, https://bugs.launchpad.net/tripleo/+bug/1536083/comments/3 | 12:43 |
openstack | Launchpad bug 1536083 in tripleo "Could not evalute MongoDB shell command: printjson(rs.conf())" [Critical,Triaged] - Assigned to Derek Higgins (derekh) | 12:43 |
derekh | EmilienM: I'll see if I can muster up enough ruby Foo to submit a fix | 12:43 |
openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Config fixes for proper nova/neutron notifications https://review.openstack.org/264217 | 12:51 |
*** rlandy has joined #tripleo | 12:55 | |
gfidente | shardy, what is best way to have newlines in the str_replace/template thing? | 12:55 |
*** athomas has quit IRC | 12:57 | |
*** athomas has joined #tripleo | 13:01 | |
gfidente | shardy, going for : | | 13:01 |
shardy | gfidente: Yeah, I'd do : | | 13:01 |
gfidente | tks | 13:02 |
*** shivrao has joined #tripleo | 13:04 | |
*** morazi has joined #tripleo | 13:05 | |
*** rcernin|lunch is now known as rcernin | 13:07 | |
*** shivrao has quit IRC | 13:08 | |
*** anande has quit IRC | 13:09 | |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6 https://review.openstack.org/270154 | 13:10 |
EmilienM | derekh: looking | 13:13 |
EmilienM | that's weird, we tested it againsts both ipv4 & ipv6 | 13:14 |
*** jayg|g0n3 is now known as jayg | 13:17 | |
*** trozet has joined #tripleo | 13:19 | |
*** trown|outttypeww is now known as trown | 13:24 | |
*** ayoung has quit IRC | 13:24 | |
*** dprince has joined #tripleo | 13:26 | |
gfidente | shardy, another question, can I concat as value {get_attr}+string ? | 13:29 |
shardy | gfidente: Yes, if it doesn't work directly you can use either str_replace or list_join | 13:31 |
gfidente | ok I'll settle for list_join | 13:32 |
shardy | gfidente: Hey, can you remind me, what was going wrong with NetworkDeployment when it runs on update? | 13:32 |
shardy | If you set NetworkDeploymentActions to include update, e.g to apply a new nic config, will it break the update due to pacemaker issues? | 13:33 |
gfidente | shardy, doing ifdown/ifup caused troubles to corosync | 13:33 |
gfidente | shardy, yeah exactly | 13:33 |
gfidente | shardy, os-net-config does the right thing when the nic config does not change though | 13:33 |
gfidente | shardy, it's the upgrades to the nic configs which triggered ifdown/ifup | 13:34 |
shardy | gfidente: thanks | 13:35 |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: WIP: Emits a different hostname for each network the node uses https://review.openstack.org/270189 | 13:38 |
*** mbound has quit IRC | 13:39 | |
jistr | anybody recalls what's the issue with "CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Controller: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 6499acd4-80c5-4fae-866f-db9877fa3b75." ? | 13:39 |
*** trozet has quit IRC | 13:39 | |
jistr | i seem to be able to re-introspect the nodes just fine, so ironic can still control them | 13:39 |
* jistr digs for more info | 13:39 | |
*** Goneri has joined #tripleo | 13:43 | |
hewbrocca | jistr: hang on, that's the damn scheduler bug | 13:45 |
*** jcoufal_ has joined #tripleo | 13:47 | |
*** hewbrocc` has joined #tripleo | 13:48 | |
EmilienM | derekh: I'm working on a fix right now | 13:49 |
*** crinkle_ has joined #tripleo | 13:49 | |
jistr | hewbrocca: thanks, unfortunately it seems to be something different. There's no mention of NodeAssociated in ironic-api.log. But it's most probably some problem between nova and ironic too. | 13:49 |
hewbrocc` | hmm OK | 13:49 |
*** hewbrocc` is now known as hewbrocca` | 13:50 | |
*** teju has left #tripleo | 13:50 | |
hewbrocca` | INteresting | 13:50 |
hewbrocca` | dtantsur: ^^^ | 13:50 |
dtantsur | moar bugs \o/ | 13:50 |
dtantsur | jistr, look for errors in both ironic-conductor and nova-compute logs | 13:51 |
dtantsur | nova tends to clean up everything after failure (so nice of it, damn) | 13:51 |
derekh | EmilienM: ack, here is my attempt, don't have ipv6 to check both cases though | 13:51 |
derekh | EmilienM: http://paste.openstack.org/show/484406/ | 13:51 |
*** liverpoo1er has joined #tripleo | 13:51 | |
*** lifeless_ has joined #tripleo | 13:51 | |
*** bandini_ has joined #tripleo | 13:51 | |
*** Slower_ has joined #tripleo | 13:51 | |
*** gothicmi1dfood has joined #tripleo | 13:51 | |
jistr | dtantsur: ironic-api.log is clean of error messages, in nova-compute there's this http://fpaste.org/312783/53297910/ | 13:52 |
dtantsur | jistr, ironic-conductor, not ironic-api | 13:52 |
jistr | dtantsur: ah thanks | 13:53 |
jistr | yea got it | 13:53 |
dtantsur | awesome error message, thanks nova :D | 13:53 |
jistr | 2016-01-20 13:02:58.975 23433 ERROR ironic.drivers.modules.iscsi_deploy [-] Deploy failed for instance 8a1f6072-acad-4c5a-8acd-38b2a8e67fde. Error: [Errno 12] Cannot allocate memory | 13:53 |
dtantsur | well, that looks like your problem | 13:53 |
trown | not enough RAM on the undercloud? | 13:53 |
* dtantsur puts this nova behavior on his troubleshooting talk for the rdo day | 13:53 | |
jistr | yeah most probably, where not enough == 6 GB | 13:54 |
*** mandre_ has joined #tripleo | 13:55 | |
*** leifmadsen_ has joined #tripleo | 13:55 | |
jistr | heat-engine grew to 1 GB after a few stack-updates | 13:55 |
*** cinerama` has joined #tripleo | 13:55 | |
jistr | (all workers summed together) | 13:55 |
*** athomas has quit IRC | 13:56 | |
*** nico_auv has quit IRC | 13:56 | |
*** 18VAAPK0U has quit IRC | 13:56 | |
*** paramite has quit IRC | 13:56 | |
*** regebro has quit IRC | 13:56 | |
*** hewbrocca has quit IRC | 13:56 | |
*** Slower has quit IRC | 13:56 | |
*** d0ugal has quit IRC | 13:56 | |
*** tserong has quit IRC | 13:56 | |
*** shadower has quit IRC | 13:56 | |
*** gothicmindfood has quit IRC | 13:56 | |
*** mcornea has quit IRC | 13:56 | |
*** liverpooler has quit IRC | 13:56 | |
*** jcoufal has quit IRC | 13:56 | |
*** mandre has quit IRC | 13:56 | |
*** bandini has quit IRC | 13:56 | |
*** crinkle has quit IRC | 13:56 | |
*** cinerama has quit IRC | 13:56 | |
*** lifeless has quit IRC | 13:56 | |
*** leifmadsen has quit IRC | 13:56 | |
*** leifmadsen_ is now known as leifmadsen | 13:56 | |
*** absubram has quit IRC | 13:56 | |
*** mandre_ is now known as mandre | 13:58 | |
*** nico_auv has joined #tripleo | 14:02 | |
*** paramite has joined #tripleo | 14:02 | |
*** tserong has joined #tripleo | 14:02 | |
*** mcornea has joined #tripleo | 14:03 | |
*** d0ugal has joined #tripleo | 14:03 | |
jistr | hmm nope still having issues getting the machines to spawn. this in nova-compute log, nothing in ironic-conductor or ironic-api logs http://fpaste.org/312786/14532985/ | 14:04 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 14:04 |
*** regebro has joined #tripleo | 14:04 | |
*** egafford has joined #tripleo | 14:05 | |
*** ukalifon has joined #tripleo | 14:05 | |
*** athomas has joined #tripleo | 14:05 | |
dtantsur | jistr, do you see ironicclient output a bit above your paste? might contain the actual error | 14:06 |
jistr | dtantsur: http://fpaste.org/312789/32988211/ so a problem between ironic api and ironic-conductor? | 14:07 |
dtantsur | jistr, it looks like conductor failed to start, can you check it? | 14:07 |
*** tosky has quit IRC | 14:08 | |
jistr | dtantsur: looks ok http://fpaste.org/312791/98954145/raw/ | 14:09 |
*** olap_ has quit IRC | 14:09 | |
* jistr tries to find where the connection between api and conductor is configured | 14:09 | |
dtantsur | jistr, it's amqp | 14:10 |
hewbrocca` | the rabbit done died | 14:10 |
dtantsur | jistr, what does 'ironic driver-list' output for you? | 14:10 |
hewbrocca` | uh-yes it did | 14:10 |
*** hewbrocca` is now known as hewbrocca | 14:10 | |
*** mbound has joined #tripleo | 14:10 | |
* hewbrocca walk this way | 14:10 | |
openstackgerrit | Jaume Devesa proposed openstack/tripleo-heat-templates: Create linux bridge vlans environments https://review.openstack.org/270205 | 14:11 |
jistr | dtantsur: driver-list is empty. Rabbit is running according to systemctl. Looking for more suspicious stuff. | 14:11 |
*** olap has joined #tripleo | 14:11 | |
dtantsur | jistr, you may try to restart the conductor then.. unless you find some failures | 14:11 |
jistr | dtantsur: that has probably done it, after restarting conductor i see driver-list populated with entries. Will retry the stack create. Thanks! | 14:13 |
openstackgerrit | Jaume Devesa proposed openstack/tripleo-heat-templates: Fix MidoNet errors https://review.openstack.org/270207 | 14:14 |
*** tosky has joined #tripleo | 14:14 | |
*** mbound has quit IRC | 14:15 | |
EmilienM | derekh: https://github.com/puppetlabs/puppetlabs-mongodb/pull/257 | 14:15 |
EmilienM | derekh: it will land today | 14:15 |
EmilienM | I'll ping you when done | 14:15 |
EmilienM | derekh: and sorry for that | 14:15 |
*** lblanchard has joined #tripleo | 14:18 | |
derekh | EmilienM: no prob and thanks for looking at it | 14:19 |
*** tzumainn has joined #tripleo | 14:20 | |
jistr | ok now i'm getting "just" the PortBindingFailed error, will try to debug neutron for a change :) i wonder if the OOM error prompted the operating system to do something not-so-nice to the undercloud in general | 14:22 |
dtantsur | it's pretty possible, I would restart everything just in case | 14:22 |
jistr | yeah ovs agent is down | 14:23 |
*** tiswanso has joined #tripleo | 14:23 | |
jistr | dtantsur: alright thanks a lot for the help :) | 14:23 |
*** athomas has quit IRC | 14:24 | |
dtantsur | yvw | 14:26 |
*** athomas has joined #tripleo | 14:28 | |
*** rpothier has joined #tripleo | 14:29 | |
*** tiswanso has quit IRC | 14:30 | |
*** tiswanso has joined #tripleo | 14:31 | |
*** absubram has joined #tripleo | 14:33 | |
*** liverpoo1er has quit IRC | 14:36 | |
*** paramite has quit IRC | 14:39 | |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6 https://review.openstack.org/270154 | 14:50 |
*** eggmaster has joined #tripleo | 14:51 | |
*** jprovazn has quit IRC | 14:52 | |
*** masco has quit IRC | 14:53 | |
rbrady | dprince: I'm available from now until 3pm EST whenever you want to meet up | 14:54 |
dprince | rbrady: how about we shoot for 1PM EST? | 14:58 |
* dprince is home with a sick kid today and will have more help by then | 14:58 | |
rbrady | dprince: that's good for me. Can we record it for the folks in Europe that cannot make it that late? | 14:58 |
dprince | anyone else interested in an ad-hoc Mistral demo... rbrady and I will be discussing it later today on BlueJeans | 14:58 |
*** electrofelix has quit IRC | 14:59 | |
dprince | rbrady: but if we record it that means I won't be able to tell you what I really think!!! | 14:59 |
dprince | rbrady: what I mean is... I might just prefer to make a separate (polished) demo recording without my rants | 14:59 |
rbrady | dprince: I thought everyone has already had the chance to experience dprince unfiltered at least once already | 15:00 |
rbrady | dprince: ack | 15:00 |
*** jaosorior has quit IRC | 15:06 | |
*** shivrao has joined #tripleo | 15:06 | |
*** shivrao has quit IRC | 15:11 | |
*** electrofelix has joined #tripleo | 15:13 | |
dtantsur | d0ugal or thrash: could you review https://review.openstack.org/#/c/250405/ please? profile matching: long, but important :) | 15:14 |
thrash | dtantsur: I'll take a look shortly. | 15:14 |
dtantsur | thanks | 15:14 |
thrash | anybody that can give some review love to https://review.openstack.org/#/c/266404/ would be awesome. :) | 15:14 |
*** electrofelix has quit IRC | 15:16 | |
*** electrofelix has joined #tripleo | 15:17 | |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6 https://review.openstack.org/270154 | 15:17 |
openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: SoftwareDeployment interface for OS::TripleO::Tasks::PackageUpdate https://review.openstack.org/270242 | 15:19 |
openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Move controller upgrade to the yum_update hook https://review.openstack.org/270243 | 15:19 |
*** trozet has joined #tripleo | 15:21 | |
thrash | dprince: d0ugal thanks | 15:23 |
d0ugal | dtantsur: Yup, I'll take a look - but it might be tomorrow morining with a strong coffee :) | 15:26 |
*** mbound has joined #tripleo | 15:26 | |
dtantsur | ack | 15:27 |
*** yamahata has joined #tripleo | 15:30 | |
*** bvandenh has quit IRC | 15:31 | |
*** jprovazn has joined #tripleo | 15:36 | |
*** fabbione has quit IRC | 15:36 | |
*** fabbione has joined #tripleo | 15:37 | |
*** tiswanso has quit IRC | 15:39 | |
openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Wire Neutron ML2 plugin and OVS agent settings as arrays https://review.openstack.org/269898 | 15:39 |
openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Revert "Revert "neutron: enable nova-event-callback by default"" https://review.openstack.org/269827 | 15:39 |
openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Fix tunnel_types hieradata on compute nodes https://review.openstack.org/269829 | 15:39 |
openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Fix neutron-nova notifications https://review.openstack.org/269828 | 15:39 |
*** tiswanso has joined #tripleo | 15:41 | |
*** electrofelix has left #tripleo | 15:42 | |
openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Don't write CLOUDNAME to the hosts file. https://review.openstack.org/267917 | 15:43 |
dtantsur | devvesa, hi! do you still plan on working on https://review.openstack.org/#/c/265827/ ? I need it for clean rebase of another patch | 15:46 |
devvesa | dtantsur: I should do it | 15:46 |
devvesa | dtantsur: What about if I do it tomorrow? Does it work for you? | 15:46 |
dtantsur | yeah, sure. thanks! | 15:47 |
devvesa | Ok. Tomorrow you'll have it :) | 15:47 |
openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder: Mark ironic-discoverd-ramdisk as deprecated in favor of ironic-agent https://review.openstack.org/270267 | 15:54 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add some swap to the undercloud https://review.openstack.org/270274 | 15:58 |
derekh | pign test passed for nonha and ceilometer jobs, need to fix HA now | 15:59 |
derekh | *ping | 15:59 |
*** fabbione has quit IRC | 16:02 | |
*** fabbione has joined #tripleo | 16:03 | |
trown | derekh: is the HA issue just a resource thing, or does it actually fail? | 16:03 |
trown | derekh: I have had issues getting HA to pass some of the tempest smoke tests, but I am almost positive it is just resource constraints | 16:04 |
*** shivrao has joined #tripleo | 16:05 | |
*** panda_ has quit IRC | 16:06 | |
*** panda_ has joined #tripleo | 16:07 | |
derekh | trown: 2016-01-20 15:37:50.143 | An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-77948921-a027-4e6c-8ac5-ca62072a4e92) | 16:07 |
derekh | trown: an error returned by one of the nova or neutron api, not sure of anything more yet | 16:08 |
*** jaosorior has joined #tripleo | 16:08 | |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6 https://review.openstack.org/270154 | 16:08 |
trown | derekh: hmm... it seems like either HA is broken, or we do not have enough resources in CI to deploy HA... I have a hard time seeing how else we would have nonha ping test pass but ha fail | 16:10 |
openstackgerrit | Jaume Devesa proposed openstack/tripleo-docs: Extending the image build information https://review.openstack.org/270290 | 16:10 |
trown | derekh: ie, there should not be anything specific on the test side for ha vs nonha | 16:10 |
derekh | trown: yup | 16:10 |
*** jaosorior has quit IRC | 16:11 | |
trown | derekh: personally, I think we should merge the ping test then, at least it will prevent merging things that break the nonha overcloud | 16:12 |
trown | derekh: would it be possible to only run the ping test on nonha and ceph, like introspection only runs on nonha? | 16:13 |
derekh | trown: fair enough, there is a couple of comments on the test in tripleo-common , we should figure out if we want to sort them out now or post merge, then do it | 16:13 |
derekh | trown: yes that would be possible also | 16:15 |
trown | deploying HA realisically in a resource constrained env might be impossible | 16:15 |
derekh | Yup, Ive wrechecked to see if it was a once off, then we can decide if we want to go without it or not | 16:17 |
*** shivrao has quit IRC | 16:19 | |
bnemec | derekh: I opened a bug for the ha failure. It's another wsrep error, which I've been seeing a lot lately: https://bugs.launchpad.net/tripleo/+bug/1536250 | 16:22 |
openstack | Launchpad bug 1536250 in tripleo "Intermittent database problems in the HA job" [High,Triaged] | 16:22 |
bnemec | I don't think it actually has anything to do with the ping test because I've seen it happen a lot on the existing job too. | 16:22 |
bnemec | Although it's possible the ping test will trigger it more often. | 16:23 |
*** Marga_ has joined #tripleo | 16:24 | |
*** rcernin has quit IRC | 16:25 | |
*** aufi has quit IRC | 16:30 | |
*** tiswanso has quit IRC | 16:34 | |
*** cinerama` is now known as cinerama | 16:37 | |
*** cwolferh has joined #tripleo | 16:39 | |
*** electrofelix has joined #tripleo | 16:39 | |
*** mbound has quit IRC | 16:53 | |
openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Move controller upgrade to the yum_update hook https://review.openstack.org/270243 | 16:55 |
openstackgerrit | Jaume Devesa proposed openstack/tripleo-docs: Extending the image build information https://review.openstack.org/270290 | 16:55 |
*** sthillma has joined #tripleo | 16:58 | |
*** sthillma_ has joined #tripleo | 16:59 | |
openstackgerrit | Jaume Devesa proposed openstack/tripleo-docs: Add MidoNet documentation in advanced deployment https://review.openstack.org/270320 | 17:00 |
*** mbound has joined #tripleo | 17:02 | |
*** sthillma has quit IRC | 17:02 | |
*** sthillma_ is now known as sthillma | 17:02 | |
*** trown is now known as trown|lunch | 17:03 | |
*** devvesa has quit IRC | 17:03 | |
*** jistr has quit IRC | 17:04 | |
*** tiswanso has joined #tripleo | 17:05 | |
*** akrivoka is now known as akrivoka|afk | 17:06 | |
*** mkovacik has quit IRC | 17:06 | |
*** tiswanso has quit IRC | 17:06 | |
*** tiswanso has joined #tripleo | 17:07 | |
afazekas | what can be used in tripleo-common repo for changing a *.ini file which is at least as fast as the devstack iniset ? | 17:09 |
*** yamahata has quit IRC | 17:10 | |
*** dtantsur is now known as dtantsur|afk | 17:11 | |
*** bandini_ is now known as bandini | 17:12 | |
*** fgimenez has quit IRC | 17:18 | |
*** rasca has quit IRC | 17:21 | |
afazekas | I would like to set the overcloud credentials in the tempest.conf | 17:22 |
afazekas | derekh, ^ | 17:23 |
afazekas | openstack-config has a similar interface but it is slower (python load time) | 17:24 |
*** rasca has joined #tripleo | 17:25 | |
*** Marga_ has quit IRC | 17:25 | |
shardy | afazekas: personally I think copying the iniset function into tripleo-common would be OK - it's only like 30 lines of code | 17:26 |
*** Marga_ has joined #tripleo | 17:26 | |
shardy | unless there's some existing alternative (I'm not aware of one) | 17:26 |
derekh | afazekas: shardy yum, I'd be fine with that, was about to say the same thing, shardy was quicker looking it up ;-) | 17:27 |
derekh | if its a fairly simple case even a subset of iniset might do | 17:27 |
derekh | :q | 17:28 |
bnemec | afazekas: Would it be possible to do it in here: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/utils.py#L114 | 17:28 |
bnemec | That's where we are already setting tempest things that are specific to the deployment. | 17:28 |
afazekas | I do not see 'tempest-deployer-input.conf' realy gets created | 17:29 |
afazekas | Where should I find it ? | 17:30 |
afazekas | Is the config supposed to contain everything ? or just the list of deployed components and available features ? | 17:31 |
* afazekas I also do not have overcloudrc , maybe something is wrong with my deployment .. | 17:32 | |
* afazekas yes it was broken .. | 17:34 | |
afazekas | openstack-config can merge ini files for that it can be faster than a shell script when the arg number is high | 17:35 |
*** yamahata has joined #tripleo | 17:37 | |
*** yamahata has quit IRC | 17:37 | |
*** yamahata has joined #tripleo | 17:37 | |
*** shivrao has joined #tripleo | 17:38 | |
*** shivrao has quit IRC | 17:39 | |
*** olap has quit IRC | 17:40 | |
*** jdob has quit IRC | 17:43 | |
afazekas | when tempest used from rpm this https://github.com/redhat-openstack/khaleesi/blob/master/playbooks/tester/templates/configure-tempest.j2 is used by khaleesi | 17:44 |
afazekas | should I follow the same configuration pattern ? | 17:44 |
afazekas | tools/config_tempest.py is not in the upstream tempest repo | 17:45 |
*** ayoung has joined #tripleo | 17:47 | |
dprince | gfidente: again https://review.openstack.org/#/c/269068 | 17:48 |
gfidente | dprince, hey not to steal a vote but does this look valid or is there a reason for not doing it that you can think https://review.openstack.org/270342 ? | 17:52 |
afazekas | derekh: Do you recommend to use the rpm specific scripts ? | 17:52 |
*** jcoufal_ has quit IRC | 17:54 | |
*** mbound has quit IRC | 17:54 | |
*** shivrao has joined #tripleo | 17:58 | |
*** weshay_xchat has quit IRC | 17:58 | |
*** shardy has quit IRC | 17:59 | |
afazekas | derekh, is the tempest usage in the tripleo ci should depend on the /usr/share/openstack-tempest*/tools/config_tempest.py which is part of the rpm ? | 17:59 |
derekh | Is that what you mean by the rpm specif scripts? | 18:00 |
*** sthillma has quit IRC | 18:00 | |
derekh | afazekas: I'm not very familiar with tempest, its been about 2 years since I used it, what is the alternative? | 18:00 |
afazekas | derekh: yes | 18:00 |
derekh | afazekas: if you mean files from /usr/share/openstack-tempest* or fomr a git checkout of tempest | 18:01 |
derekh | afazekas: then I would say we should use the rpm ones | 18:02 |
*** weshay_xchat has joined #tripleo | 18:02 | |
bnemec | This is our currently documented method of running tempest: http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_deployment_cli.html#validate-the-overcloud | 18:02 |
bnemec | If the deployer-input.conf file is not being created then it's a bug. | 18:02 |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Use pip 7 for ironic https://review.openstack.org/270010 | 18:02 |
derekh | Yup, ci should be doing what we are documenting | 18:03 |
*** weshay_xchat is now known as weshay | 18:03 | |
afazekas | derekh: these scripts are not in the official upstream tempest repo, so I do not really like to use them, the alternative is implementing something similar .. | 18:03 |
derekh | or if that can't be done perhapes something in the docs is wrong | 18:04 |
*** crinkle_ is now known as crinkle | 18:04 | |
derekh | afazekas: the problem is, if you don't use them then why do we bother package them at all | 18:04 |
derekh | I mean they are in the RDO repository that we are trying to validate, if its not a package we intent to test then we should remove the package | 18:05 |
afazekas | derekh: This was not request from QE, AFAIK customers wanted it in rpm | 18:05 |
afazekas | It is also related somehow to certification processes | 18:06 |
afazekas | bnemec: Ok, I'll will try to follow the doc | 18:07 |
derekh | afazekas: sorry, I'm not saying you wanted it there, I'm saying it is there and if we don't use it to test then its probably broken and we shoudl remove it | 18:07 |
derekh | afazekas: I think, if we as a community think its important enough a package to keep in the repository then we should be using it, thats my opinion anyways | 18:08 |
*** paramite has joined #tripleo | 18:08 | |
derekh | afazekas: sorry I got to run for the evening, others might have other opinions on here about the packaged tempest vs. tempest from git | 18:10 |
derekh | for anybody interested afazekas is looking into adding some tepest tests to our periodic jobs | 18:10 |
derekh | ttyl | 18:10 |
*** derekh has quit IRC | 18:10 | |
*** lifeless_ is now known as lifeless | 18:12 | |
*** trown|lunch is now known as trown | 18:12 | |
*** jtomasek has quit IRC | 18:15 | |
*** ukalifon has quit IRC | 18:15 | |
*** dhellmann_ has joined #tripleo | 18:20 | |
*** isq_ has joined #tripleo | 18:20 | |
*** trozet_ has joined #tripleo | 18:20 | |
*** larstobi_ has joined #tripleo | 18:21 | |
*** mandre has quit IRC | 18:21 | |
*** trozet has quit IRC | 18:22 | |
*** rpothier has quit IRC | 18:22 | |
*** athomas has quit IRC | 18:22 | |
*** akrivoka|afk has quit IRC | 18:22 | |
*** cody-somerville- has quit IRC | 18:22 | |
*** dhellmann has quit IRC | 18:22 | |
*** mburned has quit IRC | 18:22 | |
*** isq has quit IRC | 18:22 | |
*** larstobi has quit IRC | 18:22 | |
*** larstobi_ is now known as larstobi | 18:22 | |
*** dhellmann_ is now known as dhellmann | 18:22 | |
*** cody-somerville- has joined #tripleo | 18:23 | |
*** akrivoka|afk has joined #tripleo | 18:24 | |
*** athomas has joined #tripleo | 18:24 | |
*** trozet_ is now known as trozet | 18:25 | |
*** mandre has joined #tripleo | 18:25 | |
*** penick has joined #tripleo | 18:27 | |
*** paramite is now known as paramite|afk | 18:29 | |
*** gfidente is now known as gfidente|afk | 18:30 | |
*** paramite|afk is now known as paramite | 18:33 | |
*** akrivoka|afk has quit IRC | 18:35 | |
*** athomas has quit IRC | 18:35 | |
*** ayoung has quit IRC | 18:37 | |
*** ayoung has joined #tripleo | 18:37 | |
*** akrivoka|afk has joined #tripleo | 18:37 | |
*** athomas has joined #tripleo | 18:37 | |
openstackgerrit | Ethan Gafford proposed openstack/python-tripleoclient: Sahara integration https://review.openstack.org/221418 | 18:40 |
*** jdob_lt has joined #tripleo | 18:42 | |
*** mgould has quit IRC | 18:43 | |
*** electrofelix has quit IRC | 18:47 | |
*** paramite has quit IRC | 18:47 | |
*** akrivoka|afk has quit IRC | 18:48 | |
*** athomas has quit IRC | 18:48 | |
*** akrivoka|afk has joined #tripleo | 18:48 | |
*** athomas has joined #tripleo | 18:48 | |
*** gothicmi1dfood has quit IRC | 18:53 | |
*** cmyster has quit IRC | 18:53 | |
*** cmyster has joined #tripleo | 18:53 | |
*** gothicmindfood has joined #tripleo | 18:56 | |
*** hewbrocca has quit IRC | 18:56 | |
*** hewbrocca has joined #tripleo | 18:56 | |
*** Marga_ has quit IRC | 18:56 | |
*** bandini has quit IRC | 18:56 | |
*** bandini has joined #tripleo | 18:56 | |
*** tosky has quit IRC | 18:57 | |
*** tosky has joined #tripleo | 18:57 | |
*** lblanchard has quit IRC | 18:57 | |
*** lblanchard has joined #tripleo | 18:57 | |
*** weshay has quit IRC | 18:58 | |
*** weshay has joined #tripleo | 18:58 | |
*** akrivoka|afk has quit IRC | 18:59 | |
*** athomas has quit IRC | 18:59 | |
openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: [WIP] trying to add one block node https://review.openstack.org/270423 | 18:59 |
*** trozet has quit IRC | 18:59 | |
*** trozet has joined #tripleo | 18:59 | |
*** penick has quit IRC | 18:59 | |
*** penick has joined #tripleo | 18:59 | |
*** ayoung has quit IRC | 18:59 | |
*** ayoung has joined #tripleo | 18:59 | |
*** rwsu has quit IRC | 19:03 | |
*** liverpooler has joined #tripleo | 19:05 | |
*** xinwu has joined #tripleo | 19:07 | |
*** sthillma has joined #tripleo | 19:08 | |
*** Marga_ has joined #tripleo | 19:09 | |
*** mburned has joined #tripleo | 19:11 | |
*** tosky has quit IRC | 19:15 | |
*** bnemec has quit IRC | 19:17 | |
*** Goneri has quit IRC | 19:17 | |
*** dshulyak_ has quit IRC | 19:22 | |
*** Marga_ has quit IRC | 19:25 | |
*** bnemec has joined #tripleo | 19:25 | |
*** rwsu has joined #tripleo | 19:30 | |
EmilienM | ok mongodb issue is fixed upstream | 19:47 |
EmilienM | I'm going to unping ooo ci | 19:47 |
EmilienM | unpin* | 19:48 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-common: Revert "Pin the puppetlabs mongodb puppet module" https://review.openstack.org/270451 | 19:50 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-common: Revert "Pin the puppetlabs mongodb puppet module" https://review.openstack.org/270451 | 19:50 |
*** trown is now known as trown|mtg | 19:51 | |
*** Marga_ has joined #tripleo | 19:55 | |
openstackgerrit | James Slagle proposed openstack/puppet-tripleo: packages: secure upgrade workflow from dependency cycles https://review.openstack.org/268388 | 19:57 |
*** jdob_lt has quit IRC | 19:59 | |
*** Goneri has joined #tripleo | 20:02 | |
*** mcornea has quit IRC | 20:06 | |
*** panda_ has quit IRC | 20:06 | |
*** weshay has quit IRC | 20:06 | |
*** dshulyak_ has joined #tripleo | 20:18 | |
*** panda_ has joined #tripleo | 20:18 | |
*** akrivoka|afk has joined #tripleo | 20:18 | |
*** athomas has joined #tripleo | 20:18 | |
*** weshay has joined #tripleo | 20:18 | |
*** davidlenwell has quit IRC | 20:18 | |
*** davidlenwell has joined #tripleo | 20:20 | |
*** ChanServ sets mode: +v davidlenwell | 20:20 | |
*** jprovazn has quit IRC | 20:23 | |
*** dshulyak_ has quit IRC | 20:28 | |
*** panda_ has quit IRC | 20:28 | |
*** akrivoka|afk has quit IRC | 20:28 | |
*** athomas has quit IRC | 20:28 | |
*** dprince has quit IRC | 20:36 | |
*** jdob_lt has joined #tripleo | 20:37 | |
ayoung | EmilienM, running instack-install-undercloud' and the tripleo.sh equivlaent I get Error: Could not find class ::ironic::keystone::auth_inspector for instack on node instack | 20:40 |
ayoung | known issue? | 20:40 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 20:40 |
EmilienM | ayoung: that's definitly an OPM issue | 20:40 |
EmilienM | ayoung: on RDO manager? | 20:40 |
ayoung | EmilienM, Tripleo upstream | 20:41 |
EmilienM | oh | 20:41 |
ayoung | Centeos 7 | 20:41 |
EmilienM | even worse | 20:41 |
EmilienM | ayoung: let me check, asec | 20:41 |
EmilienM | ayoung: master? | 20:41 |
EmilienM | I see the class in ironic::keystone::auth_inspector (puppet-ironic) | 20:41 |
ayoung | EmilienM, um...the only git checkout was of tripleo.sh. THe rest was following the docs, so I don't know what branch | 20:41 |
EmilienM | do we pin puppet-ironic? | 20:42 |
EmilienM | let me check that | 20:42 |
ayoung | http://docs.openstack.org/developer/tripleo-docs/installation/installing.html | 20:42 |
ayoung | delorean...that is master, no? | 20:42 |
EmilienM | yeah | 20:42 |
EmilienM | derek & dprince are not around :( | 20:42 |
EmilienM | this is weird! | 20:42 |
*** dshulyak_ has joined #tripleo | 20:42 | |
*** panda_ has joined #tripleo | 20:42 | |
*** akrivoka|afk has joined #tripleo | 20:42 | |
*** athomas has joined #tripleo | 20:42 | |
EmilienM | ayoung: the puppet class is in master and tripleo CI is usng master modules | 20:42 |
*** weshay has quit IRC | 20:44 | |
egafford | ayoung, EmilienM: Possible that ayoung needs to use export DIB_INSTALLTYPE=source to pull in changes in master that haven't propagated to delorean? | 20:46 |
ayoung | EmilienM, say I want to try and run this with Keystone in HTTPD. can I do that now? | 20:47 |
egafford | Or rather export DIB_INSTALLTYPE_puppet_modules=source | 20:47 |
EmilienM | ayoung: you've seen my patch in progress | 20:47 |
ayoung | egafford, export that and then rerun tripleo-common/scripts/tripleo.sh --undercloud | 20:47 |
EmilienM | but I'm not making any progress on it right now | 20:47 |
ayoung | EmilienM, I have ,but I thought that was overcloud only | 20:47 |
EmilienM | ayoung: undercloud, right I can send a patch | 20:48 |
EmilienM | ayoung: let me 2 min | 20:48 |
ayoung | since undercloud is already failing on a Keystone issue, hey, why not? | 20:48 |
ayoung | excellent | 20:48 |
EmilienM | ayoung: let me 10 min | 20:49 |
EmilienM | I need to write the patch | 20:49 |
EmilienM | ayoung: it's actually not easy | 20:50 |
EmilienM | ayoung: keystone is not managed by Puppet on the undercloud | 20:50 |
EmilienM | :( | 20:50 |
ayoung | EmilienM, OK, for now, let me try to complete the Undercloud as is with install from source | 20:50 |
egafford | ayoung: I think, tragically, that you might have to respin your instack-virt-setup to get the puppet module in there properly; the puppet modules are packed into the undercloud node by DIB and then spawned with virsh. You may have better luck just grabbing master of the aodh module and shoving it into your undercloud node directly. | 20:51 |
EmilienM | slagle: where is init-keystone ? | 20:52 |
EmilienM | oh nevermind | 20:53 |
ayoung | egafford, what do you mean by "grabbing master of the aodh module and shoving it into your undercloud node directly." | 20:53 |
*** dshulyak_ has quit IRC | 20:53 | |
*** panda_ has quit IRC | 20:53 | |
*** akrivoka|afk has quit IRC | 20:53 | |
*** athomas has quit IRC | 20:53 | |
egafford | ayoung: Fun story: I am not totally sure. Work on the undercloud is newish to me. | 20:53 |
egafford | Let me investigate a bit. | 20:53 |
ayoung | egafford, also, I suspect that I want to destroy the VMs and recreate. Is that just a matter of using vish on the baremetal to delete the VMs and then re-run instack-virt-setup? | 20:54 |
egafford | Yup, it's that. | 20:54 |
egafford | destroy and undefine the lot. | 20:54 |
egafford | Hm; maybe you don't have to. | 20:55 |
EmilienM | ok I have a patch | 20:56 |
openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: run keystone in a wsgi process https://review.openstack.org/270477 | 20:57 |
egafford | You may wish to try export `DIB_INSTALLTYPE_puppet_modules=source` and running `tripleo.sh --undercloud` within your extant virt envs. You may also wish to spin all the way back down. | 20:57 |
EmilienM | ayoung: ^ I'll need to validate orchestration is good though | 20:57 |
egafford | I can't tell you without testing which stage is necessary. | 20:57 |
ayoung | egafford, I realize I had never updated the base OS. Doing that now | 20:57 |
egafford | But from the looks of it, you *should* be able to make it work within your extant virt envs. | 20:57 |
egafford | Ah, yeah, that's importnat. | 20:58 |
egafford | s/that/spelling/ | 20:58 |
*** weshay has joined #tripleo | 21:00 | |
ayoung | rerunning instack-virt-setup | 21:00 |
ayoung | egafford, OK, I've respun the instack-virt-setup. trying the export ^^ | 21:04 |
bnemec | egafford: ayoung: As it happens, there is a way to deploy an undercloud with a proposed patch: http://docs.openstack.org/developer/tripleo-docs/installation/installation.html#installing-the-undercloud | 21:11 |
bnemec | See the second source admonition under step 5. | 21:11 |
ayoung | bnemec, that is exactly what I am right now doing. Thanks | 21:12 |
bnemec | tripleo.sh already sets the puppet installtype to source because that's what we test in CI. | 21:12 |
ayoung | bnemec, egafford ok, undercloud completed without reported error | 21:16 |
bnemec | An auspicious start. :-) | 21:17 |
egafford | Super auspicious. | 21:23 |
*** ayoung has quit IRC | 21:24 | |
*** jayg is now known as jayg|g0n3 | 21:33 | |
egafford | bnemec: https://review.openstack.org/#/c/221418/ has now passed each gate on at least one occasion. I can keep pressing the shiny, red, candy-like recheck button, but your CI lab has other things to do, and could instead just note that set-math-wise, the patch has passed its gates. Thoughts? | 21:33 |
bnemec | egafford: Yeah, I'm fine with merging it, but we had agreed to hold off merging new patches until the ping test merges: https://review.openstack.org/#/c/262028/ | 21:36 |
bnemec | In the interest of not breaking the overcloud again before we can get the testing enabled. | 21:36 |
egafford | bnemec: I respect that completely. :) | 21:37 |
egafford | I just wanted to make sure I shouldn't keep hitting the button. | 21:37 |
bnemec | egafford: Unfortunately CI seems to be really flaky right now. Very few patches are passing all three tests at once. :-( | 21:38 |
openstackgerrit | Andrew Bramley proposed openstack/tripleo-image-elements: Update location of get-pip.py https://review.openstack.org/270494 | 21:38 |
openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Rename validate tox env to linters https://review.openstack.org/258173 | 21:40 |
egafford | bnemec: Yeah, I know. We have set math CI problems in Sahara sometimes too. Lots of moving parts. | 21:40 |
*** gfidente|afk has quit IRC | 21:40 | |
bnemec | Engine went down during stack CREATE | 21:42 |
bnemec | Bleh, I think we're going to have to bump our undercloud size again. | 21:42 |
bnemec | We must have undone the benefits of the extra memory with the vcpu bump. | 21:43 |
*** jdob_lt has quit IRC | 21:45 | |
*** jdob_lt has joined #tripleo | 21:45 | |
*** penick has quit IRC | 21:46 | |
*** panda_ has joined #tripleo | 21:48 | |
*** davidlenwell has quit IRC | 21:52 | |
*** davidlenwell has joined #tripleo | 21:53 | |
*** ChanServ sets mode: +v davidlenwell | 21:53 | |
openstackgerrit | Merged openstack/instack-undercloud: Allow setting the hostname during undercloud install https://review.openstack.org/249457 | 21:59 |
*** davidlenwell has quit IRC | 22:00 | |
*** athomas has joined #tripleo | 22:01 | |
*** akrivoka|afk has joined #tripleo | 22:01 | |
*** davidlenwell has joined #tripleo | 22:02 | |
*** ChanServ sets mode: +v davidlenwell | 22:02 | |
*** penick has joined #tripleo | 22:05 | |
*** jdob_lt has quit IRC | 22:10 | |
*** athomas has quit IRC | 22:11 | |
*** akrivoka|afk has quit IRC | 22:11 | |
*** trozet has quit IRC | 22:11 | |
*** trown|mtg is now known as trown|outttypeww | 22:11 | |
*** athomas has joined #tripleo | 22:11 | |
*** akrivoka|afk has joined #tripleo | 22:11 | |
*** akrivoka|afk has quit IRC | 22:12 | |
*** egafford has quit IRC | 22:14 | |
*** egafford has joined #tripleo | 22:15 | |
*** lblanchard has quit IRC | 22:17 | |
*** ayoung has joined #tripleo | 22:20 | |
*** athomas has quit IRC | 22:22 | |
*** thrash is now known as thrash|g0ne | 22:22 | |
*** tiswanso has quit IRC | 22:25 | |
*** tiswanso has joined #tripleo | 22:26 | |
*** tiswanso has quit IRC | 22:26 | |
openstackgerrit | Merged openstack/diskimage-builder: Move speedup section to image building guide https://review.openstack.org/268769 | 22:27 |
*** weshay has quit IRC | 22:47 | |
*** alop has joined #tripleo | 22:54 | |
alop | Can I get some eyes on https://review.openstack.org/#/c/266016/ please? | 22:55 |
*** cwolferh has quit IRC | 22:59 | |
bnemec | The CI test for overcloud ping test has passed CI. We should merge it ASAP: https://review.openstack.org/#/c/262028/6 | 23:01 |
*** weshay has joined #tripleo | 23:01 | |
*** regebro has quit IRC | 23:04 | |
*** mordred has quit IRC | 23:04 | |
bnemec | alop: Does that actually work in RHEL 6? It's not an accident that the repo name was hard-coded - rhel 6 has its own registration script: https://github.com/openstack/diskimage-builder/blob/master/elements/rhel-common/pre-install.d/00-rhsm | 23:05 |
*** regebro has joined #tripleo | 23:05 | |
*** rasca has quit IRC | 23:05 | |
bnemec | I have no idea why they're both in rhel-common. | 23:05 |
alop | bnemec: yes, if using Satellite for registration, you have to have the repos named appropriately | 23:06 |
*** mordred has joined #tripleo | 23:06 | |
*** rasca has joined #tripleo | 23:06 | |
alop | otherwise, subscription-manager tries to enable rhel-7-server-rpms AND rhel-6-server-rpms, which causes it to fail | 23:07 |
bnemec | alop: Right, I guess my concern is that the script was never supposed to work on rhel-6. That distro is supposed to use https://github.com/openstack/diskimage-builder/blob/master/elements/rhel-common/pre-install.d/00-rhsm | 23:08 |
bnemec | And since that script happens to be updated to support rhel7 too, if we then add support to the new one for rhel6 there will essentially be two completely separate methods of registering both distro version. | 23:09 |
bnemec | *versions | 23:09 |
alop | I don't think the rhel element actually uses 00-rhsm, or at least, it doesn't NOT use 00-rhel-registration | 23:12 |
alop | at the minimum, introducing the RHEL_MAJ_VER in the environment.d phase would allow us to further clean up all the 6's and 7's | 23:14 |
*** trozet has joined #tripleo | 23:23 | |
openstackgerrit | Andrew Bramley proposed openstack/tripleo-image-elements: Pin pip to 7.1.2 in heat-cfntools elements https://review.openstack.org/270526 | 23:25 |
*** eil397 has joined #tripleo | 23:26 | |
*** nico_auv has quit IRC | 23:38 | |
*** athomas has joined #tripleo | 23:41 | |
*** trozet has quit IRC | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!