| openstackgerrit | David Moreau Simard proposed openstack/ara feature/1.0: Allow configuring logging via config file https://review.openstack.org/498239 | 00:09 |
|---|---|---|
| openstackgerrit | David Moreau Simard proposed openstack/ara feature/1.0: WIP: Start using the API in the callback https://review.openstack.org/495568 | 00:09 |
| openstackgerrit | David Moreau Simard proposed openstack/ara feature/1.0: WIP: Start using the API in the callback https://review.openstack.org/495568 | 01:46 |
| openstackgerrit | Merged openstack/ara feature/1.0: Allow configuring logging via config file https://review.openstack.org/498239 | 01:54 |
| *** dmsimard has quit IRC | 06:50 | |
| *** ara-slack has quit IRC | 06:50 | |
| *** ara-discord has quit IRC | 06:51 | |
| *** dmsimard has joined #ara | 07:20 | |
| *** ChanServ sets mode: +o dmsimard | 07:20 | |
| *** resmo has joined #ara | 08:01 | |
| *** sshnaidm is now known as sshnaidm|afk | 11:05 | |
| *** resmo has quit IRC | 11:56 | |
| *** ara-slack has joined #ara | 12:14 | |
| *** ara-slack has joined #ara | 12:14 | |
| *** ChanServ sets mode: +v ara-slack | 12:14 | |
| *** ara-discord has joined #ara | 12:14 | |
| *** ara-discord has joined #ara | 12:14 | |
| *** ChanServ sets mode: +v ara-discord | 12:14 | |
| *** Mike_201 has joined #ara | 12:50 | |
| *** tristanC has joined #ara | 12:52 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:08 | |
| *** resmo has joined #ara | 13:10 | |
| *** tbielawa has joined #ara | 13:13 | |
| *** sshnaidm is now known as sshnaidm|bbl | 13:40 | |
| *** tbielawa is now known as tbielawa|coffee | 14:36 | |
| ara-slack | <nullah> Hi, I run playbooks as sudoer "ansible", but I get the following permission error with ara configured: | 14:43 |
| ara-slack | <nullah> ```ansible.errors.AnsibleError: Unable to create local directories(/var/www/ara/.ansible/tmp): [Errno 13] Permission denied: '/var/www/ara/.ansible/tmp'``` | 14:43 |
| ara-slack | <nullah> everything in /var/www/ara is owned by apache:apache as suggested, I tried adding "ansible" to the apache group (as a secondary group) but no luck. Any help would be highly appreciated! Thanks. | 14:44 |
| ara-slack | <chief_wrench> can you provide the output of “ls -la /var/www/ara/.ansible” please? | 15:05 |
| ara-slack | <nullah> yes sure | 15:06 |
| ara-slack | <nullah> ```ls -la /var/www/ara/.ansible total 12 drwx------. 3 apache apache 4096 Jun 8 22:21 . drwxr-xr-x. 4 apache apache 4096 Aug 28 14:01 .. drwx------. 120 apache apache 4096 Aug 28 14:27 tmp``` | 15:06 |
| ara-slack | <chief_wrench> grep www /etc/group | 15:07 |
| ara-slack | <nullah> no output | 15:07 |
| *** tbielawa|coffee is now known as tbielawa | 15:08 | |
| ara-slack | <chief_wrench> ah no. the group apache can not write in /var/www/ara/.ansible | 15:08 |
| ara-slack | <nullah> ah ok | 15:08 |
| ara-slack | <chief_wrench> chmod 760 /var/www/ara/.ansible | 15:08 |
| ara-slack | <nullah> ok | 15:08 |
| ara-slack | <dmsimard> Setting up WSGI is a bit tricky, the docs have seemingly innocent bits that are actually important and should be followed | 15:09 |
| ara-slack | <dmsimard> I'm making this better and less awkward in 1.0, promise :) | 15:09 |
| ara-slack | <nullah> For sure, I must have missed a step | 15:09 |
| ara-slack | <nullah> ok nice! :slightly_smiling_face: | 15:09 |
| ara-slack | <nullah> so chmod 760 did not resolve the issue, should i do a recursive chmod on /var/ww/ara/.ansible ? | 15:11 |
| ara-slack | <dmsimard> There was an Ansible role to set things up that was contributed by a user that I adopted upstream: https://github.com/openstack/ansible-role-ara | 15:11 |
| ara-slack | <nullah> ok cool. i initially deployed with this role actually | 15:11 |
| ara-slack | <chief_wrench> naw that is a smart move: deploy ara with ansible! :slightly_smiling_face: | 15:12 |
| ara-slack | <dmsimard> I never really got around to spend enough time to get it going properly, definitely accepting contributions | 15:12 |
| ara-slack | <nullah> :+1: | 15:12 |
| ara-slack | <dmsimard> @nullah are you setting this up on a RHEL-based distro or Debian-based ? | 15:14 |
| ara-slack | <nullah> RHEL-based | 15:14 |
| ara-slack | <nullah> centos 7.3.1611 | 15:15 |
| ara-slack | <dmsimard> Do you have selinux enforcing ? There's a selinux boolean to toggle | 15:15 |
| ara-slack | <nullah> Yep I toggled it, and then i even set it to permissive | 15:16 |
| ara-slack | <nullah> then rebooted | 15:16 |
| ara-slack | <nullah> please note, when i run a playbook as root, ara works | 15:16 |
| ara-slack | <nullah> but my playbook does not since i configured my environment with "ansible" user to run the playbooks | 15:16 |
| ara-slack | <nullah> (ansible user has passwordless ssh access to hosts) | 15:17 |
| ara-slack | <dmsimard> Ok -- so the general idea is that every component (callback, CLI, webapp) need to know where the database is and have access to it. | 15:17 |
| ara-slack | <dmsimard> Ultimately it doesn't matter if your ansible.cfg file is in, for example, /etc/ansible/ansible.cfg -- so long as the callback and the webapp have access to it (and then have read/write access to the required locations) | 15:18 |
| ara-slack | <nullah> true | 15:18 |
| ara-slack | <dmsimard> If you're getting data to ARA when running ansible as root and it doesn't work when running ansible as another user, maybe it's not picking up the right database location or it doesn't have access to it. | 15:19 |
| ara-slack | <nullah> I see. By the way this is the error msg: | 15:19 |
| ara-slack | <nullah> ```ansible.errors.AnsibleError: Unable to create local directories(/var/www/ara/.ansible/tmp): [Errno 13] Permission denied: '/var/www/ara/.ansible/tmp'``` | 15:19 |
| ara-slack | <dmsimard> ARA defaults the database location in ~/.ara/ansible.sqlite -- you probably have data there for the user | 15:19 |
| ara-slack | <dmsimard> Where is that from, though ? Is that from your user running ansible ? from apache ? from ansible running as root ? | 15:20 |
| ara-slack | <nullah> that is when i run a playbook as my ansible user | 15:20 |
| ara-slack | <dmsimard> It's not necessary to make your user ansible playbook runs use that tmp directory -- but if you want to have everything in the same place, you probably need to add your user to the apache group and then chmod 770 | 15:21 |
| ara-slack | <dmsimard> the ansible tmp directory defaults to ~/.ansible/tmp iirc | 15:22 |
| ara-slack | <dmsimard> (This directory and having access to it is already no longer necessary in 1.0 btw) | 15:22 |
| dmsimard | wow already 13 people on slack | 15:22 |
| ara-slack | <nullah> thanks! let me try that. and looking forward to 1.0 :slightly_smiling_face: | 15:23 |
| ara-slack | <nullah> btw i did change the tmp directory previously | 15:23 |
| ara-slack | <dmsimard> @nullah typically, the ansible.cfg file in /var/www/ara is only used by apache/webapp | 15:23 |
| ara-slack | <nullah> i mean, i tried ~/.ansible/tmp | 15:23 |
| ara-slack | <dmsimard> and then you have a different ansible.cfg file for actually running playbooks | 15:24 |
| ara-slack | <nullah> Ah ok I see. this morning i removed (and backed) that ansible.cfg and consolidated everything in /etc/home/ansible.cfg | 15:25 |
| ara-slack | <nullah> ```local_tmp = /var/www/ara/.ansible/tmp``` | 15:25 |
| ara-slack | <nullah> i had tried local | 15:25 |
| ara-slack | <nullah> when i changed local_tmp back to "~/.ansible/tmp" i got another permission error, it was: | 15:27 |
| ara-slack | <nullah> ```ERROR! Unexpected Exception: [Errno 13] Permission denied: u'/var/www/ara/.ara/ara.log'``` | 15:27 |
| ara-slack | <dmsimard> IMO leave the web app alone, give it that default config from the docs so it can work (and point to the right database location) and then configure your user's ansible to use the callback and the database location separately. | 15:28 |
| ara-slack | <nullah> ok! | 15:28 |
| ara-slack | <dmsimard> Trying to make both the web app and the user ansible work from the same config in the same directories will just make it harder on yourself | 15:28 |
| ara-slack | <nullah> true | 15:28 |
| ara-slack | <nullah> so my user ansible.cfg needs callback info | 15:29 |
| ara-slack | <nullah> what about action info | 15:29 |
| ara-slack | <dmsimard> FWIW on the fedora packaging I've been working on for ara, the database ends up in /var/lib/ara, the config in /etc/ara | 15:29 |
| ara-slack | <nullah> o | 15:30 |
| ara-slack | <nullah> ok | 15:30 |
| ara-slack | <dmsimard> library and action_plugins is necessary if you plan on using the ara_record module: http://ara.readthedocs.io/en/latest/usage.html#using-the-ara-record-module | 15:30 |
| *** sshnaidm has joined #ara | 15:30 | |
| ara-slack | <dmsimard> it allows you to attach anything to your playbook report, for example: http://ara.readthedocs.io/en/latest/faq.html#arbitrarily-recorded-data | 15:30 |
| ara-slack | <nullah> oh right | 15:30 |
| ara-slack | <dmsimard> If you don't plan on using it, you don't need to specify the action_plugin/library parameters | 15:31 |
| ara-slack | <nullah> ok | 15:31 |
| *** sshnaidm|bbl has quit IRC | 15:33 | |
| ara-slack | <nullah> so i re enabled the /var/www/ara/ansible.cfg | 15:34 |
| ara-slack | <nullah> still get this guy ```ERROR! Unexpected Exception: [Errno 13] Permission denied: u'/var/www/ara/.ara/ara.log'``` | 15:35 |
| ara-slack | <nullah> permissions: | 15:36 |
| ara-slack | <nullah> ```-rw-r--r--. 1 apache apache 72 Jun 8 22:21 ara.log``` | 15:36 |
| *** resmo has quit IRC | 15:37 | |
| ara-slack | <dmsimard> Who gets that ? The user ? | 15:37 |
| ara-slack | <nullah> yes | 15:38 |
| ara-slack | <nullah> when running a playbook | 15:38 |
| ara-slack | <dmsimard> Your user is using /var/www/ara/ansible.cfg ? | 15:38 |
| ara-slack | <nullah> I'm not sure lol | 15:38 |
| ara-slack | <dmsimard> Unless you've exported ANSIBLE_CONFIG to point to /var/www/ara/ansible.cfg, it's probably not using that. There's a hierarchy of files, it's like $CWD/ansible.cfg -> ~/ansible.cfg -> /etc/ansible/ansible.cfg | 15:39 |
| ara-slack | <nullah> ok then it is using /etc/ansible/ansible.cfg | 15:40 |
| ara-slack | <dmsimard> Okay, I guess in that config file you set ara_dir to /var/www/ara/.ara and it's trying to write logs there. What are the permissions on /var/www/ara and /var/www/ara/.ara ? | 15:43 |
| ara-slack | <nullah> for /var/www/ara: | 15:44 |
| ara-slack | <nullah> ```drwxr-xr-x. 4 apache apache 4096 Aug 28 15:33 . drwxr-xr-x. 5 root root 4096 Jun 8 19:37 .. drwxrw----. 3 apache apache 4096 Jun 8 22:21 .ansible -rw-r--r--. 1 apache apache 467 Aug 28 15:33 ansible.cfg drwx------. 2 apache apache 4096 Aug 28 14:27 .ara -rwxr-xr-x. 1 apache apache 1662 Jun 8 18:51 ara-wsgi``` | 15:44 |
| ara-slack | <nullah> for /var/www/.ara: (i've added write permission on log in vain) | 15:45 |
| ara-slack | <nullah> ```drwx------. 2 apache apache 4096 Aug 28 14:27 . drwxr-xr-x. 4 apache apache 4096 Aug 28 15:33 .. -rw-r--r--. 1 apache apache 73728 Aug 28 14:27 ansible.sqlite -rw-rw-rw-. 1 apache apache 72 Jun 8 22:21 ara.log``` | 15:45 |
| ara-slack | <nullah> And yes that is indeed the ara_dir value in /etc/ansible/ansible.cfg | 15:46 |
| ara-slack | <nullah> and this is /var/www/ara/ansible.cfg: | 15:47 |
| ara-slack | <nullah> ```[defaults] # This directory is required to store temporary files for Ansible and ARA local_tmp = /var/www/ara/.ansible/tmp callback_plugins = /usr/lib/python2.7/site-packages/ara/plugins/callbacks host_key_checking = False action_plugins = /usr/lib/python2.7/site-packages/ara/plugins/actions library = /usr/lib/python2.7/site-packages/ara/plugins/modules [ara] # This will default the database and logs location to be inside that directory. d | 15:47 |
| ara-slack | = /var/www/ara/.ara``` | 15:47 |
| ara-slack | <nullah> maybe i put too much info in it? anyways i'm starting to feel bad this is taking up too much of your time lol | 15:49 |
| ara-slack | <nullah> if nobody else experienced this i must have missed a step in configuration | 15:51 |
| ara-slack | <dmsimard> Looks like .ara is 700 | 15:52 |
| ara-slack | <dmsimard> So your user would not have the necessary rights to write there | 15:52 |
| ara-slack | <nullah> ah of course! let me modify that then | 15:53 |
| ara-slack | <dmsimard> No worries, happy to help. Like I said, I know this is a bit clunky and definitely on the to-do for 1.0. WSGI support was something added "after the fact", it wasn't implemented originally so that's why it's a bit awkward. | 15:54 |
| ara-slack | <nullah> folder /var/www/ara has the following permission: | 15:54 |
| ara-slack | <nullah> ```drwxr-xr-x. 4 apache apache 4096 Aug 28 15:33 ara``` | 15:54 |
| ara-slack | <nullah> so i need to add w | 15:54 |
| ara-slack | <dmsimard> that's /var/www/ara, what about /var/www/ara/.ara ? | 15:54 |
| ara-slack | <nullah> oops sorry .ara | 15:54 |
| ara-slack | <nullah> yes exactly what you said | 15:55 |
| ara-slack | <nullah> changed it now | 15:55 |
| ara-slack | <dmsimard> 750 should be sufficient for the user to be able to enter the directory if he's member of the apache group | 15:55 |
| ara-slack | <nullah> ok | 15:56 |
| ara-slack | <nullah> i just did a chmod a+rw | 15:56 |
| ara-slack | <dmsimard> after that, the files need to be written to, so you'll need to set the permissions and ownership accordingly | 15:56 |
| ara-slack | <dmsimard> So, thinking back... I'm realizing that, usually, when ARA is deployed with WSGI, it's because you'll be using some form of centralized setup (with MySQL).. So you're not actually running Ansible from that node, you're using Ansible from other nodes and pushing data to the MySQL database. | 15:57 |
| ara-slack | <nullah> oh I see | 15:58 |
| ara-slack | <dmsimard> WSGI and MySQL are interesting for larger-scale deployments or when you want to aggregate results from different nodes across different servers | 15:58 |
| ara-slack | <nullah> i put everything on my control machine, and using sqlite | 15:58 |
| ara-slack | <nullah> out of the box | 15:58 |
| ara-slack | <dmsimard> But it's probably overkill for simple setups | 15:58 |
| ara-slack | <nullah> I see | 15:58 |
| ara-slack | <dmsimard> If you're running on a smaller scale, you could just run the embedded webserver. Some users I know have set up a systemd unit file that manages that. Some users also use nginx just to reverse proxy the app. | 15:59 |
| *** tbielawa is now known as tbielawa|lunch | 15:59 | |
| ara-slack | <dmsimard> I guess it depends on the use case. For, example, we rely on ARA a lot for CI and we generate static reports at the end of the jobs and then attach that static report with the job logs. The job logs are hosted and the report is available there. | 16:00 |
| ara-slack | <dmsimard> For example: http://logs.openstack.org/80/495880/10/check/gate-openstack-ansible-openstack-ansible-ceph-ubuntu-xenial/ee2be12/logs/ara/ | 16:00 |
| ara-slack | <nullah> Cool | 16:00 |
| ara-slack | <nullah> well then... I think I'll wait for 1.0 , i still get the permission error on ara.log even though i just made it 777 lol | 16:05 |
| ara-slack | <nullah> So I'm obviously doing something wrong. But really looking forward to 1.0 and thank you so much for the help!! | 16:05 |
| ara-slack | <dmsimard> Ah, too bad, we should totally make this work. Is selinux still permissive if you do "getenforce" ? | 16:21 |
| ara-slack | <nullah> :slightly_smiling_face: | 16:22 |
| ara-slack | <nullah> yes its still permissive | 16:22 |
| ara-slack | <nullah> and just to recap when running playbook as user i get | 16:22 |
| ara-slack | <nullah> ```IOError: [Errno 13] Permission denied: u'/var/www/ara/.ara/ara.log'``` | 16:22 |
| ara-slack | <dmsimard> And your user is member of the "apache" group ? | 16:22 |
| ara-slack | <nullah> yes apparently: | 16:26 |
| ara-slack | <nullah> ```$ groups ansible ansible : ansible apache``` | 16:26 |
| ara-slack | <nullah> furthermore permissiong on ara.log are: | 16:26 |
| ara-slack | <nullah> ```-rwxrwxrwx. 1 apache apache 72 Jun 8 22:21 /var/www/ara/.ara/ara.log``` | 16:27 |
| ara-slack | <nullah> still need to run ls -al with sudo to get that info though... | 16:27 |
| ara-slack | <dmsimard> So maybe there are permission restrictions further up the tree, look at /var/www perhaps | 16:33 |
| ara-slack | <nullah> ah right | 16:36 |
| ara-slack | <nullah> for /var/www : ```drwxr-xr-x. 5 root root 4096 Jun 8 19:37 www``` | 16:36 |
| ara-slack | <dmsimard> "everyone" has read, the folder is 755 so that should be fine | 16:37 |
| ara-slack | <dmsimard> but maybe there is a folder somewhere along the way that doesn't have the right permissions | 16:37 |
| ara-slack | <nullah> contents of /var/www/ara: | 16:38 |
| ara-slack | <nullah> ```$ ls -al /var/www/ara/ total 24 drwxr-xr-x. 4 apache apache 4096 Aug 28 15:33 . drwxr-xr-x. 5 root root 4096 Jun 8 19:37 .. drwxrw----. 3 apache apache 4096 Jun 8 22:21 .ansible -rw-r--r--. 1 apache apache 467 Aug 28 15:33 ansible.cfg drwxrw-rw-. 2 apache apache 4096 Aug 28 14:27 .ara -rwxr-xr-x. 1 apache apache 1662 Jun 8 18:51 ara-wsgi``` | 16:38 |
| ara-slack | <nullah> oh and /var/www/ara itself: ```drwxr-xr-x. 4 apache apache 4096 Aug 28 15:33 ara``` | 16:39 |
| ara-slack | Command sent from Slack by dmsimard: | 16:39 |
| ara-slack | .ara is wrong | 16:39 |
| ara-slack | <dmsimard> in order to enter a directory, you need the execute permission bit | 16:39 |
| ara-slack | <dmsimard> 755 is fine, right now you're in... 744 ? | 16:39 |
| ara-slack | <nullah> ah! good to know :slightly_smiling_face: | 16:39 |
| ara-slack | <dmsimard> or 766 | 16:40 |
| ara-slack | <dmsimard> do a chmod 755 on .ara | 16:40 |
| ara-slack | <nullah> ok! | 16:40 |
| ara-slack | <dmsimard> I mean, if you really want to restrict things, it needs to be at least 750 | 16:41 |
| ara-slack | <nullah> works | 16:41 |
| ara-slack | <nullah> :slightly_smiling_face: | 16:41 |
| ara-slack | <dmsimard> Yay \o/ | 16:41 |
| ara-slack | <nullah> :the_horns: | 16:41 |
| ara-slack | <nullah> i get this pink error msg though but playbook runs | 16:41 |
| ara-slack | <nullah> ```[WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin (<ansible.plugins.callback./usr/lib/python2.7/site- packages/ara/plugins/callbacks/log_ara.CallbackModule object at 0x2921e90>)``` | 16:41 |
| ara-slack | <nullah> but i can try troubleshooting that myself for a bit haha | 16:42 |
| ara-slack | <nullah> other pink messages | 16:42 |
| ara-slack | <dmsimard> put 755 on .ansible (and .ansible/tmp/) too | 16:42 |
| ara-slack | <nullah> ```[WARNING]: Failure using method (v2_runner_on_skipped) in callback plugin (<ansible.plugins.callback./usr/lib/python2.7/site- packages/ara/plugins/callbacks/log_ara.CallbackModule object at 0x2921e90>): 'NoneType' object has no attribute 'name'``` | 16:42 |
| ara-slack | <nullah> ok | 16:42 |
| ara-slack | <dmsimard> ansible/ara is probably trying to read/write to that directory and it can't | 16:42 |
| ara-slack | <dmsimard> it really boils down to permissions and making sure the webapp and the callback can access things | 16:43 |
| ara-slack | <dmsimard> you don't need to make it hard on yourself if you're just trying ara out | 16:43 |
| ara-slack | <nullah> true | 16:43 |
| ara-slack | <dmsimard> Don't bother with wsgi/mysql unless you get to a point where you need them :slightly_smiling_face: | 16:44 |
| ara-slack | <nullah> true! I deployed with the openstack/ara playbook | 16:44 |
| ara-slack | <nullah> so not sure what it did | 16:44 |
| ara-slack | <nullah> :P | 16:44 |
| ara-slack | <dmsimard> yeah, it's probably not doing everything properly, sorry about that. There *is* a disclaimer "Please note that this role is still a work in progress and in development." though :P | 16:45 |
| ara-slack | <dmsimard> There are some ara roles here too https://github.com/BonnyCI/hoist/tree/master/roles | 16:45 |
| ara-slack | <dmsimard> They deploy ara with uwsgi though, not mod_wsgi | 16:45 |
| ara-slack | <dmsimard> I need to get them to contribute their stuff back upstream :slightly_smiling_face: | 16:45 |
| ara-slack | <nullah> ehehe yeah i did it at my own risk I guess! | 16:45 |
| ara-slack | <nullah> i should probably redeploy it from scratch following the docs | 16:46 |
| ara-slack | <nullah> in order to do that id just delete /var/www/ara recursively? | 16:46 |
| ara-slack | <nullah> and then start over? anything else i should delete? | 16:46 |
| ara-slack | <dmsimard> nothing else I don't think | 16:47 |
| ara-slack | <nullah> ok great! ill probably do that. | 16:47 |
| ara-slack | <nullah> cause im still getting pink messages | 16:47 |
| ara-slack | <dmsimard> I don't think I've really tested running Ansible from the same box as the webapp in a WSGI scenario so that might be uncharted territory you're in | 16:49 |
| ara-slack | <dmsimard> Happy to accept patches for docs and other things :slightly_smiling_face: | 16:49 |
| dmsimard | Slack translates ":)" to a slight_smiling_face macro eh | 16:50 |
| ara-slack | <nullah> yeah eheh, almost a Smize | 16:53 |
| ara-slack | <nullah> Anyways thanks again. | 16:56 |
| ara-slack | <dmsimard> np, happy to help | 16:56 |
| ara-slack | <dmsimard> @nullah any cloudops guys going to denver ? | 16:56 |
| ara-slack | <nullah> not that i know of nope | 16:57 |
| ara-slack | <nullah> you are going i imagine? | 16:57 |
| ara-slack | <nullah> Sorry, what's in denver? lol I thought you meant ansiblefest but thats in SF looks like | 16:59 |
| ara-slack | <dmsimard> OpenStack PTG :slightly_smiling_face: | 16:59 |
| ara-slack | <dmsimard> AnsibleFest is just before the PTG in SFO yeah | 17:00 |
| ara-slack | <nullah> ah that was what i was just googling ok lol | 17:00 |
| ara-slack | <nullah> Not that I know of no. | 17:00 |
| ara-slack | <dmsimard> I almost got to go to AnsibleFest but it was too expensive to send me to both AnsibleFest *and* the PTG :P | 17:00 |
| ara-slack | <nullah> ehehehe | 17:00 |
| ara-slack | <nullah> We went to Boston back in May for the Summit, was pretty great. :slightly_smiling_face: | 17:02 |
| *** tbielawa|lunch is now known as tbielawa | 17:09 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:21 | |
| *** Mike_201 has quit IRC | 18:22 | |
| *** tbielawa has quit IRC | 20:52 | |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Test https://review.openstack.org/498237 | 23:28 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!