| *** jmcbride has joined #openstack-dns | 00:03 | |
| *** _ducttape_ has quit IRC | 00:09 | |
| *** bpokorny has quit IRC | 00:21 | |
| *** james_li has quit IRC | 00:23 | |
| *** rudrajit_ has quit IRC | 00:24 | |
| *** km has joined #openstack-dns | 00:27 | |
| *** bpokorny has joined #openstack-dns | 00:33 | |
| *** jmcbride has quit IRC | 00:33 | |
| *** bpokorny_ has joined #openstack-dns | 00:58 | |
| *** bpokorny has quit IRC | 01:01 | |
| *** ducttape_ has joined #openstack-dns | 01:03 | |
| *** ducttape_ has quit IRC | 01:18 | |
| *** penick has joined #openstack-dns | 01:19 | |
| *** penick has quit IRC | 01:28 | |
| *** stanzgy has joined #openstack-dns | 01:30 | |
| *** stanzgy has quit IRC | 01:39 | |
| *** stanzgy has joined #openstack-dns | 01:39 | |
| *** rudrajit has joined #openstack-dns | 01:40 | |
| *** amit213 has joined #openstack-dns | 01:41 | |
| *** bpokorny_ has quit IRC | 01:41 | |
| *** bpokorny has joined #openstack-dns | 01:42 | |
| *** ducttape_ has joined #openstack-dns | 01:43 | |
| *** openstackgerrit has quit IRC | 01:47 | |
| *** openstackgerrit has joined #openstack-dns | 01:47 | |
| *** ChanServ sets mode: +v openstackgerrit | 01:47 | |
| *** fawadkhaliq has quit IRC | 01:53 | |
| *** jasonsb has joined #openstack-dns | 01:56 | |
| *** jasonsb has quit IRC | 02:02 | |
| *** mariusv_ has joined #openstack-dns | 02:19 | |
| *** ducttape_ has quit IRC | 02:23 | |
| *** stanzgy has quit IRC | 02:23 | |
| *** mariusv has quit IRC | 02:23 | |
| *** ducttape_ has joined #openstack-dns | 02:34 | |
| *** stanzgy has joined #openstack-dns | 02:38 | |
| *** bpokorny has quit IRC | 03:08 | |
| *** ccneill has quit IRC | 03:14 | |
| *** f1ller has joined #openstack-dns | 03:25 | |
| *** JonathanS has joined #openstack-dns | 03:26 | |
| *** ducttape_ has quit IRC | 03:26 | |
| *** JonathanD has quit IRC | 03:26 | |
| *** filler has quit IRC | 03:26 | |
| *** clayton has quit IRC | 03:26 | |
| *** richm has quit IRC | 03:26 | |
| *** HenryG has quit IRC | 03:26 | |
| *** HenryG has joined #openstack-dns | 03:26 | |
| *** richm has joined #openstack-dns | 03:27 | |
| *** clayton has joined #openstack-dns | 03:28 | |
| *** jet-rongl has joined #openstack-dns | 03:32 | |
| *** a7ndrew has quit IRC | 03:37 | |
| *** a7ndrew has joined #openstack-dns | 03:37 | |
| *** rsyed_away is now known as rsyed | 04:12 | |
| *** rsyed is now known as rsyed_away | 04:16 | |
| *** jasonsb has joined #openstack-dns | 04:45 | |
| *** amit213 has quit IRC | 06:33 | |
| *** richm has quit IRC | 06:35 | |
| *** stanzgy_ has joined #openstack-dns | 07:03 | |
| *** amit213 has joined #openstack-dns | 07:04 | |
| *** stanzgy has quit IRC | 07:05 | |
| *** amit213 has quit IRC | 07:06 | |
| *** amit213 has joined #openstack-dns | 07:08 | |
| *** amit213 has quit IRC | 07:10 | |
| *** amit213 has joined #openstack-dns | 07:15 | |
| *** jasonsb has quit IRC | 07:16 | |
| *** amit213 has quit IRC | 07:18 | |
| *** boris-42 has quit IRC | 07:34 | |
| *** boris-42 has joined #openstack-dns | 07:35 | |
| *** stanzgy_ has quit IRC | 07:39 | |
| *** amit213 has joined #openstack-dns | 07:40 | |
| *** amit213 has quit IRC | 07:42 | |
| *** pcaruana has joined #openstack-dns | 07:45 | |
| *** jasonsb has joined #openstack-dns | 07:47 | |
| *** jasonsb has quit IRC | 07:52 | |
| *** amit213 has joined #openstack-dns | 08:09 | |
| *** amit213 has quit IRC | 08:13 | |
| *** amit213 has joined #openstack-dns | 08:16 | |
| *** amit213 has quit IRC | 08:18 | |
| *** stanzgy has joined #openstack-dns | 08:21 | |
| *** amit213 has joined #openstack-dns | 08:24 | |
| *** rudrajit has quit IRC | 08:47 | |
| *** jasonsb has joined #openstack-dns | 08:51 | |
| *** kei_yama has quit IRC | 08:52 | |
| *** jasonsb has quit IRC | 08:55 | |
| *** nyechiel has joined #openstack-dns | 09:04 | |
| *** jet-rongl has quit IRC | 09:05 | |
| *** jschwarz has joined #openstack-dns | 09:06 | |
| *** ducttape_ has joined #openstack-dns | 09:08 | |
| *** ducttape_ has quit IRC | 09:12 | |
| *** jordanP has joined #openstack-dns | 09:14 | |
| *** mpbnka has joined #openstack-dns | 09:49 | |
| mpbnka | Hi, I have a question regarding how to configure designate with multiple pools | 09:50 |
|---|---|---|
| *** mpbnka has quit IRC | 09:53 | |
| *** mpbnka has joined #openstack-dns | 09:53 | |
| *** openstackgerrit_ has joined #openstack-dns | 09:53 | |
| *** mpbnka has quit IRC | 09:53 | |
| *** mpbnka has joined #openstack-dns | 09:54 | |
| mpbnka | Hi, I want to configure designate with multiple pools. I have a single pool setup configured and it is working good | 09:56 |
| mpbnka | how to extend this to manage multiple pools | 09:56 |
| *** openstackgerrit_ has quit IRC | 10:02 | |
| *** boris-42 has quit IRC | 10:04 | |
| *** jordan_ has joined #openstack-dns | 10:16 | |
| *** jordanP has quit IRC | 10:20 | |
| *** km has quit IRC | 10:22 | |
| *** km has joined #openstack-dns | 10:25 | |
| *** stanzgy has quit IRC | 10:37 | |
| *** jasonsb has joined #openstack-dns | 10:52 | |
| *** at-work has joined #openstack-dns | 10:54 | |
| at-work | Hello. I'm trying to get designate working on CentOS 7, with Bind 9, and am finding the documentation quite confusing. I understand that the agent should write files to /var/lib/designate, but I'm not seeing anything there. What should I check first? | 10:55 |
| *** jasonsb has quit IRC | 10:57 | |
| openstackgerrit | Merged openstack/designate: Updated from global requirements https://review.openstack.org/285016 | 11:30 |
| openstackgerrit | Merged openstack/designate-dashboard: Updated from global requirements https://review.openstack.org/285017 | 11:31 |
| openstackgerrit | Merged openstack/designate: Improve error handling for TCP connections https://review.openstack.org/286555 | 11:31 |
| openstackgerrit | Merged openstack/designate: Improve error handling for TCP connections https://review.openstack.org/286557 | 11:33 |
| *** openstackgerrit has quit IRC | 12:03 | |
| *** openstackgerrit has joined #openstack-dns | 12:03 | |
| *** ChanServ sets mode: +v openstackgerrit | 12:03 | |
| *** nyechiel has quit IRC | 12:30 | |
| *** nyechiel has joined #openstack-dns | 12:34 | |
| *** nyechiel_ has joined #openstack-dns | 12:48 | |
| *** nyechiel has quit IRC | 12:48 | |
| *** jasonsb has joined #openstack-dns | 12:54 | |
| *** nyechiel has joined #openstack-dns | 12:57 | |
| *** nyechiel_ has quit IRC | 12:58 | |
| *** jasonsb has quit IRC | 12:59 | |
| *** ducttape_ has joined #openstack-dns | 13:13 | |
| *** stanzgy has joined #openstack-dns | 13:18 | |
| *** km has quit IRC | 13:30 | |
| *** stanzgy has quit IRC | 13:32 | |
| *** ducttape_ has quit IRC | 13:33 | |
| *** richm has joined #openstack-dns | 13:43 | |
| *** mpbnka has quit IRC | 14:07 | |
| *** jordan__ has joined #openstack-dns | 14:13 | |
| *** rsyed_away is now known as rsyed | 14:16 | |
| *** stanzgy has joined #openstack-dns | 14:17 | |
| *** jordan_ has quit IRC | 14:17 | |
| *** stanzgy has quit IRC | 14:40 | |
| *** stanzgy has joined #openstack-dns | 14:46 | |
| *** mlavalle has joined #openstack-dns | 14:53 | |
| *** jmcbride has joined #openstack-dns | 14:53 | |
| *** pglass has joined #openstack-dns | 14:58 | |
| *** johnbelamaric has joined #openstack-dns | 15:00 | |
| *** ducttape_ has joined #openstack-dns | 15:03 | |
| *** stanzgy has quit IRC | 15:17 | |
| *** jmcbride has quit IRC | 15:19 | |
| *** jmcbride has joined #openstack-dns | 15:19 | |
| elarson | mugsie: thanks for the adding me to the core groups in launchpad! | 15:23 |
| mugsie | and, if you log into gerrit, you should have a few extra buttons :P | 15:24 |
| elarson | I noticed and managed to avoid pressing them to see what happens ;) | 15:24 |
| *** jasonsb has joined #openstack-dns | 15:25 | |
| rsyed | http://i.giphy.com/kFWjojQsWdSAU.gif | 15:25 |
| elarson | !m rsyed | 15:26 |
| openstack | elarson: Error: "m" is not a valid command. | 15:26 |
| elarson | never change openstack! | 15:26 |
| *** darkxploit has joined #openstack-dns | 15:31 | |
| *** stanzgy has joined #openstack-dns | 15:37 | |
| *** stanzgy has quit IRC | 15:42 | |
| eandersson | wtf | 15:42 |
| eandersson | https://github.com/openstack/designate/blob/master/designate/service.py#L342 | 15:43 |
| eandersson | wtf | 15:43 |
| eandersson | -_- | 15:45 |
| eandersson | Kiall, | 15:45 |
| eandersson | -___- | 15:45 |
| eandersson | How can this ever have worked | 15:45 |
| eandersson | You guys are not tracking how much gets sent over TCP | 15:46 |
| eandersson | So as soon as the TCP gets fragmented this will fail | 15:46 |
| eandersson | This NEEDS to be client.sendall | 15:46 |
| eandersson | or you need to track how much was sent e.g. https://github.com/eandersson/amqp-storm/blob/master/amqpstorm/io.py#L107 | 15:47 |
| eandersson | rsyed, | 15:54 |
| eandersson | So if there is a bug with TCP, it is def caused by this issue. | 15:55 |
| rsyed | hey eandersson...yeah i'm trying to understand if this problem ever manifested in our testing: | 15:56 |
| rsyed | so far our only problem had been lack of responsiveness over TCP (but UDP keeps responding) which was consistent with the error handling issue you identified | 15:56 |
| eandersson | This will happen whenever you exceed the mtu | 15:58 |
| eandersson | but this is a MAJOR bug | 15:58 |
| eandersson | like I can't believe that designate has ever worked in a real enviornment | 15:58 |
| eandersson | in a isolated enviornment this will never be an issue | 15:59 |
| eandersson | but as soon as you start hitting a real network, and the payload gets big enough that it get segmented | 15:59 |
| eandersson | this will always fail | 15:59 |
| rsyed | i was about to suggest that payloads may never exceed MTU, but MTU is crazy small in most networks... | 16:00 |
| eandersson | I just spent like 5 hours trying to figure this out | 16:01 |
| eandersson | until I realized that it only happened on sites with over 300 records | 16:01 |
| eandersson | rsyed: Do a patch and make that a sendall asap! :D | 16:03 |
| pglass | eandersson: you should definitely report that bug here: https://bugs.launchpad.net/designate | 16:04 |
| rsyed | eandersson haha i was going to actually let you have the honor this time :-P | 16:04 |
| elarson | rsyed: eandersson is the puppet master! we are his puppets! | 16:09 |
| rsyed | apparently! | 16:09 |
| elarson | that might be other end of the timeout | 16:10 |
| eandersson | It causes a on PDNS side Remote nameserver closed TCP connection | 16:11 |
| elarson | for example, pool manager sends something large enough to go over the mtu, mdns timesout, we don't catch the error | 16:11 |
| elarson | eandersson: and that! | 16:11 |
| elarson | eandersson: are you coming to austin for the summit? | 16:12 |
| eandersson | Yep | 16:12 |
| elarson | eandersson: do you drink alcohol? | 16:12 |
| eandersson | Yep | 16:12 |
| elarson | eandersson: it shall be free!!!!! | 16:12 |
| eandersson | haha | 16:12 |
| eandersson | looking forward to it | 16:12 |
| openstackgerrit | Federico Ceratto proposed openstack/designate: Fix rrset serialization, improve mdns tests https://review.openstack.org/286552 | 16:16 |
| mugsie | elarson: are you going to have a chance to fix the ALIAS patches? | 16:18 |
| mugsie | I can grant a feature freeze exception if it can be done by mitaka release :) | 16:18 |
| elarson | mugsie: I'm working on it today. I'd like to get that transaction refactor merged b/c I can fix the tests more easily then, but if not, nbd. | 16:19 |
| mugsie | thats fine | 16:19 |
| mugsie | just wanted to check it was on the radar | 16:19 |
| elarson | definitely | 16:19 |
| elarson | federico3: thanks for taking a look at that review! | 16:22 |
| mugsie | eandersson: we definitly tested it with zones over the fragmentation size | 16:28 |
| *** ryanpetrello has quit IRC | 16:30 | |
| timsim | I definitely did for MTU of 1500, but they were always listening on loopback :| | 16:34 |
| *** ccneill has joined #openstack-dns | 16:40 | |
| *** jmcbride has quit IRC | 16:42 | |
| eandersson | Yea, but a client.send will not work for sure | 16:43 |
| eandersson | you need to keep track of how much have been sent | 16:43 |
| eandersson | since there is no garuantee that everything was sent with a simple client.send | 16:43 |
| eandersson | unless you use client.sendall | 16:44 |
| mugsie | It has worked - it could be that it is another eventlet overide that is allowing it to act as a sendall | 16:44 |
| mugsie | but we definitly broke the size of a packet in testing this | 16:44 |
| *** darkxploit has quit IRC | 16:46 | |
| mugsie | eandersson: is it failing for you with in a particular way? | 16:48 |
| eandersson | when the payload is too large, it will only send part of the payload when using client.send | 16:48 |
| mugsie | have you seen it fail like that? | 16:49 |
| *** bpokorny has joined #openstack-dns | 16:49 | |
| eandersson | yes | 16:49 |
| eandersson | it's easy to reproduce | 16:49 |
| mugsie | because I have seen it not fail with large responces | 16:50 |
| eandersson | well as long as you have a network | 16:50 |
| eandersson | you could set up a instance in the US, have it talk to a pdns in EU | 16:50 |
| eandersson | then simply add like 400 records | 16:51 |
| eandersson | and the AXFR would fail | 16:51 |
| eandersson | that is obviously an extreme scenario | 16:51 |
| eandersson | but would make it easy to reproduce | 16:51 |
| *** ryanpetrello has joined #openstack-dns | 16:51 | |
| eandersson | "Now we come to the major stumbling block of sockets - send and recv operate on the network buffers. They do not necessarily handle all the bytes you hand them (or expect from them), because their major focus is handling the network buffers. In general, they return when the associated network buffers have been filled (send) or emptied (recv). They then tell you how many bytes they handled. It is your responsibility to call them again until | 16:52 |
| eandersson | your message has been completely dealt with." | 16:52 |
| eandersson | https://docs.python.org/2/howto/sockets.html#socket-howto | 16:52 |
| eandersson | That explains the bug. | 16:53 |
| *** james_li has joined #openstack-dns | 16:55 | |
| mugsie | ok, can you file that, and I will set up a test harness in a bit today, and retest in a geo situat[3~iuon | 16:55 |
| eandersson | https://docs.python.org/2/library/socket.html#socket.socket.sendall | 16:56 |
| eandersson | sounds good mugsie | 16:56 |
| *** ryanpetrello has quit IRC | 17:05 | |
| *** ryanpetrello has joined #openstack-dns | 17:05 | |
| eandersson | but just to make sure we are on the same page, a geo situation would fail for sure, but depending on network setup it can fail anywhere really | 17:07 |
| *** nyechiel has quit IRC | 17:07 | |
| *** jmcbride1 has joined #openstack-dns | 17:08 | |
| mugsie | eandersson: yeah, want to do a geo to make sure it breaks | 17:09 |
| eandersson | sure, but that is just not the call you want to make for this type of all | 17:10 |
| eandersson | *of operation | 17:10 |
| eandersson | as per the python documentation | 17:11 |
| eandersson | but having a test case is always good | 17:11 |
| openstackgerrit | Eric Larson proposed openstack/designate: Move the transaction decorator to storage https://review.openstack.org/283855 | 17:13 |
| *** JonathanS is now known as JonathanD | 17:14 | |
| eandersson | I got a bit worked up over this as this caused me to miss my lunch by a few hours. =] | 17:14 |
| eandersson | Finally able to enjoy my sandwich lol | 17:16 |
| openstackgerrit | Federico Ceratto proposed openstack/designate: Log zone serial on fetch and update https://review.openstack.org/284238 | 17:18 |
| *** rudrajit has joined #openstack-dns | 17:23 | |
| *** rudrajit has quit IRC | 17:27 | |
| *** at-work has quit IRC | 17:27 | |
| *** rudrajit has joined #openstack-dns | 17:27 | |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: WIP: Enable use of Pools YAML https://review.openstack.org/283592 | 17:34 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Add New Pools DB Tables and Code https://review.openstack.org/283591 | 17:34 |
| openstackgerrit | Merged openstack/designate: Move SOA refresh interval into unit tests https://review.openstack.org/283759 | 17:34 |
| *** rudrajit has quit IRC | 17:39 | |
| *** jasonsb has quit IRC | 17:47 | |
| andrewbogott | In designate kilo, I have verbose = False/debug=False and yet mdns is generating 1 Gb of locks per day | 17:51 |
| andrewbogott | *logs | 17:51 |
| andrewbogott | and better yet, it’s generating /two/ logfiles, one in /var/log/designate and one in /var/log/upstart | 17:51 |
| andrewbogott | is there some special config to quiet things down? | 17:52 |
| andrewbogott | here’s a little snip of what mdns is writing: https://dpaste.de/Lj3Z | 17:53 |
| andrewbogott | the other services are also logging noisily | 17:53 |
| *** richm has quit IRC | 17:54 | |
| *** richm has joined #openstack-dns | 17:58 | |
| *** ducttape_ has quit IRC | 17:58 | |
| andrewbogott | Ah, connection_debug = 100 | 18:02 |
| *** mariusv_ has quit IRC | 18:05 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-designateclient: Updated from global requirements https://review.openstack.org/288027 | 18:06 |
| *** mariusv has joined #openstack-dns | 18:06 | |
| *** mariusv has quit IRC | 18:06 | |
| *** mariusv has joined #openstack-dns | 18:06 | |
| *** penick has joined #openstack-dns | 18:09 | |
| mugsie | andrewbogott: yeah - sqla is *very* noisy | 18:10 |
| andrewbogott | All of those "Got lower serial for 'eqiad.wmflabs.' to '208.80.154.12:53'. Expected:'1457028627'. Got:'1457028619'.Retries left=‘1'" | 18:10 |
| andrewbogott | things, those are a bug, right? https://bugs.launchpad.net/designate/+bug/1525019 ? | 18:11 |
| openstack | Launchpad bug 1525019 in Designate "mdns.get_serial_number retries for no reason" [High,In progress] - Assigned to Kiall Mac Innes (kiall) | 18:11 |
| *** jschwarz has quit IRC | 18:17 | |
| *** rudrajit has joined #openstack-dns | 18:17 | |
| mugsie | yeah, it is | 18:17 |
| *** rudrajit has quit IRC | 18:18 | |
| *** rudrajit has joined #openstack-dns | 18:18 | |
| *** ducttape_ has joined #openstack-dns | 18:19 | |
| *** rudrajit has quit IRC | 18:31 | |
| *** rudrajit has joined #openstack-dns | 18:32 | |
| *** ccneill has quit IRC | 18:41 | |
| mugsie | eandersson: ping? | 18:43 |
| mugsie | this seems to be working for me right now - I am doing a "dig @104.42.225.133 -p 5354 largetestzone.tld. AXFR" and it is working | 18:44 |
| mugsie | ;; XFR size: 1242 records (messages 2, bytes 89062) | 18:44 |
| elarson | mugsie: I think mr. andersson has gone home for the day | 18:52 |
| elarson | mugsie: I was curious though if you can fake the mtu in a test and see if you don't get all the data | 18:53 |
| elarson | I don't know if that is a function of the socket's file handle that you can't inspect in python, but I'd be surprised if that is the case | 18:54 |
| mugsie | actually, scratch that - it failed | 18:55 |
| elarson | badoosh! | 18:55 |
| elarson | sweet | 18:55 |
| mugsie | when it got to 3 TCP messages it failed | 18:55 |
| mugsie | and only when it traversed the internet | 18:57 |
| rsyed | created a bug here, i was originally going to incorporate more information from eandersson's debug session, but it's ok for now: https://bugs.launchpad.net/designate/+bug/1552864 | 19:00 |
| openstack | Launchpad bug 1552864 in Designate "TCP connections are not sending their entire buffers" [Undecided,New] | 19:00 |
| *** ducttape_ has quit IRC | 19:06 | |
| *** ducttape_ has joined #openstack-dns | 19:07 | |
| *** ccneill has joined #openstack-dns | 19:08 | |
| elarson | mugsie, timsim, Kiall: btw, if there are any tasks a core needs to do that would be good for me to try out just let me know. (ie +2 workflow +1 on a patch that looks good, etc.) | 19:12 |
| mugsie | elarson: graham.hayes.ie/designate/dashboard is a good start | 19:15 |
| mugsie | there does not seem to be any patches currently needing +W atm though | 19:16 |
| elarson | yeah, I didn't see any either | 19:17 |
| *** ducttape_ has quit IRC | 19:35 | |
| *** jordan__ has quit IRC | 19:52 | |
| elarson | I'm trying to backport my quota fix to kilo and doing tox -e py27 doesn't work b/c of a lack of db migration version files. anyone else ever seen that? | 19:54 |
| *** boris-42 has joined #openstack-dns | 19:54 | |
| timsim | elarson: Yeah, clean out your pycs | 19:56 |
| rsyed | if it's .testrepository again, i'm not sure what to say to you elarson | 19:56 |
| *** johnbelamaric has quit IRC | 19:58 | |
| elarson | rsyed: I deleted it! | 19:58 |
| elarson | timsim: I thought tox did that for me? | 19:58 |
| * elarson will do it of course, just wanting to confirm | 19:59 | |
| elarson | ah ok, not in kilo | 19:59 |
| *** ducttape_ has joined #openstack-dns | 20:00 | |
| elarson | see what I get for assuming? | 20:01 |
| elarson | broken tests | 20:01 |
| openstackgerrit | Eric Larson proposed openstack/designate: Ensure the zone records quota is enforced https://review.openstack.org/288096 | 20:04 |
| timsim | Never assume tox has your back. | 20:11 |
| elarson | it is too bad there isn't a Makefile with a nice make clean target... | 20:12 |
| *** penick has quit IRC | 20:16 | |
| timsim | We should just write a Makefile. Surely openstack wouldn't care. | 20:22 |
| *** penick has joined #openstack-dns | 20:28 | |
| elarson | oh they care | 20:28 |
| elarson | https://review.openstack.org/#/c/183108/10/Makefile | 20:32 |
| * timsim sobs | 20:35 | |
| timsim | Well, all you have to do is convince a couple of us ;) | 20:35 |
| rsyed | you've come a long way in a year, you're now core...you have the power | 20:36 |
| timsim | Surely some project has a damn Makefile | 20:37 |
| rsyed | https://github.com/search?utf8=%E2%9C%93&q=user%3Aopenstack+filename%3AMakefile&type=Code&ref=searchresults | 20:39 |
| timsim | `say done` | 20:39 |
| Kiall | eandersson: myself, federico3 and mugsie (okay, mostly federico3 and mugsie) have been hunched around mugsie's desk... while there's no doubt the code is wrong... evenlet 0.17.4 seems to work fine with it.. anything in the 0.18 series explodes | 20:40 |
| mugsie | looks like https://github.com/eventlet/eventlet/commit/c315ee86dac996ac533b738f7c8777f4d01a0472 is what changed, but in trying to do the correct behavior is causing other issues | 20:41 |
| rsyed | mugsie commented that he required a cooperative yield (sleep(0)) for things to be ok; was that 0.18 as well? | 20:41 |
| mugsie | now, it was a false herring | 20:42 |
| Kiall | red herring ;) | 20:42 |
| mugsie | it masked it enough | 20:42 |
| mugsie | but after hittiing the endpoint repeatedly, it started to fail | 20:42 |
| *** nyechiel has joined #openstack-dns | 20:48 | |
| *** jmcbride1 has quit IRC | 20:53 | |
| *** jmcbride has joined #openstack-dns | 20:54 | |
| *** jmcbride1 has joined #openstack-dns | 20:55 | |
| *** jmcbride has quit IRC | 20:58 | |
| *** james_li_ has joined #openstack-dns | 21:02 | |
| *** james_li_ has quit IRC | 21:03 | |
| *** james_li_ has joined #openstack-dns | 21:04 | |
| *** pglbutt has joined #openstack-dns | 21:04 | |
| *** johnbelamaric has joined #openstack-dns | 21:05 | |
| *** nyechiel has quit IRC | 21:05 | |
| *** pglass has quit IRC | 21:06 | |
| *** james_li has quit IRC | 21:06 | |
| *** ZZelle_ has joined #openstack-dns | 21:14 | |
| *** penick has quit IRC | 21:18 | |
| *** nyechiel has joined #openstack-dns | 21:21 | |
| elarson | so close! `- Failed: 10` | 21:23 |
| *** rudrajit has quit IRC | 21:36 | |
| *** jschwarz has joined #openstack-dns | 21:40 | |
| *** penick has joined #openstack-dns | 21:42 | |
| *** nyechiel has quit IRC | 21:47 | |
| *** penick has quit IRC | 21:53 | |
| *** fawadkhaliq has joined #openstack-dns | 21:55 | |
| *** rudrajit has joined #openstack-dns | 22:01 | |
| *** eandersson_ has joined #openstack-dns | 22:05 | |
| eandersson_ | Kiall: keep in mind that since the TCP buffer is shared on the system, load might impact it as well. | 22:06 |
| eandersson_ | but yea, possible that I am running 0.18 since the Liberty upgrade. | 22:06 |
| eandersson_ | I do however know that this removed any issues on larger sites, after the patch. | 22:07 |
| eandersson_ | *eventlet 0.18 | 22:11 |
| eandersson_ | Looking at the code, it just looks like a normal socket though, so not sure why eventlet would impact it. | 22:13 |
| eandersson_ | https://github.com/openstack/designate/blob/master/designate/utils.py#L460 | 22:14 |
| eandersson_ | Since the issue is not with the python code, but rather the network buffer being full. | 22:14 |
| eandersson_ | I would def put this as a critical issue, as once you reach a certain number of records, this will fail, if you have enough network fragmantation. | 22:15 |
| eandersson_ | ah nvm it was already set to critical :D | 22:16 |
| eandersson_ | but I am not sure if you guys uncovered the right bug, as the bug I am seeing is silent. | 22:17 |
| eandersson_ | Designate does not know that something went wrong, as Python does not see this as en error. | 22:17 |
| eandersson_ | socket.send will just silentently fail, unless you check the bytes sent yourself | 22:17 |
| eandersson_ | e.g. if len(data_to_send) != client.send(): raise IOError('oh wow, not everything got sent') | 22:19 |
| eandersson_ | while client.send(tcp_response) won't raise anything, as it's just returning the number of bytes sent | 22:20 |
| eandersson_ | there is no exceptation that it sent the entire message (as per the python/system specs) | 22:20 |
| eandersson_ | oh man, I just keep going =] | 22:22 |
| eandersson_ | I am on fire today | 22:22 |
| eandersson_ | Will be nice to meet (most of you) in person again in Austin | 22:25 |
| *** jmcbride1 has quit IRC | 22:29 | |
| *** penick has joined #openstack-dns | 22:29 | |
| eandersson_ | Ahh, monkey patching, but even in the eventlet documentation they are using sendall for these type of scenarios | 22:29 |
| eandersson_ | At least it makes more sense to me why it started happening in Liberty. | 22:32 |
| *** rsyed is now known as rsyed_away | 22:34 | |
| *** james_li has joined #openstack-dns | 22:37 | |
| *** james_li_ has quit IRC | 22:41 | |
| *** pglbutt has quit IRC | 22:43 | |
| *** ducttape_ has quit IRC | 22:47 | |
| *** penick has quit IRC | 22:52 | |
| *** jschwarz has quit IRC | 22:54 | |
| *** eandersson_ has quit IRC | 22:56 | |
| *** eandersson_ has joined #openstack-dns | 22:57 | |
| eandersson_ | https://github.com/eventlet/eventlet/issues/260 | 22:57 |
| eandersson_ | Makes a lot of sense, was a bad decision to implement send, similar to sendall. | 22:57 |
| eandersson_ | I am glad they fixed it. | 22:58 |
| *** penick has joined #openstack-dns | 23:13 | |
| *** fawadkhaliq has quit IRC | 23:18 | |
| *** fawadkhaliq has joined #openstack-dns | 23:18 | |
| *** fawadkhaliq has quit IRC | 23:21 | |
| *** fawadkhaliq has joined #openstack-dns | 23:21 | |
| *** eandersson_ has quit IRC | 23:28 | |
| *** kei_yama has joined #openstack-dns | 23:33 | |
| *** penick has quit IRC | 23:37 | |
| *** km has joined #openstack-dns | 23:41 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
