How to send a header+buffer efficiently with TCP/IP?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
Suppose, that I have two buffers:
- A small header (10-20 bytes)
- Payload (varies between ~10KiB - ~10MiB, but usually less than 100KiB)
I need to send this to the server. The server immediately responds when it received the buffer.
Now, if I use two send calls for this (one for the header, and one for the payload), I get very bad latency on windows: after the sends, recv (server response) arrives very slowly (40-100ms). If I turn on TCP_NODELAY for the socket, response comes faster, but I'm still not satisfied with the result (5-10ms. But server responds immediately. Ping time to the server is 0.1ms - it's on a local network).
I read here, that we should avoid write-write-read. But, this is the exact thing what I'm doing now. How should I solve this?
The straightforward solution is to copy the header and payload into a new buffer, and send that instead. But I'd like to avoid this, as I need to allocate a new larger buffer for this (not to mention the copy).
Another solution would be to allocate a smallish buffer (like 4KB), and send the data in chunks. But, this way, I'm not sure how this 4KB chunk will be cut into TCP/IP segments. It may happen, that the last segment will be small, so this causes a little inefficiency.
Linux has MSG_MORE, which may solve this issue on Linux. But I see no Windows equivalent.
What is the most efficient (fast) solution to this problem?
windows sockets tcp
asked Nov 15 '18 at 13:37
gezageza
13.8k33179
|
show 9 more comments
Suppose, that I have two buffers:
- A small header (10-20 bytes)
- Payload (varies between ~10KiB - ~10MiB, but usually less than 100KiB)
I need to send this to the server. The server immediately responds when it received the buffer.
Now, if I use two send calls for this (one for the header, and one for the payload), I get very bad latency on windows: after the sends, recv (server response) arrives very slowly (40-100ms). If I turn on TCP_NODELAY for the socket, response comes faster, but I'm still not satisfied with the result (5-10ms. But server responds immediately. Ping time to the server is 0.1ms - it's on a local network).
I read here, that we should avoid write-write-read. But, this is the exact thing what I'm doing now. How should I solve this?
The straightforward solution is to copy the header and payload into a new buffer, and send that instead. But I'd like to avoid this, as I need to allocate a new larger buffer for this (not to mention the copy).
Another solution would be to allocate a smallish buffer (like 4KB), and send the data in chunks. But, this way, I'm not sure how this 4KB chunk will be cut into TCP/IP segments. It may happen, that the last segment will be small, so this causes a little inefficiency.
Linux has MSG_MORE, which may solve this issue on Linux. But I see no Windows equivalent.
What is the most efficient (fast) solution to this problem?
windows sockets tcp
asked Nov 15 '18 at 13:37
gezageza
13.8k33179
1
"copy the header and payload into a new buffer" Have you tried that? It might be that the latency is introduced by TCP slow start (or whatever it is called). A bigger send might not help in accelerating TCP startup at all. Maybe the TCP window size is the limiting factor.
– usr
Nov 15 '18 at 14:02
You misunderstood Nagle's algorithm - it's what means you don't have to care how many sends you use (ignoring whatever overheadsenditself has). You also seem to be conflating many different issues - you need to take a closer look on what's really bothering you. TCP_NODELAY is something you can use (for the last "packet" of data to be sent) if you want a "quick" response, but if 5-10ms isn't good enough for you, you probably can't use TCP in the first place, or you're micro-optimizing :) None of your solutions other than TCP_NODELAY should help with that - why would they?
– Luaan
Nov 15 '18 at 14:38
1
@Luaan: I'm not too experienced with TCP programming. I just expect better response times. If ping time 0.1ms, then I expect TCP response 0.2-0.3ms or something like that. 5-10ms is way too high. Is it common to have such a large response time with TCP, if ping is 0.1ms?
– geza
Nov 15 '18 at 14:48
5-10ms isn't a long response time. Keep in mind that even just the default timer resolution of most OSes (if they have a timer resolution) is in the region of 10ms or so. Ping does an average of multiple ICMP requests, which is about the fastest thing you can do on LAN. TCP requires establishing a connection (SYN, ACK, SYN-ACK), and a confirmation of every single TCP segment sent. That's a lot of messages being sent around, in both directions. And you're sending a lot of data - while ping only sends a couple of bytes. They just don't compare. Don't expect to get closer to zero than 5-10ms :)
– Luaan
Nov 15 '18 at 15:03
@usr: not tried yet. It would be hard to implement for my current code unfortunately. But if I get positive feedback about it, I'll do it, of course.
– geza
Nov 15 '18 at 15:04
|
show 9 more comments
Suppose, that I have two buffers:
- A small header (10-20 bytes)
- Payload (varies between ~10KiB - ~10MiB, but usually less than 100KiB)
I need to send this to the server. The server immediately responds when it received the buffer.
Now, if I use two send calls for this (one for the header, and one for the payload), I get very bad latency on windows: after the sends, recv (server response) arrives very slowly (40-100ms). If I turn on TCP_NODELAY for the socket, response comes faster, but I'm still not satisfied with the result (5-10ms. But server responds immediately. Ping time to the server is 0.1ms - it's on a local network).
I read here, that we should avoid write-write-read. But, this is the exact thing what I'm doing now. How should I solve this?
The straightforward solution is to copy the header and payload into a new buffer, and send that instead. But I'd like to avoid this, as I need to allocate a new larger buffer for this (not to mention the copy).
Another solution would be to allocate a smallish buffer (like 4KB), and send the data in chunks. But, this way, I'm not sure how this 4KB chunk will be cut into TCP/IP segments. It may happen, that the last segment will be small, so this causes a little inefficiency.
Linux has MSG_MORE, which may solve this issue on Linux. But I see no Windows equivalent.
What is the most efficient (fast) solution to this problem?
windows sockets tcp
asked Nov 15 '18 at 13:37
gezageza
13.8k33179
Suppose, that I have two buffers:
- A small header (10-20 bytes)
- Payload (varies between ~10KiB - ~10MiB, but usually less than 100KiB)
I need to send this to the server. The server immediately responds when it received the buffer.
Now, if I use two send calls for this (one for the header, and one for the payload), I get very bad latency on windows: after the sends, recv (server response) arrives very slowly (40-100ms). If I turn on TCP_NODELAY for the socket, response comes faster, but I'm still not satisfied with the result (5-10ms. But server responds immediately. Ping time to the server is 0.1ms - it's on a local network).
I read here, that we should avoid write-write-read. But, this is the exact thing what I'm doing now. How should I solve this?
The straightforward solution is to copy the header and payload into a new buffer, and send that instead. But I'd like to avoid this, as I need to allocate a new larger buffer for this (not to mention the copy).
Another solution would be to allocate a smallish buffer (like 4KB), and send the data in chunks. But, this way, I'm not sure how this 4KB chunk will be cut into TCP/IP segments. It may happen, that the last segment will be small, so this causes a little inefficiency.
Linux has MSG_MORE, which may solve this issue on Linux. But I see no Windows equivalent.
What is the most efficient (fast) solution to this problem?
windows sockets tcp
windows sockets tcp
asked Nov 15 '18 at 13:37
gezageza
13.8k33179
asked Nov 15 '18 at 13:37
gezageza
13.8k33179
asked Nov 15 '18 at 13:37
gezageza
13.8k33179
asked Nov 15 '18 at 13:37
gezageza
13.8k33179
asked Nov 15 '18 at 13:37
gezageza
13.8k33179
13.8k33179
1
"copy the header and payload into a new buffer" Have you tried that? It might be that the latency is introduced by TCP slow start (or whatever it is called). A bigger send might not help in accelerating TCP startup at all. Maybe the TCP window size is the limiting factor.
– usr
Nov 15 '18 at 14:02
You misunderstood Nagle's algorithm - it's what means you don't have to care how many sends you use (ignoring whatever overheadsenditself has). You also seem to be conflating many different issues - you need to take a closer look on what's really bothering you. TCP_NODELAY is something you can use (for the last "packet" of data to be sent) if you want a "quick" response, but if 5-10ms isn't good enough for you, you probably can't use TCP in the first place, or you're micro-optimizing :) None of your solutions other than TCP_NODELAY should help with that - why would they?
– Luaan
Nov 15 '18 at 14:38
1
@Luaan: I'm not too experienced with TCP programming. I just expect better response times. If ping time 0.1ms, then I expect TCP response 0.2-0.3ms or something like that. 5-10ms is way too high. Is it common to have such a large response time with TCP, if ping is 0.1ms?
– geza
Nov 15 '18 at 14:48
5-10ms isn't a long response time. Keep in mind that even just the default timer resolution of most OSes (if they have a timer resolution) is in the region of 10ms or so. Ping does an average of multiple ICMP requests, which is about the fastest thing you can do on LAN. TCP requires establishing a connection (SYN, ACK, SYN-ACK), and a confirmation of every single TCP segment sent. That's a lot of messages being sent around, in both directions. And you're sending a lot of data - while ping only sends a couple of bytes. They just don't compare. Don't expect to get closer to zero than 5-10ms :)
– Luaan
Nov 15 '18 at 15:03
@usr: not tried yet. It would be hard to implement for my current code unfortunately. But if I get positive feedback about it, I'll do it, of course.
– geza
Nov 15 '18 at 15:04
|
show 9 more comments
1
"copy the header and payload into a new buffer" Have you tried that? It might be that the latency is introduced by TCP slow start (or whatever it is called). A bigger send might not help in accelerating TCP startup at all. Maybe the TCP window size is the limiting factor.
– usr
Nov 15 '18 at 14:02
You misunderstood Nagle's algorithm - it's what means you don't have to care how many sends you use (ignoring whatever overheadsenditself has). You also seem to be conflating many different issues - you need to take a closer look on what's really bothering you. TCP_NODELAY is something you can use (for the last "packet" of data to be sent) if you want a "quick" response, but if 5-10ms isn't good enough for you, you probably can't use TCP in the first place, or you're micro-optimizing :) None of your solutions other than TCP_NODELAY should help with that - why would they?
– Luaan
Nov 15 '18 at 14:38
1
@Luaan: I'm not too experienced with TCP programming. I just expect better response times. If ping time 0.1ms, then I expect TCP response 0.2-0.3ms or something like that. 5-10ms is way too high. Is it common to have such a large response time with TCP, if ping is 0.1ms?
– geza
Nov 15 '18 at 14:48
5-10ms isn't a long response time. Keep in mind that even just the default timer resolution of most OSes (if they have a timer resolution) is in the region of 10ms or so. Ping does an average of multiple ICMP requests, which is about the fastest thing you can do on LAN. TCP requires establishing a connection (SYN, ACK, SYN-ACK), and a confirmation of every single TCP segment sent. That's a lot of messages being sent around, in both directions. And you're sending a lot of data - while ping only sends a couple of bytes. They just don't compare. Don't expect to get closer to zero than 5-10ms :)
– Luaan
Nov 15 '18 at 15:03
@usr: not tried yet. It would be hard to implement for my current code unfortunately. But if I get positive feedback about it, I'll do it, of course.
– geza
Nov 15 '18 at 15:04
1
1
"copy the header and payload into a new buffer" Have you tried that? It might be that the latency is introduced by TCP slow start (or whatever it is called). A bigger send might not help in accelerating TCP startup at all. Maybe the TCP window size is the limiting factor.
– usr
Nov 15 '18 at 14:02
"copy the header and payload into a new buffer" Have you tried that? It might be that the latency is introduced by TCP slow start (or whatever it is called). A bigger send might not help in accelerating TCP startup at all. Maybe the TCP window size is the limiting factor.
– usr
Nov 15 '18 at 14:02
You misunderstood Nagle's algorithm - it's what means you don't have to care how many sends you use (ignoring whatever overhead
send itself has). You also seem to be conflating many different issues - you need to take a closer look on what's really bothering you. TCP_NODELAY is something you can use (for the last "packet" of data to be sent) if you want a "quick" response, but if 5-10ms isn't good enough for you, you probably can't use TCP in the first place, or you're micro-optimizing :) None of your solutions other than TCP_NODELAY should help with that - why would they?– Luaan
Nov 15 '18 at 14:38
You misunderstood Nagle's algorithm - it's what means you don't have to care how many sends you use (ignoring whatever overhead
send itself has). You also seem to be conflating many different issues - you need to take a closer look on what's really bothering you. TCP_NODELAY is something you can use (for the last "packet" of data to be sent) if you want a "quick" response, but if 5-10ms isn't good enough for you, you probably can't use TCP in the first place, or you're micro-optimizing :) None of your solutions other than TCP_NODELAY should help with that - why would they?– Luaan
Nov 15 '18 at 14:38
1
1
@Luaan: I'm not too experienced with TCP programming. I just expect better response times. If ping time 0.1ms, then I expect TCP response 0.2-0.3ms or something like that. 5-10ms is way too high. Is it common to have such a large response time with TCP, if ping is 0.1ms?
– geza
Nov 15 '18 at 14:48
@Luaan: I'm not too experienced with TCP programming. I just expect better response times. If ping time 0.1ms, then I expect TCP response 0.2-0.3ms or something like that. 5-10ms is way too high. Is it common to have such a large response time with TCP, if ping is 0.1ms?
– geza
Nov 15 '18 at 14:48
5-10ms isn't a long response time. Keep in mind that even just the default timer resolution of most OSes (if they have a timer resolution) is in the region of 10ms or so. Ping does an average of multiple ICMP requests, which is about the fastest thing you can do on LAN. TCP requires establishing a connection (SYN, ACK, SYN-ACK), and a confirmation of every single TCP segment sent. That's a lot of messages being sent around, in both directions. And you're sending a lot of data - while ping only sends a couple of bytes. They just don't compare. Don't expect to get closer to zero than 5-10ms :)
– Luaan
Nov 15 '18 at 15:03
5-10ms isn't a long response time. Keep in mind that even just the default timer resolution of most OSes (if they have a timer resolution) is in the region of 10ms or so. Ping does an average of multiple ICMP requests, which is about the fastest thing you can do on LAN. TCP requires establishing a connection (SYN, ACK, SYN-ACK), and a confirmation of every single TCP segment sent. That's a lot of messages being sent around, in both directions. And you're sending a lot of data - while ping only sends a couple of bytes. They just don't compare. Don't expect to get closer to zero than 5-10ms :)
– Luaan
Nov 15 '18 at 15:03
@usr: not tried yet. It would be hard to implement for my current code unfortunately. But if I get positive feedback about it, I'll do it, of course.
– geza
Nov 15 '18 at 15:04
@usr: not tried yet. It would be hard to implement for my current code unfortunately. But if I get positive feedback about it, I'll do it, of course.
– geza
Nov 15 '18 at 15:04
|
show 9 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53320717%2fhow-to-send-a-headerbuffer-efficiently-with-tcp-ip%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53320717%2fhow-to-send-a-headerbuffer-efficiently-with-tcp-ip%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
"copy the header and payload into a new buffer" Have you tried that? It might be that the latency is introduced by TCP slow start (or whatever it is called). A bigger send might not help in accelerating TCP startup at all. Maybe the TCP window size is the limiting factor.
– usr
Nov 15 '18 at 14:02
You misunderstood Nagle's algorithm - it's what means you don't have to care how many sends you use (ignoring whatever overhead
senditself has). You also seem to be conflating many different issues - you need to take a closer look on what's really bothering you. TCP_NODELAY is something you can use (for the last "packet" of data to be sent) if you want a "quick" response, but if 5-10ms isn't good enough for you, you probably can't use TCP in the first place, or you're micro-optimizing :) None of your solutions other than TCP_NODELAY should help with that - why would they?– Luaan
Nov 15 '18 at 14:38
1
@Luaan: I'm not too experienced with TCP programming. I just expect better response times. If ping time 0.1ms, then I expect TCP response 0.2-0.3ms or something like that. 5-10ms is way too high. Is it common to have such a large response time with TCP, if ping is 0.1ms?
– geza
Nov 15 '18 at 14:48
5-10ms isn't a long response time. Keep in mind that even just the default timer resolution of most OSes (if they have a timer resolution) is in the region of 10ms or so. Ping does an average of multiple ICMP requests, which is about the fastest thing you can do on LAN. TCP requires establishing a connection (SYN, ACK, SYN-ACK), and a confirmation of every single TCP segment sent. That's a lot of messages being sent around, in both directions. And you're sending a lot of data - while ping only sends a couple of bytes. They just don't compare. Don't expect to get closer to zero than 5-10ms :)
– Luaan
Nov 15 '18 at 15:03
@usr: not tried yet. It would be hard to implement for my current code unfortunately. But if I get positive feedback about it, I'll do it, of course.
– geza
Nov 15 '18 at 15:04