BigQueryException: API limit exceeded
I'm using Bigquery Java API to run ~1000 copy jobs simultaneously (With scala.concurrent.Future) with WriteDisposition WRITE_APPEND, but I'm getting
com.google.cloud.bigquery.BigQueryException: API limit exceeded: Unable to return a row that exceeds the API limits. To retrieve the row, export the table
I thought this is caused by too much concurrency, then I tried to use Monix's Task to limit the parallelism to at most 20:
def execute(queries: List[Query]): Future[Seq[Boolean]] =
val tasks: Iterator[Task[List[Boolean]]] = queries.map(q => BqApi.copyTable(q, destinationTable))
.sliding(20, 20)
.map(Task.gather(_))
val results: Task[List[Boolean]] = Task.sequence(tasks)
.map(_.flatten.toList)
results.runAsync
where BqApi.copyTable executes the query and copy the result to the destination table then returns a Task[Boolean].
The same exception still happens.
But if I change the WriteDisposition to WRITE_TRUNCATE, the exception goes away.
Can anyone help me to understand what happens under the hood? And why Bigquery API behaves like this?
scala google-bigquery
asked Nov 13 '18 at 0:38
GregoryGregory
4313
add a comment |
I'm using Bigquery Java API to run ~1000 copy jobs simultaneously (With scala.concurrent.Future) with WriteDisposition WRITE_APPEND, but I'm getting
com.google.cloud.bigquery.BigQueryException: API limit exceeded: Unable to return a row that exceeds the API limits. To retrieve the row, export the table
I thought this is caused by too much concurrency, then I tried to use Monix's Task to limit the parallelism to at most 20:
def execute(queries: List[Query]): Future[Seq[Boolean]] =
val tasks: Iterator[Task[List[Boolean]]] = queries.map(q => BqApi.copyTable(q, destinationTable))
.sliding(20, 20)
.map(Task.gather(_))
val results: Task[List[Boolean]] = Task.sequence(tasks)
.map(_.flatten.toList)
results.runAsync
where BqApi.copyTable executes the query and copy the result to the destination table then returns a Task[Boolean].
The same exception still happens.
But if I change the WriteDisposition to WRITE_TRUNCATE, the exception goes away.
Can anyone help me to understand what happens under the hood? And why Bigquery API behaves like this?
scala google-bigquery
asked Nov 13 '18 at 0:38
GregoryGregory
4313
add a comment |
I'm using Bigquery Java API to run ~1000 copy jobs simultaneously (With scala.concurrent.Future) with WriteDisposition WRITE_APPEND, but I'm getting
com.google.cloud.bigquery.BigQueryException: API limit exceeded: Unable to return a row that exceeds the API limits. To retrieve the row, export the table
I thought this is caused by too much concurrency, then I tried to use Monix's Task to limit the parallelism to at most 20:
def execute(queries: List[Query]): Future[Seq[Boolean]] =
val tasks: Iterator[Task[List[Boolean]]] = queries.map(q => BqApi.copyTable(q, destinationTable))
.sliding(20, 20)
.map(Task.gather(_))
val results: Task[List[Boolean]] = Task.sequence(tasks)
.map(_.flatten.toList)
results.runAsync
where BqApi.copyTable executes the query and copy the result to the destination table then returns a Task[Boolean].
The same exception still happens.
But if I change the WriteDisposition to WRITE_TRUNCATE, the exception goes away.
Can anyone help me to understand what happens under the hood? And why Bigquery API behaves like this?
scala google-bigquery
asked Nov 13 '18 at 0:38
GregoryGregory
4313
I'm using Bigquery Java API to run ~1000 copy jobs simultaneously (With scala.concurrent.Future) with WriteDisposition WRITE_APPEND, but I'm getting
com.google.cloud.bigquery.BigQueryException: API limit exceeded: Unable to return a row that exceeds the API limits. To retrieve the row, export the table
I thought this is caused by too much concurrency, then I tried to use Monix's Task to limit the parallelism to at most 20:
def execute(queries: List[Query]): Future[Seq[Boolean]] =
val tasks: Iterator[Task[List[Boolean]]] = queries.map(q => BqApi.copyTable(q, destinationTable))
.sliding(20, 20)
.map(Task.gather(_))
val results: Task[List[Boolean]] = Task.sequence(tasks)
.map(_.flatten.toList)
results.runAsync
where BqApi.copyTable executes the query and copy the result to the destination table then returns a Task[Boolean].
The same exception still happens.
But if I change the WriteDisposition to WRITE_TRUNCATE, the exception goes away.
Can anyone help me to understand what happens under the hood? And why Bigquery API behaves like this?
scala google-bigquery
scala google-bigquery
asked Nov 13 '18 at 0:38
GregoryGregory
4313
asked Nov 13 '18 at 0:38
GregoryGregory
4313
asked Nov 13 '18 at 0:38
GregoryGregory
4313
asked Nov 13 '18 at 0:38
GregoryGregory
4313
asked Nov 13 '18 at 0:38
GregoryGregory
4313
4313
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
This message is encountered when a query exceeds a maximum response size. Since copy jobs use jobs.insert, maybe you're hitting the maximum row size which are in the query jobs limits. I suggest filling a BigQuery bug on its issue tracker to describe your behavior properly regarding the Java API.
answered Nov 13 '18 at 13:43
F10F10
1,2962414
Thanks for your reply, but if I set WriteDisposition to WRITE_TRUNCATE, the exception never shows up, so the 'exceed max row size' assumption may not be valid since the query part in both cases are the same.
– Gregory
Nov 13 '18 at 18:10
Since WRITE_TRUNCATE overwrites the table data and you're not getting the API limit message by using this, then I'll suggest to fill an issue so the BigQuery team can review this behavior.
– F10
Nov 20 '18 at 14:50
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53272122%2fbigqueryexception-api-limit-exceeded%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This message is encountered when a query exceeds a maximum response size. Since copy jobs use jobs.insert, maybe you're hitting the maximum row size which are in the query jobs limits. I suggest filling a BigQuery bug on its issue tracker to describe your behavior properly regarding the Java API.
answered Nov 13 '18 at 13:43
F10F10
1,2962414
Thanks for your reply, but if I set WriteDisposition to WRITE_TRUNCATE, the exception never shows up, so the 'exceed max row size' assumption may not be valid since the query part in both cases are the same.
– Gregory
Nov 13 '18 at 18:10
Since WRITE_TRUNCATE overwrites the table data and you're not getting the API limit message by using this, then I'll suggest to fill an issue so the BigQuery team can review this behavior.
– F10
Nov 20 '18 at 14:50
add a comment |
This message is encountered when a query exceeds a maximum response size. Since copy jobs use jobs.insert, maybe you're hitting the maximum row size which are in the query jobs limits. I suggest filling a BigQuery bug on its issue tracker to describe your behavior properly regarding the Java API.
answered Nov 13 '18 at 13:43
F10F10
1,2962414
Thanks for your reply, but if I set WriteDisposition to WRITE_TRUNCATE, the exception never shows up, so the 'exceed max row size' assumption may not be valid since the query part in both cases are the same.
– Gregory
Nov 13 '18 at 18:10
Since WRITE_TRUNCATE overwrites the table data and you're not getting the API limit message by using this, then I'll suggest to fill an issue so the BigQuery team can review this behavior.
– F10
Nov 20 '18 at 14:50
add a comment |
This message is encountered when a query exceeds a maximum response size. Since copy jobs use jobs.insert, maybe you're hitting the maximum row size which are in the query jobs limits. I suggest filling a BigQuery bug on its issue tracker to describe your behavior properly regarding the Java API.
answered Nov 13 '18 at 13:43
F10F10
1,2962414
This message is encountered when a query exceeds a maximum response size. Since copy jobs use jobs.insert, maybe you're hitting the maximum row size which are in the query jobs limits. I suggest filling a BigQuery bug on its issue tracker to describe your behavior properly regarding the Java API.
answered Nov 13 '18 at 13:43
F10F10
1,2962414
answered Nov 13 '18 at 13:43
F10F10
1,2962414
answered Nov 13 '18 at 13:43
F10F10
1,2962414
answered Nov 13 '18 at 13:43
F10F10
1,2962414
1,2962414
Thanks for your reply, but if I set WriteDisposition to WRITE_TRUNCATE, the exception never shows up, so the 'exceed max row size' assumption may not be valid since the query part in both cases are the same.
– Gregory
Nov 13 '18 at 18:10
Since WRITE_TRUNCATE overwrites the table data and you're not getting the API limit message by using this, then I'll suggest to fill an issue so the BigQuery team can review this behavior.
– F10
Nov 20 '18 at 14:50
add a comment |
Thanks for your reply, but if I set WriteDisposition to WRITE_TRUNCATE, the exception never shows up, so the 'exceed max row size' assumption may not be valid since the query part in both cases are the same.
– Gregory
Nov 13 '18 at 18:10
Since WRITE_TRUNCATE overwrites the table data and you're not getting the API limit message by using this, then I'll suggest to fill an issue so the BigQuery team can review this behavior.
– F10
Nov 20 '18 at 14:50
Thanks for your reply, but if I set WriteDisposition to WRITE_TRUNCATE, the exception never shows up, so the 'exceed max row size' assumption may not be valid since the query part in both cases are the same.
– Gregory
Nov 13 '18 at 18:10
Thanks for your reply, but if I set WriteDisposition to WRITE_TRUNCATE, the exception never shows up, so the 'exceed max row size' assumption may not be valid since the query part in both cases are the same.
– Gregory
Nov 13 '18 at 18:10
Since WRITE_TRUNCATE overwrites the table data and you're not getting the API limit message by using this, then I'll suggest to fill an issue so the BigQuery team can review this behavior.
– F10
Nov 20 '18 at 14:50
Since WRITE_TRUNCATE overwrites the table data and you're not getting the API limit message by using this, then I'll suggest to fill an issue so the BigQuery team can review this behavior.
– F10
Nov 20 '18 at 14:50
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53272122%2fbigqueryexception-api-limit-exceeded%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
