GitLab Dynamically run jobs in parallel
I have a project which generate go main files in folders which follow certain convention. The problem is I need to build those projects parallelly in Gitlab CI pipeline when there is a code push and I cant hardcode them in .gitlab-ci.yml as they are generated on fly. I need to build those go projects in parallel and need to pass build stage if all the individual project builds are succeeded. Can someone please let me know if it is possible in Gitlab.
gitlab gitlab-ci gitlab-ci-runner
edited Nov 14 '18 at 10:58
Mustafa Ehsan
2,3651428
asked Nov 14 '18 at 4:32
Venu ChittaVenu Chitta
85311
add a comment |
I have a project which generate go main files in folders which follow certain convention. The problem is I need to build those projects parallelly in Gitlab CI pipeline when there is a code push and I cant hardcode them in .gitlab-ci.yml as they are generated on fly. I need to build those go projects in parallel and need to pass build stage if all the individual project builds are succeeded. Can someone please let me know if it is possible in Gitlab.
gitlab gitlab-ci gitlab-ci-runner
edited Nov 14 '18 at 10:58
Mustafa Ehsan
2,3651428
asked Nov 14 '18 at 4:32
Venu ChittaVenu Chitta
85311
1
Gitlab CI allows you to run any script you want. You'll have to expand on what you're struggling with that can't be solved withfindandparallelin bash. Do you want to run each project as a separate job? Cause that's not supported as of now: gitlab.com/gitlab-org/gitlab-ce/issues/50572 . Of course this can be done but the solutions won't be pretty.
– Jakub Kania
Nov 14 '18 at 10:04
add a comment |
I have a project which generate go main files in folders which follow certain convention. The problem is I need to build those projects parallelly in Gitlab CI pipeline when there is a code push and I cant hardcode them in .gitlab-ci.yml as they are generated on fly. I need to build those go projects in parallel and need to pass build stage if all the individual project builds are succeeded. Can someone please let me know if it is possible in Gitlab.
gitlab gitlab-ci gitlab-ci-runner
edited Nov 14 '18 at 10:58
Mustafa Ehsan
2,3651428
asked Nov 14 '18 at 4:32
Venu ChittaVenu Chitta
85311
I have a project which generate go main files in folders which follow certain convention. The problem is I need to build those projects parallelly in Gitlab CI pipeline when there is a code push and I cant hardcode them in .gitlab-ci.yml as they are generated on fly. I need to build those go projects in parallel and need to pass build stage if all the individual project builds are succeeded. Can someone please let me know if it is possible in Gitlab.
gitlab gitlab-ci gitlab-ci-runner
gitlab gitlab-ci gitlab-ci-runner
edited Nov 14 '18 at 10:58
Mustafa Ehsan
2,3651428
asked Nov 14 '18 at 4:32
Venu ChittaVenu Chitta
85311
edited Nov 14 '18 at 10:58
Mustafa Ehsan
2,3651428
asked Nov 14 '18 at 4:32
Venu ChittaVenu Chitta
85311
edited Nov 14 '18 at 10:58
Mustafa Ehsan
2,3651428
edited Nov 14 '18 at 10:58
Mustafa Ehsan
2,3651428
edited Nov 14 '18 at 10:58
Mustafa Ehsan
2,3651428
2,3651428
asked Nov 14 '18 at 4:32
Venu ChittaVenu Chitta
85311
asked Nov 14 '18 at 4:32
Venu ChittaVenu Chitta
85311
asked Nov 14 '18 at 4:32
Venu ChittaVenu Chitta
85311
85311
1
Gitlab CI allows you to run any script you want. You'll have to expand on what you're struggling with that can't be solved withfindandparallelin bash. Do you want to run each project as a separate job? Cause that's not supported as of now: gitlab.com/gitlab-org/gitlab-ce/issues/50572 . Of course this can be done but the solutions won't be pretty.
– Jakub Kania
Nov 14 '18 at 10:04
add a comment |
1
Gitlab CI allows you to run any script you want. You'll have to expand on what you're struggling with that can't be solved withfindandparallelin bash. Do you want to run each project as a separate job? Cause that's not supported as of now: gitlab.com/gitlab-org/gitlab-ce/issues/50572 . Of course this can be done but the solutions won't be pretty.
– Jakub Kania
Nov 14 '18 at 10:04
1
1
Gitlab CI allows you to run any script you want. You'll have to expand on what you're struggling with that can't be solved with
find and parallel in bash. Do you want to run each project as a separate job? Cause that's not supported as of now: gitlab.com/gitlab-org/gitlab-ce/issues/50572 . Of course this can be done but the solutions won't be pretty.– Jakub Kania
Nov 14 '18 at 10:04
Gitlab CI allows you to run any script you want. You'll have to expand on what you're struggling with that can't be solved with
find and parallel in bash. Do you want to run each project as a separate job? Cause that's not supported as of now: gitlab.com/gitlab-org/gitlab-ce/issues/50572 . Of course this can be done but the solutions won't be pretty.– Jakub Kania
Nov 14 '18 at 10:04
add a comment |
1 Answer
1
active
oldest
votes
Unfortunately gitlab CI doesn't support this it seems as per https://gitlab.com/gitlab-org/gitlab-ce/issues/23455. As mentioned in this issue and @Jakub, I am looking for something like this(Some glob or template sort of solution) so that I could parallelize jobs on fly:
build_%:
image: dnd
stage: build
glob: microservices/*/Dockerfile
context: microservices/1
script:
- docker build 1
parallel: true
So I have decided to use go templates for now until Gitlab CI comes with one solution.
answered Nov 14 '18 at 10:35
Venu ChittaVenu Chitta
85311
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53293232%2fgitlab-dynamically-run-jobs-in-parallel%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Unfortunately gitlab CI doesn't support this it seems as per https://gitlab.com/gitlab-org/gitlab-ce/issues/23455. As mentioned in this issue and @Jakub, I am looking for something like this(Some glob or template sort of solution) so that I could parallelize jobs on fly:
build_%:
image: dnd
stage: build
glob: microservices/*/Dockerfile
context: microservices/1
script:
- docker build 1
parallel: true
So I have decided to use go templates for now until Gitlab CI comes with one solution.
answered Nov 14 '18 at 10:35
Venu ChittaVenu Chitta
85311
add a comment |
Unfortunately gitlab CI doesn't support this it seems as per https://gitlab.com/gitlab-org/gitlab-ce/issues/23455. As mentioned in this issue and @Jakub, I am looking for something like this(Some glob or template sort of solution) so that I could parallelize jobs on fly:
build_%:
image: dnd
stage: build
glob: microservices/*/Dockerfile
context: microservices/1
script:
- docker build 1
parallel: true
So I have decided to use go templates for now until Gitlab CI comes with one solution.
answered Nov 14 '18 at 10:35
Venu ChittaVenu Chitta
85311
add a comment |
Unfortunately gitlab CI doesn't support this it seems as per https://gitlab.com/gitlab-org/gitlab-ce/issues/23455. As mentioned in this issue and @Jakub, I am looking for something like this(Some glob or template sort of solution) so that I could parallelize jobs on fly:
build_%:
image: dnd
stage: build
glob: microservices/*/Dockerfile
context: microservices/1
script:
- docker build 1
parallel: true
So I have decided to use go templates for now until Gitlab CI comes with one solution.
answered Nov 14 '18 at 10:35
Venu ChittaVenu Chitta
85311
Unfortunately gitlab CI doesn't support this it seems as per https://gitlab.com/gitlab-org/gitlab-ce/issues/23455. As mentioned in this issue and @Jakub, I am looking for something like this(Some glob or template sort of solution) so that I could parallelize jobs on fly:
build_%:
image: dnd
stage: build
glob: microservices/*/Dockerfile
context: microservices/1
script:
- docker build 1
parallel: true
So I have decided to use go templates for now until Gitlab CI comes with one solution.
answered Nov 14 '18 at 10:35
Venu ChittaVenu Chitta
85311
edited Nov 14 '18 at 17:21
answered Nov 14 '18 at 10:35
Venu ChittaVenu Chitta
85311
answered Nov 14 '18 at 10:35
Venu ChittaVenu Chitta
85311
answered Nov 14 '18 at 10:35
Venu ChittaVenu Chitta
85311
85311
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53293232%2fgitlab-dynamically-run-jobs-in-parallel%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Gitlab CI allows you to run any script you want. You'll have to expand on what you're struggling with that can't be solved with
findandparallelin bash. Do you want to run each project as a separate job? Cause that's not supported as of now: gitlab.com/gitlab-org/gitlab-ce/issues/50572 . Of course this can be done but the solutions won't be pretty.– Jakub Kania
Nov 14 '18 at 10:04