Which numpy operations copy and which mutate?
up vote
1
down vote
favorite
Is there a general rule of thumb for knowing which operations on a numpy.ndarray
produce a copy of the values and which mutate them in-place?
I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.
numpy
add a comment |
up vote
1
down vote
favorite
Is there a general rule of thumb for knowing which operations on a numpy.ndarray
produce a copy of the values and which mutate them in-place?
I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.
numpy
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
Is there a general rule of thumb for knowing which operations on a numpy.ndarray
produce a copy of the values and which mutate them in-place?
I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.
numpy
Is there a general rule of thumb for knowing which operations on a numpy.ndarray
produce a copy of the values and which mutate them in-place?
I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.
numpy
numpy
asked Nov 11 at 4:08
scanny
9,49712042
9,49712042
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
add a comment |
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
add a comment |
1 Answer
1
active
oldest
votes
up vote
2
down vote
accepted
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resize
ndarray.sort
- All of the in-place arithmetic operators (eg
+=
,*=
,/=
, etc) numpy.fill_diagonal
numpy.random.shuffle
ndarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswap
numpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1
will set every value in an array to 1
), and you can reshape an array by assigning a new shape directly to .shape
, eg arr.shape = (2,3)
(won't always work, see notes here).
There's also some functions that support an out
keyword arg. These functions will behave as mutators if you pass the same array as both input and out
.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray
methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten
and ndarray.ravel
do the same thing (returned a flattened array). However, the docs for ndarray.flatten
explicitly say that it will return a copy, whereas the docs for ndarray.ravel
say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id
of the .base
of your result to the id
of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53245763%2fwhich-numpy-operations-copy-and-which-mutate%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
accepted
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resize
ndarray.sort
- All of the in-place arithmetic operators (eg
+=
,*=
,/=
, etc) numpy.fill_diagonal
numpy.random.shuffle
ndarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswap
numpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1
will set every value in an array to 1
), and you can reshape an array by assigning a new shape directly to .shape
, eg arr.shape = (2,3)
(won't always work, see notes here).
There's also some functions that support an out
keyword arg. These functions will behave as mutators if you pass the same array as both input and out
.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray
methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten
and ndarray.ravel
do the same thing (returned a flattened array). However, the docs for ndarray.flatten
explicitly say that it will return a copy, whereas the docs for ndarray.ravel
say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id
of the .base
of your result to the id
of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
up vote
2
down vote
accepted
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resize
ndarray.sort
- All of the in-place arithmetic operators (eg
+=
,*=
,/=
, etc) numpy.fill_diagonal
numpy.random.shuffle
ndarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswap
numpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1
will set every value in an array to 1
), and you can reshape an array by assigning a new shape directly to .shape
, eg arr.shape = (2,3)
(won't always work, see notes here).
There's also some functions that support an out
keyword arg. These functions will behave as mutators if you pass the same array as both input and out
.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray
methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten
and ndarray.ravel
do the same thing (returned a flattened array). However, the docs for ndarray.flatten
explicitly say that it will return a copy, whereas the docs for ndarray.ravel
say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id
of the .base
of your result to the id
of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
up vote
2
down vote
accepted
up vote
2
down vote
accepted
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resize
ndarray.sort
- All of the in-place arithmetic operators (eg
+=
,*=
,/=
, etc) numpy.fill_diagonal
numpy.random.shuffle
ndarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswap
numpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1
will set every value in an array to 1
), and you can reshape an array by assigning a new shape directly to .shape
, eg arr.shape = (2,3)
(won't always work, see notes here).
There's also some functions that support an out
keyword arg. These functions will behave as mutators if you pass the same array as both input and out
.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray
methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten
and ndarray.ravel
do the same thing (returned a flattened array). However, the docs for ndarray.flatten
explicitly say that it will return a copy, whereas the docs for ndarray.ravel
say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id
of the .base
of your result to the id
of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resize
ndarray.sort
- All of the in-place arithmetic operators (eg
+=
,*=
,/=
, etc) numpy.fill_diagonal
numpy.random.shuffle
ndarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswap
numpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1
will set every value in an array to 1
), and you can reshape an array by assigning a new shape directly to .shape
, eg arr.shape = (2,3)
(won't always work, see notes here).
There's also some functions that support an out
keyword arg. These functions will behave as mutators if you pass the same array as both input and out
.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray
methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten
and ndarray.ravel
do the same thing (returned a flattened array). However, the docs for ndarray.flatten
explicitly say that it will return a copy, whereas the docs for ndarray.ravel
say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id
of the .base
of your result to the id
of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
edited Nov 11 at 5:14
answered Nov 11 at 4:30
tel
5,68011430
5,68011430
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53245763%2fwhich-numpy-operations-copy-and-which-mutate%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00