Which numpy operations copy and which mutate?









up vote
1
down vote

favorite












Is there a general rule of thumb for knowing which operations on a numpy.ndarray produce a copy of the values and which mutate them in-place?



I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.










share|improve this question





















  • There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
    – hpaulj
    Nov 11 at 4:44










  • No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
    – tel
    Nov 11 at 5:16










  • @tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
    – scanny
    Nov 11 at 7:46










  • Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
    – tel
    Nov 11 at 8:00














up vote
1
down vote

favorite












Is there a general rule of thumb for knowing which operations on a numpy.ndarray produce a copy of the values and which mutate them in-place?



I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.










share|improve this question





















  • There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
    – hpaulj
    Nov 11 at 4:44










  • No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
    – tel
    Nov 11 at 5:16










  • @tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
    – scanny
    Nov 11 at 7:46










  • Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
    – tel
    Nov 11 at 8:00












up vote
1
down vote

favorite









up vote
1
down vote

favorite











Is there a general rule of thumb for knowing which operations on a numpy.ndarray produce a copy of the values and which mutate them in-place?



I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.










share|improve this question













Is there a general rule of thumb for knowing which operations on a numpy.ndarray produce a copy of the values and which mutate them in-place?



I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.







numpy






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 11 at 4:08









scanny

9,49712042




9,49712042











  • There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
    – hpaulj
    Nov 11 at 4:44










  • No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
    – tel
    Nov 11 at 5:16










  • @tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
    – scanny
    Nov 11 at 7:46










  • Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
    – tel
    Nov 11 at 8:00
















  • There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
    – hpaulj
    Nov 11 at 4:44










  • No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
    – tel
    Nov 11 at 5:16










  • @tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
    – scanny
    Nov 11 at 7:46










  • Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
    – tel
    Nov 11 at 8:00















There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44




There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44












No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16




No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16












@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46




@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46












Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00




Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00












1 Answer
1






active

oldest

votes

















up vote
2
down vote



accepted










Functions that mutate in place



Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.



Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:



  • ndarray.resize

  • ndarray.sort

  • All of the in-place arithmetic operators (eg +=, *=, /=, etc)

  • numpy.fill_diagonal

  • numpy.random.shuffle

  • ndarray.partition

and here's a list of the functions/methods that can optionally mutate in place:



  • ndarray.byteswap

  • numpy.nan_to_num

Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).



There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.



Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.



Notes on view vs copy return values



One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.



For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.



In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:



arr = np.array([[1, 2],
[3, 4],
[5, 6]])

arrflat = arr.flatten()
assert arrflat.base is not arr

arrravel = arr.ravel()
assert arrravel.base is arr





share|improve this answer






















  • I'm not sure he's asking about views vs. copies. Both are new array objects.
    – hpaulj
    Nov 11 at 4:40










Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53245763%2fwhich-numpy-operations-copy-and-which-mutate%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
2
down vote



accepted










Functions that mutate in place



Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.



Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:



  • ndarray.resize

  • ndarray.sort

  • All of the in-place arithmetic operators (eg +=, *=, /=, etc)

  • numpy.fill_diagonal

  • numpy.random.shuffle

  • ndarray.partition

and here's a list of the functions/methods that can optionally mutate in place:



  • ndarray.byteswap

  • numpy.nan_to_num

Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).



There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.



Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.



Notes on view vs copy return values



One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.



For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.



In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:



arr = np.array([[1, 2],
[3, 4],
[5, 6]])

arrflat = arr.flatten()
assert arrflat.base is not arr

arrravel = arr.ravel()
assert arrravel.base is arr





share|improve this answer






















  • I'm not sure he's asking about views vs. copies. Both are new array objects.
    – hpaulj
    Nov 11 at 4:40














up vote
2
down vote



accepted










Functions that mutate in place



Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.



Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:



  • ndarray.resize

  • ndarray.sort

  • All of the in-place arithmetic operators (eg +=, *=, /=, etc)

  • numpy.fill_diagonal

  • numpy.random.shuffle

  • ndarray.partition

and here's a list of the functions/methods that can optionally mutate in place:



  • ndarray.byteswap

  • numpy.nan_to_num

Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).



There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.



Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.



Notes on view vs copy return values



One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.



For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.



In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:



arr = np.array([[1, 2],
[3, 4],
[5, 6]])

arrflat = arr.flatten()
assert arrflat.base is not arr

arrravel = arr.ravel()
assert arrravel.base is arr





share|improve this answer






















  • I'm not sure he's asking about views vs. copies. Both are new array objects.
    – hpaulj
    Nov 11 at 4:40












up vote
2
down vote



accepted







up vote
2
down vote



accepted






Functions that mutate in place



Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.



Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:



  • ndarray.resize

  • ndarray.sort

  • All of the in-place arithmetic operators (eg +=, *=, /=, etc)

  • numpy.fill_diagonal

  • numpy.random.shuffle

  • ndarray.partition

and here's a list of the functions/methods that can optionally mutate in place:



  • ndarray.byteswap

  • numpy.nan_to_num

Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).



There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.



Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.



Notes on view vs copy return values



One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.



For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.



In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:



arr = np.array([[1, 2],
[3, 4],
[5, 6]])

arrflat = arr.flatten()
assert arrflat.base is not arr

arrravel = arr.ravel()
assert arrravel.base is arr





share|improve this answer














Functions that mutate in place



Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.



Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:



  • ndarray.resize

  • ndarray.sort

  • All of the in-place arithmetic operators (eg +=, *=, /=, etc)

  • numpy.fill_diagonal

  • numpy.random.shuffle

  • ndarray.partition

and here's a list of the functions/methods that can optionally mutate in place:



  • ndarray.byteswap

  • numpy.nan_to_num

Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).



There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.



Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.



Notes on view vs copy return values



One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.



For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.



In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:



arr = np.array([[1, 2],
[3, 4],
[5, 6]])

arrflat = arr.flatten()
assert arrflat.base is not arr

arrravel = arr.ravel()
assert arrravel.base is arr






share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 11 at 5:14

























answered Nov 11 at 4:30









tel

5,68011430




5,68011430











  • I'm not sure he's asking about views vs. copies. Both are new array objects.
    – hpaulj
    Nov 11 at 4:40
















  • I'm not sure he's asking about views vs. copies. Both are new array objects.
    – hpaulj
    Nov 11 at 4:40















I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40




I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40

















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53245763%2fwhich-numpy-operations-copy-and-which-mutate%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Use pre created SQLite database for Android project in kotlin

Darth Vader #20

Ondo