As a library developer, how to prevent users from using an object across multiple Threads/Processes?










1















I am building a library for Python and noticed that one of my users was using the objects incorrectly, by sharing it between 2 Processes.



So, I wanted to bake in a way to prevent the user from doing that.



One way would be to add a "check" before every method call, using a decorator, but that adds a bit of overhead on each method call.



import multiprocessing
import threading


def is_parent():
return (
multiprocessing.current_process().name == "MainProcess"
and threading.current_thread() == threading.main_thread()
)


def process_unsafe(fn):
def wrapper(*args, **kwargs):
if not is_parent():
raise RuntimeError("Not allowed!")
fn(*args, **kwargs)

return wrapper


class NonProcessSafe:
@process_unsafe
def foo(self):
pass

@process_unsafe
def bar(self):
pass


nps = NonProcessSafe()

nps.foo()
nps.bar()


def child():
# complains
nps.foo()


p = multiprocessing.Process(target=child)
p.start()
p.join()


Is there any more efficient way to achieve this?










share|improve this question

















  • 3





    Definitely mention it in the documentation. I doubt there's a good way to close all the loopholes with code, so the only winning game might be not to play.

    – timgeb
    Nov 14 '18 at 7:31











  • Please give details why it is incorrect usage to share these objects between processes or threads. What bad effects will happen if you share? (Sharing between processes - how? Via shared memory?)

    – digitalarbeiter
    Nov 14 '18 at 7:38











  • Apparently, the documentation doesn't seem to be enough of a barrier... zproc.readthedocs.io/en/latest/user/…

    – Dev Aggarwal
    Nov 14 '18 at 7:41











  • I have it written over all the objects' API docs too, but python makes it easy to get this wrong :(

    – Dev Aggarwal
    Nov 14 '18 at 7:43











  • @digitalarbeiter Okay will plan to write a detailed page on this.

    – Dev Aggarwal
    Nov 14 '18 at 7:43















1















I am building a library for Python and noticed that one of my users was using the objects incorrectly, by sharing it between 2 Processes.



So, I wanted to bake in a way to prevent the user from doing that.



One way would be to add a "check" before every method call, using a decorator, but that adds a bit of overhead on each method call.



import multiprocessing
import threading


def is_parent():
return (
multiprocessing.current_process().name == "MainProcess"
and threading.current_thread() == threading.main_thread()
)


def process_unsafe(fn):
def wrapper(*args, **kwargs):
if not is_parent():
raise RuntimeError("Not allowed!")
fn(*args, **kwargs)

return wrapper


class NonProcessSafe:
@process_unsafe
def foo(self):
pass

@process_unsafe
def bar(self):
pass


nps = NonProcessSafe()

nps.foo()
nps.bar()


def child():
# complains
nps.foo()


p = multiprocessing.Process(target=child)
p.start()
p.join()


Is there any more efficient way to achieve this?










share|improve this question

















  • 3





    Definitely mention it in the documentation. I doubt there's a good way to close all the loopholes with code, so the only winning game might be not to play.

    – timgeb
    Nov 14 '18 at 7:31











  • Please give details why it is incorrect usage to share these objects between processes or threads. What bad effects will happen if you share? (Sharing between processes - how? Via shared memory?)

    – digitalarbeiter
    Nov 14 '18 at 7:38











  • Apparently, the documentation doesn't seem to be enough of a barrier... zproc.readthedocs.io/en/latest/user/…

    – Dev Aggarwal
    Nov 14 '18 at 7:41











  • I have it written over all the objects' API docs too, but python makes it easy to get this wrong :(

    – Dev Aggarwal
    Nov 14 '18 at 7:43











  • @digitalarbeiter Okay will plan to write a detailed page on this.

    – Dev Aggarwal
    Nov 14 '18 at 7:43













1












1








1


0






I am building a library for Python and noticed that one of my users was using the objects incorrectly, by sharing it between 2 Processes.



So, I wanted to bake in a way to prevent the user from doing that.



One way would be to add a "check" before every method call, using a decorator, but that adds a bit of overhead on each method call.



import multiprocessing
import threading


def is_parent():
return (
multiprocessing.current_process().name == "MainProcess"
and threading.current_thread() == threading.main_thread()
)


def process_unsafe(fn):
def wrapper(*args, **kwargs):
if not is_parent():
raise RuntimeError("Not allowed!")
fn(*args, **kwargs)

return wrapper


class NonProcessSafe:
@process_unsafe
def foo(self):
pass

@process_unsafe
def bar(self):
pass


nps = NonProcessSafe()

nps.foo()
nps.bar()


def child():
# complains
nps.foo()


p = multiprocessing.Process(target=child)
p.start()
p.join()


Is there any more efficient way to achieve this?










share|improve this question














I am building a library for Python and noticed that one of my users was using the objects incorrectly, by sharing it between 2 Processes.



So, I wanted to bake in a way to prevent the user from doing that.



One way would be to add a "check" before every method call, using a decorator, but that adds a bit of overhead on each method call.



import multiprocessing
import threading


def is_parent():
return (
multiprocessing.current_process().name == "MainProcess"
and threading.current_thread() == threading.main_thread()
)


def process_unsafe(fn):
def wrapper(*args, **kwargs):
if not is_parent():
raise RuntimeError("Not allowed!")
fn(*args, **kwargs)

return wrapper


class NonProcessSafe:
@process_unsafe
def foo(self):
pass

@process_unsafe
def bar(self):
pass


nps = NonProcessSafe()

nps.foo()
nps.bar()


def child():
# complains
nps.foo()


p = multiprocessing.Process(target=child)
p.start()
p.join()


Is there any more efficient way to achieve this?







python multithreading multiprocessing






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 14 '18 at 7:26









Dev AggarwalDev Aggarwal

352212




352212







  • 3





    Definitely mention it in the documentation. I doubt there's a good way to close all the loopholes with code, so the only winning game might be not to play.

    – timgeb
    Nov 14 '18 at 7:31











  • Please give details why it is incorrect usage to share these objects between processes or threads. What bad effects will happen if you share? (Sharing between processes - how? Via shared memory?)

    – digitalarbeiter
    Nov 14 '18 at 7:38











  • Apparently, the documentation doesn't seem to be enough of a barrier... zproc.readthedocs.io/en/latest/user/…

    – Dev Aggarwal
    Nov 14 '18 at 7:41











  • I have it written over all the objects' API docs too, but python makes it easy to get this wrong :(

    – Dev Aggarwal
    Nov 14 '18 at 7:43











  • @digitalarbeiter Okay will plan to write a detailed page on this.

    – Dev Aggarwal
    Nov 14 '18 at 7:43












  • 3





    Definitely mention it in the documentation. I doubt there's a good way to close all the loopholes with code, so the only winning game might be not to play.

    – timgeb
    Nov 14 '18 at 7:31











  • Please give details why it is incorrect usage to share these objects between processes or threads. What bad effects will happen if you share? (Sharing between processes - how? Via shared memory?)

    – digitalarbeiter
    Nov 14 '18 at 7:38











  • Apparently, the documentation doesn't seem to be enough of a barrier... zproc.readthedocs.io/en/latest/user/…

    – Dev Aggarwal
    Nov 14 '18 at 7:41











  • I have it written over all the objects' API docs too, but python makes it easy to get this wrong :(

    – Dev Aggarwal
    Nov 14 '18 at 7:43











  • @digitalarbeiter Okay will plan to write a detailed page on this.

    – Dev Aggarwal
    Nov 14 '18 at 7:43







3




3





Definitely mention it in the documentation. I doubt there's a good way to close all the loopholes with code, so the only winning game might be not to play.

– timgeb
Nov 14 '18 at 7:31





Definitely mention it in the documentation. I doubt there's a good way to close all the loopholes with code, so the only winning game might be not to play.

– timgeb
Nov 14 '18 at 7:31













Please give details why it is incorrect usage to share these objects between processes or threads. What bad effects will happen if you share? (Sharing between processes - how? Via shared memory?)

– digitalarbeiter
Nov 14 '18 at 7:38





Please give details why it is incorrect usage to share these objects between processes or threads. What bad effects will happen if you share? (Sharing between processes - how? Via shared memory?)

– digitalarbeiter
Nov 14 '18 at 7:38













Apparently, the documentation doesn't seem to be enough of a barrier... zproc.readthedocs.io/en/latest/user/…

– Dev Aggarwal
Nov 14 '18 at 7:41





Apparently, the documentation doesn't seem to be enough of a barrier... zproc.readthedocs.io/en/latest/user/…

– Dev Aggarwal
Nov 14 '18 at 7:41













I have it written over all the objects' API docs too, but python makes it easy to get this wrong :(

– Dev Aggarwal
Nov 14 '18 at 7:43





I have it written over all the objects' API docs too, but python makes it easy to get this wrong :(

– Dev Aggarwal
Nov 14 '18 at 7:43













@digitalarbeiter Okay will plan to write a detailed page on this.

– Dev Aggarwal
Nov 14 '18 at 7:43





@digitalarbeiter Okay will plan to write a detailed page on this.

– Dev Aggarwal
Nov 14 '18 at 7:43












1 Answer
1






active

oldest

votes


















0














As @timgeb mentioned in the comments, there's probably no way to close all the loopholes. Furthermore I'm not sure I fully understand your problem and process/thread/state sharing model...



Anyway, here's an Ansatz that might get you somewhere, at least with regards to processes (in the Unix/Linux sense):



import os

class State:

def __init__(self):
self._pid = os.getpid()
self._state =

def __setitem__(self, key, value):
if os.getpid() != self._pid:
raise RuntimeError("this process must not change state")
self._state[key] = value


The State instance remembers the ID of the process that created it, and can then check against that ID in all (relevant) methods. This will of course require the State be created by its "owner" process, in which case it is invisible to any parent or sibling process anyway, and visible only to itself and its children.



For threads, use threading.get_ident() instead of os.getpid().



If it is always the "main" process that creates the state and then forks worker children, instead you could (in the main process) temporarily mark the state as read-only, fork the worker child so it gets a read-only copy, and then in the main process switch state back to read-write:



class State:
def __init__(self):
self._read_only = False
self._state =
def __setattr__(self, key, value):
if self._read_only:
raise RuntimeError("read only!")
self._state[key] = value


In main process:



def new_worker(fct):
state = State()
state._read_only = True
pid = os.fork() # or whatever you use to create a new process
if pid:
fct() # state is _read-only=True here in child process
else:
state._read_only = False
state_list.append((pid, state))





share|improve this answer























  • Thank you for taking the time to understand the problem and my library, but this doesn't also apply to other classes like Context and Process. I still have to make the if os.getpid() != self._pid: check, which will add a overhead, as I mentioned in the question. It is better than the "MainProcess" thing I'm doing..

    – Dev Aggarwal
    Nov 14 '18 at 9:14












  • > I'm not sure I fully understand your problem and process/thread/state sharing model -- It lets you do shared state parallelism without actually having shared memory objects, by operator overloading and message passing :)

    – Dev Aggarwal
    Nov 14 '18 at 9:17












  • Okay, but what's the scenario where one process tries to use objects from another process? I can think of two scenarios: a) it's a shared object, and your __setitem__/message passing takes care of it, and b) the object was created in the main/parent process and inherited into all worker/children as local copies (via the copy on write mechanism of the operating system), so changes stay local to each child process. Is that the issue?

    – digitalarbeiter
    Nov 14 '18 at 10:04











  • For the State object specifically, the issue is that the message passing (all dict methods) depend on a few non-thread-safe objects, ZMQ Context, Sockets, an "identity" that helps the framework separate different clients etc. For others, like Context, there are the Process objects which are neither serialize-able nor thread-safe. Basically, the whole architecture is designed to eliminate the need for shared objects, and I want to have a way to prevent the user from doing it.

    – Dev Aggarwal
    Nov 14 '18 at 10:46












  • I wonder if I can tap into the object copying mechanism in CPython...

    – Dev Aggarwal
    Nov 14 '18 at 10:46










Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53295048%2fas-a-library-developer-how-to-prevent-users-from-using-an-object-across-multipl%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














As @timgeb mentioned in the comments, there's probably no way to close all the loopholes. Furthermore I'm not sure I fully understand your problem and process/thread/state sharing model...



Anyway, here's an Ansatz that might get you somewhere, at least with regards to processes (in the Unix/Linux sense):



import os

class State:

def __init__(self):
self._pid = os.getpid()
self._state =

def __setitem__(self, key, value):
if os.getpid() != self._pid:
raise RuntimeError("this process must not change state")
self._state[key] = value


The State instance remembers the ID of the process that created it, and can then check against that ID in all (relevant) methods. This will of course require the State be created by its "owner" process, in which case it is invisible to any parent or sibling process anyway, and visible only to itself and its children.



For threads, use threading.get_ident() instead of os.getpid().



If it is always the "main" process that creates the state and then forks worker children, instead you could (in the main process) temporarily mark the state as read-only, fork the worker child so it gets a read-only copy, and then in the main process switch state back to read-write:



class State:
def __init__(self):
self._read_only = False
self._state =
def __setattr__(self, key, value):
if self._read_only:
raise RuntimeError("read only!")
self._state[key] = value


In main process:



def new_worker(fct):
state = State()
state._read_only = True
pid = os.fork() # or whatever you use to create a new process
if pid:
fct() # state is _read-only=True here in child process
else:
state._read_only = False
state_list.append((pid, state))





share|improve this answer























  • Thank you for taking the time to understand the problem and my library, but this doesn't also apply to other classes like Context and Process. I still have to make the if os.getpid() != self._pid: check, which will add a overhead, as I mentioned in the question. It is better than the "MainProcess" thing I'm doing..

    – Dev Aggarwal
    Nov 14 '18 at 9:14












  • > I'm not sure I fully understand your problem and process/thread/state sharing model -- It lets you do shared state parallelism without actually having shared memory objects, by operator overloading and message passing :)

    – Dev Aggarwal
    Nov 14 '18 at 9:17












  • Okay, but what's the scenario where one process tries to use objects from another process? I can think of two scenarios: a) it's a shared object, and your __setitem__/message passing takes care of it, and b) the object was created in the main/parent process and inherited into all worker/children as local copies (via the copy on write mechanism of the operating system), so changes stay local to each child process. Is that the issue?

    – digitalarbeiter
    Nov 14 '18 at 10:04











  • For the State object specifically, the issue is that the message passing (all dict methods) depend on a few non-thread-safe objects, ZMQ Context, Sockets, an "identity" that helps the framework separate different clients etc. For others, like Context, there are the Process objects which are neither serialize-able nor thread-safe. Basically, the whole architecture is designed to eliminate the need for shared objects, and I want to have a way to prevent the user from doing it.

    – Dev Aggarwal
    Nov 14 '18 at 10:46












  • I wonder if I can tap into the object copying mechanism in CPython...

    – Dev Aggarwal
    Nov 14 '18 at 10:46















0














As @timgeb mentioned in the comments, there's probably no way to close all the loopholes. Furthermore I'm not sure I fully understand your problem and process/thread/state sharing model...



Anyway, here's an Ansatz that might get you somewhere, at least with regards to processes (in the Unix/Linux sense):



import os

class State:

def __init__(self):
self._pid = os.getpid()
self._state =

def __setitem__(self, key, value):
if os.getpid() != self._pid:
raise RuntimeError("this process must not change state")
self._state[key] = value


The State instance remembers the ID of the process that created it, and can then check against that ID in all (relevant) methods. This will of course require the State be created by its "owner" process, in which case it is invisible to any parent or sibling process anyway, and visible only to itself and its children.



For threads, use threading.get_ident() instead of os.getpid().



If it is always the "main" process that creates the state and then forks worker children, instead you could (in the main process) temporarily mark the state as read-only, fork the worker child so it gets a read-only copy, and then in the main process switch state back to read-write:



class State:
def __init__(self):
self._read_only = False
self._state =
def __setattr__(self, key, value):
if self._read_only:
raise RuntimeError("read only!")
self._state[key] = value


In main process:



def new_worker(fct):
state = State()
state._read_only = True
pid = os.fork() # or whatever you use to create a new process
if pid:
fct() # state is _read-only=True here in child process
else:
state._read_only = False
state_list.append((pid, state))





share|improve this answer























  • Thank you for taking the time to understand the problem and my library, but this doesn't also apply to other classes like Context and Process. I still have to make the if os.getpid() != self._pid: check, which will add a overhead, as I mentioned in the question. It is better than the "MainProcess" thing I'm doing..

    – Dev Aggarwal
    Nov 14 '18 at 9:14












  • > I'm not sure I fully understand your problem and process/thread/state sharing model -- It lets you do shared state parallelism without actually having shared memory objects, by operator overloading and message passing :)

    – Dev Aggarwal
    Nov 14 '18 at 9:17












  • Okay, but what's the scenario where one process tries to use objects from another process? I can think of two scenarios: a) it's a shared object, and your __setitem__/message passing takes care of it, and b) the object was created in the main/parent process and inherited into all worker/children as local copies (via the copy on write mechanism of the operating system), so changes stay local to each child process. Is that the issue?

    – digitalarbeiter
    Nov 14 '18 at 10:04











  • For the State object specifically, the issue is that the message passing (all dict methods) depend on a few non-thread-safe objects, ZMQ Context, Sockets, an "identity" that helps the framework separate different clients etc. For others, like Context, there are the Process objects which are neither serialize-able nor thread-safe. Basically, the whole architecture is designed to eliminate the need for shared objects, and I want to have a way to prevent the user from doing it.

    – Dev Aggarwal
    Nov 14 '18 at 10:46












  • I wonder if I can tap into the object copying mechanism in CPython...

    – Dev Aggarwal
    Nov 14 '18 at 10:46













0












0








0







As @timgeb mentioned in the comments, there's probably no way to close all the loopholes. Furthermore I'm not sure I fully understand your problem and process/thread/state sharing model...



Anyway, here's an Ansatz that might get you somewhere, at least with regards to processes (in the Unix/Linux sense):



import os

class State:

def __init__(self):
self._pid = os.getpid()
self._state =

def __setitem__(self, key, value):
if os.getpid() != self._pid:
raise RuntimeError("this process must not change state")
self._state[key] = value


The State instance remembers the ID of the process that created it, and can then check against that ID in all (relevant) methods. This will of course require the State be created by its "owner" process, in which case it is invisible to any parent or sibling process anyway, and visible only to itself and its children.



For threads, use threading.get_ident() instead of os.getpid().



If it is always the "main" process that creates the state and then forks worker children, instead you could (in the main process) temporarily mark the state as read-only, fork the worker child so it gets a read-only copy, and then in the main process switch state back to read-write:



class State:
def __init__(self):
self._read_only = False
self._state =
def __setattr__(self, key, value):
if self._read_only:
raise RuntimeError("read only!")
self._state[key] = value


In main process:



def new_worker(fct):
state = State()
state._read_only = True
pid = os.fork() # or whatever you use to create a new process
if pid:
fct() # state is _read-only=True here in child process
else:
state._read_only = False
state_list.append((pid, state))





share|improve this answer













As @timgeb mentioned in the comments, there's probably no way to close all the loopholes. Furthermore I'm not sure I fully understand your problem and process/thread/state sharing model...



Anyway, here's an Ansatz that might get you somewhere, at least with regards to processes (in the Unix/Linux sense):



import os

class State:

def __init__(self):
self._pid = os.getpid()
self._state =

def __setitem__(self, key, value):
if os.getpid() != self._pid:
raise RuntimeError("this process must not change state")
self._state[key] = value


The State instance remembers the ID of the process that created it, and can then check against that ID in all (relevant) methods. This will of course require the State be created by its "owner" process, in which case it is invisible to any parent or sibling process anyway, and visible only to itself and its children.



For threads, use threading.get_ident() instead of os.getpid().



If it is always the "main" process that creates the state and then forks worker children, instead you could (in the main process) temporarily mark the state as read-only, fork the worker child so it gets a read-only copy, and then in the main process switch state back to read-write:



class State:
def __init__(self):
self._read_only = False
self._state =
def __setattr__(self, key, value):
if self._read_only:
raise RuntimeError("read only!")
self._state[key] = value


In main process:



def new_worker(fct):
state = State()
state._read_only = True
pid = os.fork() # or whatever you use to create a new process
if pid:
fct() # state is _read-only=True here in child process
else:
state._read_only = False
state_list.append((pid, state))






share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 14 '18 at 9:07









digitalarbeiterdigitalarbeiter

1,7101112




1,7101112












  • Thank you for taking the time to understand the problem and my library, but this doesn't also apply to other classes like Context and Process. I still have to make the if os.getpid() != self._pid: check, which will add a overhead, as I mentioned in the question. It is better than the "MainProcess" thing I'm doing..

    – Dev Aggarwal
    Nov 14 '18 at 9:14












  • > I'm not sure I fully understand your problem and process/thread/state sharing model -- It lets you do shared state parallelism without actually having shared memory objects, by operator overloading and message passing :)

    – Dev Aggarwal
    Nov 14 '18 at 9:17












  • Okay, but what's the scenario where one process tries to use objects from another process? I can think of two scenarios: a) it's a shared object, and your __setitem__/message passing takes care of it, and b) the object was created in the main/parent process and inherited into all worker/children as local copies (via the copy on write mechanism of the operating system), so changes stay local to each child process. Is that the issue?

    – digitalarbeiter
    Nov 14 '18 at 10:04











  • For the State object specifically, the issue is that the message passing (all dict methods) depend on a few non-thread-safe objects, ZMQ Context, Sockets, an "identity" that helps the framework separate different clients etc. For others, like Context, there are the Process objects which are neither serialize-able nor thread-safe. Basically, the whole architecture is designed to eliminate the need for shared objects, and I want to have a way to prevent the user from doing it.

    – Dev Aggarwal
    Nov 14 '18 at 10:46












  • I wonder if I can tap into the object copying mechanism in CPython...

    – Dev Aggarwal
    Nov 14 '18 at 10:46

















  • Thank you for taking the time to understand the problem and my library, but this doesn't also apply to other classes like Context and Process. I still have to make the if os.getpid() != self._pid: check, which will add a overhead, as I mentioned in the question. It is better than the "MainProcess" thing I'm doing..

    – Dev Aggarwal
    Nov 14 '18 at 9:14












  • > I'm not sure I fully understand your problem and process/thread/state sharing model -- It lets you do shared state parallelism without actually having shared memory objects, by operator overloading and message passing :)

    – Dev Aggarwal
    Nov 14 '18 at 9:17












  • Okay, but what's the scenario where one process tries to use objects from another process? I can think of two scenarios: a) it's a shared object, and your __setitem__/message passing takes care of it, and b) the object was created in the main/parent process and inherited into all worker/children as local copies (via the copy on write mechanism of the operating system), so changes stay local to each child process. Is that the issue?

    – digitalarbeiter
    Nov 14 '18 at 10:04











  • For the State object specifically, the issue is that the message passing (all dict methods) depend on a few non-thread-safe objects, ZMQ Context, Sockets, an "identity" that helps the framework separate different clients etc. For others, like Context, there are the Process objects which are neither serialize-able nor thread-safe. Basically, the whole architecture is designed to eliminate the need for shared objects, and I want to have a way to prevent the user from doing it.

    – Dev Aggarwal
    Nov 14 '18 at 10:46












  • I wonder if I can tap into the object copying mechanism in CPython...

    – Dev Aggarwal
    Nov 14 '18 at 10:46
















Thank you for taking the time to understand the problem and my library, but this doesn't also apply to other classes like Context and Process. I still have to make the if os.getpid() != self._pid: check, which will add a overhead, as I mentioned in the question. It is better than the "MainProcess" thing I'm doing..

– Dev Aggarwal
Nov 14 '18 at 9:14






Thank you for taking the time to understand the problem and my library, but this doesn't also apply to other classes like Context and Process. I still have to make the if os.getpid() != self._pid: check, which will add a overhead, as I mentioned in the question. It is better than the "MainProcess" thing I'm doing..

– Dev Aggarwal
Nov 14 '18 at 9:14














> I'm not sure I fully understand your problem and process/thread/state sharing model -- It lets you do shared state parallelism without actually having shared memory objects, by operator overloading and message passing :)

– Dev Aggarwal
Nov 14 '18 at 9:17






> I'm not sure I fully understand your problem and process/thread/state sharing model -- It lets you do shared state parallelism without actually having shared memory objects, by operator overloading and message passing :)

– Dev Aggarwal
Nov 14 '18 at 9:17














Okay, but what's the scenario where one process tries to use objects from another process? I can think of two scenarios: a) it's a shared object, and your __setitem__/message passing takes care of it, and b) the object was created in the main/parent process and inherited into all worker/children as local copies (via the copy on write mechanism of the operating system), so changes stay local to each child process. Is that the issue?

– digitalarbeiter
Nov 14 '18 at 10:04





Okay, but what's the scenario where one process tries to use objects from another process? I can think of two scenarios: a) it's a shared object, and your __setitem__/message passing takes care of it, and b) the object was created in the main/parent process and inherited into all worker/children as local copies (via the copy on write mechanism of the operating system), so changes stay local to each child process. Is that the issue?

– digitalarbeiter
Nov 14 '18 at 10:04













For the State object specifically, the issue is that the message passing (all dict methods) depend on a few non-thread-safe objects, ZMQ Context, Sockets, an "identity" that helps the framework separate different clients etc. For others, like Context, there are the Process objects which are neither serialize-able nor thread-safe. Basically, the whole architecture is designed to eliminate the need for shared objects, and I want to have a way to prevent the user from doing it.

– Dev Aggarwal
Nov 14 '18 at 10:46






For the State object specifically, the issue is that the message passing (all dict methods) depend on a few non-thread-safe objects, ZMQ Context, Sockets, an "identity" that helps the framework separate different clients etc. For others, like Context, there are the Process objects which are neither serialize-able nor thread-safe. Basically, the whole architecture is designed to eliminate the need for shared objects, and I want to have a way to prevent the user from doing it.

– Dev Aggarwal
Nov 14 '18 at 10:46














I wonder if I can tap into the object copying mechanism in CPython...

– Dev Aggarwal
Nov 14 '18 at 10:46





I wonder if I can tap into the object copying mechanism in CPython...

– Dev Aggarwal
Nov 14 '18 at 10:46



















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53295048%2fas-a-library-developer-how-to-prevent-users-from-using-an-object-across-multipl%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Use pre created SQLite database for Android project in kotlin

Darth Vader #20

Ondo