Pfthb

I'm new in Bayesian inference and I can't found the answer to this:

In real life scenario people use MCMC to compute the posterior distribution given the likelihood and the prior. Analytical solutions are not possibles. Bayesian people often say "we update our prior believe given some data to have the posterior". But something is not ok to me here: the posterior is never in the same form than the prior, right ? Unless you have a conjugate prior, which is really rare.

So what does it mean ? Your prior is a gamma distribution, and you end-up with a posterior with a completely different shape. Did you really update the prior distribution ? certainly no. We can't compare apples and oranges.

Does it means that we had a prior belief with a certain shape (gamma distribution), then we update this belief so we have a new shape (not even described analytically) as the output of the MCMC.

I'm very confused with this idea of "Bayesian update", because in practice, if you end-up with a complete new kind of distribution for the posterior, you can't reuse it as a new prior for the next batch of data, right ? So it means that this is just "a one shot update" of the prior belief.

It seems to me that the Bayesian update mean you update your belief in the sense of you change the prior distribution to something else. It's like saying "I've changed my mind, it is no longer a gamma distribution".

On the other hand when I follow some lectures, they never say that. They talk about Bayesian update related to the use of conjugate prior. In this case the math are nice so the posterior can be used as a prior. But this never happen in real life, right ? I mean, you don't use MCMC if you know that the posterior will be of the same family of the prior ?

Posteriors and priors are all distributions on the parameter space, so they can be compared as such, even if they are not of the same shape. If you are interested in performing multiple updates, i.e. to go from a posterior p(theta|Y1) to a second posterior p(theta|Y1,Y2), etc, then you can absolutely use p(theta|Y1) as a prior, and p(Y2|Y1,theta) as a likelihood. In fact some methods called sequential Monte Carlo can be used to recursively approximate a sequence of such posteriors, see for instance Chopin 2002, A sequential particle filter method for static models
, https://academic.oup.com/biomet/article-abstract/89/3/539/251804?redirectedFrom=PDF

Pierrot's answer is correct, but as this seems to be a question about intuition, I wanted to give what might be a more intuitive approach to thinking about the question.

I (+1)'d this question because it is in a way insightful; you are taking seriously that you need to understand what a method really means. However, when you take a higher-level view to the meaning behind the MCMC method, you also need to take a higher-level view to what it means to update the prior. If your prior belief can be described as a gamma distribution and your posterior belief can't, you have certainly updated your prior beliefs, and this does not bring any problems of "compar[ing] apples and oranges" (with respect to saying whether or not you've updated your prior). If you assign a different probability to various events in your posterior belief to the probabilities you assign in your prior belief, you have updated your beliefs, and this has nothing to do with whether your prior and posterior are the same type of probability distribution with just different parameters, or are represented by different types of distribution entirely.

In other words, a Bayesian update does not require the posterior distribution to be of the same form of the prior. This fact does not mean that only a "one-shot" update is possible either.

Let me give a simple example. Suppose your prior belief is that it is just as likely on any particular day that it will rain (call this event $B$) than that it will not. Then your prior is a discrete uniform distribution assigning probability $0.5$ to $B$ and $0.5$ to $neg B$. However, suppose you have also observe that it is April (call this event $A$) and you know both that the probability that a day is in April is $frac112$ and that the joint probability that it will rain on a day and that the day is a day in April is $frac348$. Then

$$
Pr(B|A) = fracPr(A cap B)Pr(A) = frac3/481/12 = 0.75
$$

Now your posterior belief assigns probability $0.75$ to $B$ and $0.25$ to $neg B$, which is not a uniform distribution, but we would still validly call this a Bayesian update. Moreover, this need not be a one-shot update, because you could easily further update your belief about $B$ if, for example, you observed dark clouds in the sky, etc.

This answer does not address another question that might be lurking within the question you asked -- how you use the posterior approximation from MCMC as a prior in another update. Perhaps Pierrot's answer can give you some insight there.