Pfthb

Suppose we have a population of size 10,000. We choose 1000 persons from the population, uniformly and independently, and ask them some question(s). The answers are random variables $X_i$.

Since we choose independently, we possibly can ask the same person multiple times (obtaining the same answers). In other words, we sample with replacement.

My point of view is: $X_i$ are independent and identically distributed random variables, and there is absolutely no difference here with sampling infinite population. For example, WLLN or CLT apply identically in both scenarios (sampling finite population with replacement and sampling infinite population).

Am I correct, or I am missing something?

Suppose 25% of the people in a population of 10,000 favor Proposition A.
You are going to poll 1000 of them asking their opinion on the proposition.
Suppose you take precautions to sample without replacement (to avoid interviewing
anyone twice). Then let the proportion of those interviewed who are in favor be $hat p_hyp.$

By contrast, suppose you take no such precautions (in effect,
sampling with replacement). Then let your estimate of the proportion in favor
be $hat p_bin.$ Your question is whether there will be an important
difference between the estimates $hat p_hyp$ and $hat p_bin.$

The R code below simulates 100,000 such polls according to each method, and
summarizes results to simulate $E(hat p_hyp)$ and $E(hat p_bin)$
and also the standard deviations of these estimates.

set.seed(1111); m = 10^5; n = 1000
pop = c(rep(1, 2500), rep(0,7500))
p.hyp = replicate(m, sum(sample(pop,n))/n) # sampling without replacement
mean(p.hyp); sd(p.hyp)
[1] 0.2500288
[1] 0.01297334
p.bin = replicate(m, sum(sample(pop,n,repl=T))/n) # sampling with replacement
mean(p.bin); sd(p.bin)
[1] 0.2500593
[1] 0.01369168

As you can see, both estimates have expected values very near to the true
proportion 25%. The estimate from sampling without replacement is a little
less variable.

A general "rule of thumb" is that if you are sampling 10% or less of the population,
the difference between sampling with and without replacement can viewed as negligible.

Specifically, if $X sim mathsfBinom(n, p),$ then $E(X) = np$ and
$Var(X) = np(1-p).$
If $Y$ is hypergeometric based on a sample of size $n$ from a population with $S$ successes and $N-S$
failures, then $E(Y) = n(S/N)$ and $Var(Y) = n(S/N)(1-S/N)[(N-n)/(N-1)].$
Letting $p = S/N$ this amounts to $E(Y) = np$ and
$Var(Y)=np(1-p)left(fracN-nN-1right).$

The factor $fracN-nN-1$ in
$Var(Y)$ is sometimes called the 'finite population correction'; notice
that if $n < N/10$ then this factor is close to $1.$ [It's called
'finite population correction' because there is no practical difference between
sampling with and without replacement when sampling from an infinite population.]

Of course this factor is not the only distinction between the two distributions. However,
for large $n$ and for $p$ not
too close to 0 or 1, both the binomial and hypergeometric distributions
can be approximated by a normal distribution that matches means and variances.
In terms of normal approximations the only distinction between sampling
with and without replacement often amounts to the finite population correction.