Pfthb

Currently, my dataframe is printing out 2 dataframes but I want it to simply be one dataframe with all the values passed on from omp_name_lists and comp_rating_lists. I know the function is passing values page by page so I would like to know how can I tell python wait for all the page values to append and then create_dataframe.

def get_data(get_comp_name,get_comp_rating):


 comp_name_lists, comp_rating_lists = ,
 for i, v in zip(get_comp_name, get_comp_rating):
 comp_name_lists.append(i.a.text)
 comp_rating_lists.append(v.text)

 return create_dataframe(comp_name_lists,comp_rating_lists)

def create_dataframe(comp_name_lists,comp_rating_lists): # This is where 
 """This gets all of our data that we scraped and stores into a pandas dataframe"""
 df = pd.DataFrame('CompanyName' : comp_name_lists, 'CompanyRating' : comp_rating_lists)
 df = df.reset_index(drop=True)
 print(df)

Desired Output:

 A B
0 a 3
1 b 7
2 c 1
3 d 2
4 e 8
5 f 9
6 g 7
7 h 8

If you want to be one dataframe with all the values, so you should have a global variable
because it will be not re-initialize, unless you restart script. If you pass value by return create_dataframe(comp_name_lists,comp_rating_lists), it will create a new space which independent from the others and the value of df is empty that mean you can not done what you expect in this way. In my consideration, create a dataframe after you finished fetching data

resultset = 'CompanyName' : , 'CompanyRating' : 

def get_data(get_comp_name,get_comp_rating):


 comp_name_lists, comp_rating_lists = ,
 for i, v in zip(get_comp_name, get_comp_rating):
 comp_name_lists.append(i.a.text)
 comp_rating_lists.append(v.text)

 resultset["CompanyName"] += comp_name_lists
 resultset["CompanyRating"] += comp_rating_lists

def create_dataframe(data): # This is where 
 """This gets all of our data that we scraped and stores into a pandas dataframe"""
 df = pd.DataFrame(data)
 df = df.reset_index(drop=True)
 print(df)