Pfthb

I am writing a Python 3 code where the task is to open about 550 files in a directory, read their contents and append it to a string variable 'all_text' which will be say around millions of line long as a single line.

The inefficient code I was using till now is as follows-

all_text += str(file_content)

But then I read that using 'join()' method is efficient, so I tried the following code-

all_text = ''.join(file_content)

The problem with this code is that this is removing the previously held contents of 'all_text' variable and writing the current file's content only!

How do I get around this problem?

Thanks for your help!

join() has a definition str.join(iterable) where iterable is a generator or a list or a set and so on. So it is helpful if you already have a list of strings read from the files and you are concatenating them using join.
For example

numList = ['1', '2', '3', '4']
seperator = ', '
print(seperator.join(numList))

numTuple = ('1', '2', '3', '4')
print(seperator.join(numTuple))

s1 = 'abc'
s2 = '123'

""" Each character of s2 is concatenated to the front of s1""" 
print('s1.join(s2):', s1.join(s2))

""" Each character of s1 is concatenated to the front of s2""" 
print('s2.join(s1):', s2.join(s1))

You can get all lines in a file using join like ''.join(readlines(f))

Now you can accomplish your task using join as follows using fileinput module

import fileinput
files= ['package-lock.json', 'sqldump.sql', 'felony.json', 'maindata.csv']
allfiles = fileinput.input(files)
all_text = ''.join(allfiles)

Refer to this answer to know the most efficient way to concat files into a string.

Suggestion: As you mentioned there would be millions of lines, did you consider the memory it is going to consume to store it in a variable? So it is better you do what you are planning to do on the fly while reading the lines instead of storing it in a variable.