Pfthb

I'm attempting to create a parser to translate a "custom" file into JSON so I can more easily manipulate its contents (For argument's sake, call the "custom" formate a .qwerty).

I've already created a Lexer which breaks down the file into individual lexemes (tokens) which structure is [token_type, token_value]. Now I am struggling to parse the lexemes into their correct dictionaries, as it is difficult to insert data into a sub-sub-dictionary since Keys aren't constant. As well as insert data into arrays stored in dictionaries.

It should be noted I am attempting to sequentially parse tokens into an actual python json object then dump the json object.

An example of the file can be seen below, along with what the end result is meant to resemble.

FILE: ABC.querty

Dict_abc_1

 Dict_abc_2
 HeaderGUID="";
 Version_TPI="999";
 EncryptionType="0";
 

 Dict_abc_3
 FamilyName="John Doe";
 

 Dict_abc_4
 Array_abc
 TimeStamp="2018-11-07 01:00:00"; otherinfo="";
 TimeStamp="2018-11-07 01:00:00"; otherinfo="";
 TimeStamp="2018-11-07 01:00:00"; otherinfo="";
 TimeStamp="2018-11-07 02:53:57"; otherinfo="";
 TimeStamp="2018-11-07 02:53:57"; otherinfo="";
 

 Dict_abc_5
 LastContact="2018-11-08 01:00:00";
 BatteryStatus=99;
 BUStatus=PowerOn;
 LastCallTime="2018-11-08 01:12:46";
 LastSuccessPoll="2018-11-08 01:12:46";
 CallResult=Successful;
 
 

Code=999999;

FILE: ABC.json

 
 "Dict_abc_1":
 "Dict_abc_2":
 "HeaderGUID":"",
 "Version_TPI":"999",
 "EncryptionType":"0"
 ,

 "Dict_abc_3":
 "FamilyName":"John Doe"
 ,

 "Dict_abc_4":
 "Array_abc":[
 "TimeStamp":"2018-11-07 01:00:00", "otherinfo":"",
 "TimeStamp":"2018-11-07 01:00:00", "otherinfo":"",
 "TimeStamp":"2018-11-07 01:00:00", "otherinfo":"",
 "TimeStamp":"2018-11-07 02:53:57", "otherinfo":"",
 "TimeStamp":"2018-11-07 02:53:57", "otherinfo":""
 ],

 "Dict_abc_5":
 "LastContact":"2018-11-08 01:00:00",
 "BatteryStatus":99,
 "BUStatus":"PowerOn",
 "LastCallTime":"2018-11-08 01:12:46",
 "LastSuccessPoll":"2018-11-08 01:12:46",
 "CallResult":"Successful"
 
 
 ,
 "Code":999999

Additional token information,
Token types can either be (with possible values)

• 
  IDENTIFIER contain the name of the variable identifier


• 
  VARIABLE containing actual data belonging to the parent IDENTIFIER


• 
  OPERATOR equal "="


• 
  OPEN_BRACKET equal ""


• 
  CLOSE_BRACKET equal ""

An example of ABC.querty's lexemes can be seen HERE

fundamental logical extract of main.py

def main():
 content = open_file(file_name) ## read file

 lexer = Lexer(content) ## create lexer class
 tokens = lexer.tokenize() ## create lexems as seen in pastebin

 parser = Parser(tokens).parse() ## create parser class given tokens 

 print(json.dumps(parser, sort_keys=True,indent=4, separators=(',', ': ')))

parser.py

import re

class Parser(object):
 def __init__(self, tokens):
 self.tokens = tokens
 self.token_index = 0
 self.json_object = 
 self.current_object = 
 self.path = [self.json_object]

 def parse(self):
 while self.token_index < len(self.tokens):

 token = self.getToken()
 token_type = token[0]
 token_value = token[1]

 print("%s t %s" % (token_type, token_value))

 if token_type in "IDENTIFIER":
 self.increment()
 identifier_type = self.getToken()
 if identifier_type[0] in "OPEN_BRACKET":
 identifier_two_type = self.getToken(1)
 if identifier_two_type[0] in ["OPERATOR","IDENTIFIER"]:
 ## make dict in current dict 
 pass

 elif identifier_two_type[0] in "OPEN_BRACKET":
 ## make array in current dict 
 pass

 elif identifier_type[0] in "OPERATOR":
 ## insert data into current dict
 pass


 if token_type in "CLOSE_BRACKET":
 identifier_type = self.getToken()
 if "OPEN_BRACKET" in identifier_type[0]:
 #still in array of current dict
 pass
 elif "IDENTIFIER" in identifier_type[0]:
 self.changeDirectory()

 else:
 #end script
 pass

 self.increment()
 print(self.path)
 return self.json_object


 def changeDirectory(self):
 if len(self.path) > 0:
 self.path = self.path.pop()
 self.current_object = -1

 def increment(self):
 if self.token_index < len(self.tokens):
 self.token_index+=1

 def getToken(self, x=0):
 return self.tokens[self.token_index+x]

Additional parse information,
Currently, I was trying to store the current dictionary in a path array to allow me to insert into dictionaries and arrays within dictionaries.

Any suggestions or solutions are very much appreciated,

Thanks.

Last time I solved this problem I find out that finite-state machine is very helpful. I want to recommend the way after you have tokens but I don't know how it's called in english. The principle is: you go through tokens and add one by one on stack. After adding on stack you are checking stack for some rules. Like you combine primitive tokens into expressions that might be a part of more complex expressions.

For example "FamilyName":"John Doe". Tokens are "FamilyName", : and "John Doe".

You add first token on stack.
stack = ["FamilyName"].
Rule 1: str_obj -> E. So you create Expression(type='str', value="FamilyName") and stack is now stack = [Expression].

Then you add next token.
stack = [Expression, ':']. No rules for ':'. Go next.

stack = [Expression, ':', "FamilyName"]. Again we meet rule 1. So stack becomes stack = [Expression, ':', Expression]. Then we see another rule. Rule 2: E:E -> E. Use it like Expression(type='kv_pair, value=(Expression, Expression)). And stack becomes stack=[Expression].

And if you describes all the rules it will work like that. Hope it helps.