[Programmer Lip Lai] Anyone interested in building tokenizer?

[nubitol]

or I think its called lexer?

the logic is for each character of the content of a file, determine the token, such as
1. matching ) of a ( while inside can contain another (
2. no count for ) that is inside of a string/comment, while comment can be // /* or --
3. when no 2 is inside a string, the string may contain string enclosure such as " or ' that is escaped, the lexer need to know whether it is an escaped enclosure instead of enclosure itself

this kind of programming sounds fun

 

[Resonance29]

? isn't it available on the web.

 

[potatoes_are_fluffy]

This isnt even a challenging coding interview question

 

[nubitol]

? isn't it available on the web.

its available in every programming language

but building one is fun

 

[nubitol]

This isnt even a challenging coding interview question

its for fun, not a serious programming stuff

 

[nubitol]

anyone interested in building one?

 

[needsomeadvice]

Use a stack (or two) can solve liao..

 

[*Sisyphus*]

i = "siaolang"
HOUGANG_CHALET = ["IMH", "siaolang"]
while i in HOUGANG_CHALET:
    print("I must take medicine")
else:
    print("Missy will scold")

 

[mrBooBoo]

theres framework used to build programming language call antlr.




Read HWZ Forum Rules!

 

[The_Davis]

I return my tokenizer to school liao

Nowadays no chance to use

 

[Perisher]

i = "siaolang"
HOUGANG_CHALET = ["IMH", "siaolang"]
while i in HOUGANG_CHALET:
print("I must take medicine")
else:
print("Missy will scold")

me no programmer but can understand sia...

 

[demon0192]

use regex can niaos lol

 

[*Sisyphus*]

me no programmer but can understand sia...

Python supposed to be human readable. As proven. But u must run it to get the gist..............

 

[ramlee]

I watch my nails grow more fun

 

[Perisher]

i = "siaolang"
HOUGANG_CHALET = ["IMH", "siaolang"]
while i in HOUGANG_CHALET:
print("I must take medicine")
else:
print("Missy will scold")

Got this whole chunk of error when i try putting whole thing into a python compiler to run...
https://www.w3schools.com/python/trypython.asp?filename=demo_default
Traceback (most recent call last):
File "/usr/lib/python3.8/py_compile.py", line 144, in compile
code = loader.source_to_code(source_bytes, dfile or file,
File "<frozen importlib._bootstrap_external>", line 846, in source_to_code
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "./prog.py", line 5
print("I must take medicine")
^
IndentationError: expected an indented block

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.8/py_compile.py", line 150, in compile
raise py_exc
py_compile.PyCompileError: Sorry: IndentationError: expected an indented block (prog.py, line 5)

 

[*Sisyphus*]

Got these whole chunk of error when i try putting whole thing into a python compiler to run...
https://www.w3schools.com/python/trypython.asp?filename=demo_default
Traceback (most recent call last):
File "/usr/lib/python3.8/py_compile.py", line 144, in compile
code = loader.source_to_code(source_bytes, dfile or file,
File "<frozen importlib._bootstrap_external>", line 846, in source_to_code
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "./prog.py", line 5
print("I must take medicine")
^
IndentationError: expected an indented block

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.8/py_compile.py", line 150, in compile
raise py_exc
py_compile.PyCompileError: Sorry: IndentationError: expected an indented block (prog.py, line 5)

must indent the print statements...

i = "siaolang"
HOUGANG_CHALET = ["IMH", "siaolang"]
while i in HOUGANG_CHALET:
          print("I must take medicine")
else:
          print("Missy will scold")

nothing much lah..just loop endlessly like siaolang....

 

[glarerder]

i = "siaolang"
HOUGANG_CHALET = ["IMH", "siaolang"]
while i in HOUGANG_CHALET:
print("I must take medicine")
else:
print("Missy will scold")

use while with else

 

[Phen8210]

this ts is a pro

 

[Phen8210]

Got this whole chunk of error when i try putting whole thing into a python compiler to run...
https://www.w3schools.com/python/trypython.asp?filename=demo_default
Traceback (most recent call last):
File "/usr/lib/python3.8/py_compile.py", line 144, in compile
code = loader.source_to_code(source_bytes, dfile or file,
File "<frozen importlib._bootstrap_external>", line 846, in source_to_code
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "./prog.py", line 5
print("I must take medicine")
^
IndentationError: expected an indented block

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.8/py_compile.py", line 150, in compile
raise py_exc
py_compile.PyCompileError: Sorry: IndentationError: expected an indented block (prog.py, line 5)

dont know u serious or trolling

 

[lalalalalala]

Like regex?