Banner Grabbing Script in Python

[twinbaby]

#! /usr/bin/python3
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect (("demo.testfire.net",80))
#send a basic http request
output="GET / HTTP/1.0\nHost: demo.testfire.net\n\n"
#s.send(output.encode('utf-8'))
s.send(output)
page = ""
while 1:
    data = s.recv(1024)
    if data == "":
        break
    page = page + data
#close the socket and print the results
s.close()
print (page)

I am trying to implement a simple crawler script in python3. But it is showing me the following error message.
TypeError: a bytes-like object is required, not 'str'

 

[davidktw]

#! /usr/bin/python3
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect (("demo.testfire.net",80))
#send a basic http request
output="GET / HTTP/1.0\nHost: demo.testfire.net\n\n"
#s.send(output.encode('utf-8'))
s.send(output)
page = ""
while 1:
    data = s.recv(1024)
    if data == "":
        break
    page = page + data
#close the socket and print the results
s.close()
print (page)

I am trying to implement a simple crawler script in python3. But it is showing me the following error message.
TypeError: a bytes-like object is required, not 'str'

The socket library is a low-level networking library. The send method do not take in string, it take in bytes array.
Hence you should be doing

output = b'GET / HTTP/1.0\nHost: demo.testfire.net\n\n'
s.send(output)

Refer to https://docs.python.org/3/library/socket.html for more information.

In fact, if your intention is to work on the HTTP layer, there isn't a need for you to use the low-level socket library.
Just the HTTP client library like https://docs.python.org/3/library/http.client.html will do

 

[twinbaby]

#! /usr/bin/python3
import socket
target_host = "demo.testfire.net"
#replace the IP with the machine you want ot connect
target_port = 80
#create a socket object
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#connect the client
client.connect((target_host,target_port))
#send some data
output="GET / HTTP/1.0\nHost: demo.testfire.net\n\n"
client.send(output)
#receive some data
response = client.recv(4096)
print response

I found another implementation that works. without the 'b

 

[davidktw]

I found another implementation that works. without the 'b

When you use the words "found" and "works", have you tested it yourself ?

 

[twinbaby]

When you use the words "found" and "works", have you tested it yourself ?

Yes, there is response header:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=588495970725CA72A497CA6ACB04BE6D; Path=/; HttpOnly
Content-Type: text/html;charset=ISO-8859-1
Date: Sun, 05 Dec 2021 09:28:32 GMT
Connection: close

 

[davidktw]

Yes, there is response header:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=588495970725CA72A497CA6ACB04BE6D; Path=/; HttpOnly
Content-Type: text/html;charset=ISO-8859-1
Date: Sun, 05 Dec 2021 09:28:32 GMT
Connection: close

Sure doesn't work on my python installation 3.9.6

$ python test4.py
Traceback (most recent call last):
  File "/Users/davidktw/test4.py", line 12, in <module>
    client.send(output)
TypeError: a bytes-like object is required, not 'str'

UPDATE: I believe you are using python2.

On further research
https://docs.python.org/3.0/whatsnew/3.0.html#text-vs-data-instead-of-unicode-vs-8-bithttps://stackoverflow.com/questions/29513683/difference-between-python3-and-python2-socket-send-data

 

[twinbaby]

Sure doesn't work on my python installation 3.9.6

$ python test4.py
Traceback (most recent call last):
  File "/Users/davidktw/test4.py", line 12, in <module>
    client.send(output)
TypeError: a bytes-like object is required, not 'str'

UPDATE: I believe you are using python2.

On further research
https://docs.python.org/3.0/whatsnew/3.0.html#text-vs-data-instead-of-unicode-vs-8-bithttps://stackoverflow.com/questions/29513683/difference-between-python3-and-python2-socket-send-data

I just use 2to3 command in python.
I also got the same error:
TypeError: a bytes-like object is required, not 'str'
After using: output = b'GET / HTTP/1.0\nHost: demo.testfire.net\n\n'

Edit:
Found the issue, my IDE is using python2 to run python3 code.

 

[davidktw]

I just use 2to3 command in python.
I also got the same error:
TypeError: a bytes-like object is required, not 'str'
After using: output = b'GET / HTTP/1.0\nHost: demo.testfire.net\n\n'

What are you trying to do ?
Referring to the first set of codes in your first post
here are the changes.

#! /usr/bin/python3
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect (("demo.testfire.net",80))
#send a basic http request
output=b"GET / HTTP/1.0\nHost: demo.testfire.net\n\n"
#s.send(output.encode('utf-8'))
s.send(output)
page = ""
while 1:
    data = s.recv(1024)
    if data == "":
        break
    print(data.decode('utf-8')) # ADDED BY davidktw
    page = page + data.decode('utf-8') # CHANGED TO UTF-8 decoding
#close the socket and print the results
s.close()
print (page)

Your flow is still wrong because it will never get out of the data receive loop as socket.recv is a blocking call.

 

[Trader11]

I found another implementation that works. without the 'b

Is this for oscp? Use requests library lah...easier

 

[Trader11]

The socket library is a low-level networking library. The send method do not take in string, it take in bytes array.
Hence you should be doing

output = b'GET / HTTP/1.0\nHost: demo.testfire.net\n\n'
s.send(output)

Refer to https://docs.python.org/3/library/socket.html for more information.

In fact, if your intention is to work on the HTTP layer, there isn't a need for you to use the low-level socket library.
Just the HTTP client library like https://docs.python.org/3/library/http.client.html will do

How do you learn socket programming in general?

 

[davidktw]

How do you learn socket programming in general?

First time I touch it is during my uni days coding SFTP server and client for a networking module.

During my early days of work, I touch a lot of networking when I'm designing antivirus and antispam solutions for email systems.

Even today I have been dealing with sockets related tasks.

Have I not share in this forum that Networking is an indispensable knowledge for software developers ? It's not a networking expert domain, it is a software developers' domain. Network expert are just more focus into ground and infrastructure, but software developers need to know deep enough to be productive in their daily work.

 

[davidktw]

@twinbaby This is probably what you want

#! /usr/bin/env python3
import socket
import select

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect (("demo.testfire.net",80))
#send a basic http request
output=b"GET / HTTP/1.0\nHost: demo.testfire.net\n\n"
#s.send(output.encode('utf-8'))
s.send(output)
page = ""
while True:
    ready = select.select([s], [], [], 2)
    if ready[0]:
        data = s.recv(4096)
        if len(data) <= 0 :
            break
        page = page + data.decode('utf-8')
#close the socket and print the results
s.close()
print (page)

 

[Trader11]

@twinbaby This is probably what you want

#! /usr/bin/env python3
import socket
import select

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect (("demo.testfire.net",80))
#send a basic http request
output=b"GET / HTTP/1.0\nHost: demo.testfire.net\n\n"
#s.send(output.encode('utf-8'))
s.send(output)
page = ""
while True:
    ready = select.select([s], [], [], 2)
    if ready[0]:
        data = s.recv(4096)
        if len(data) <= 0 :
            break
        page = page + data.decode('utf-8')
#close the socket and print the results
s.close()
print (page)

Bro you just helped him do his homework!

 

[Trader11]

First time I touch it is during my uni days coding SFTP server and client for a networking module.

During my early days of work, I touch a lot of networking when I'm designing antivirus and antispam solutions for email systems.

Even today I have been dealing with sockets related tasks.

Have I not share in this forum that Networking is an indispensable knowledge for software developers ? It's not a networking expert domain, it is a software developers' domain. Network expert are just more focus into ground and infrastructure, but software developers need to know deep enough to be productive in their daily work.

Any good books to read about socket programming that you recommend?

Do you use web sockets?

 

[davidktw]

Bro you just helped him do his homework!

Well that's for him to decide if he wants to pass or fail in the future.

 

[davidktw]

Any good books to read about socket programming that you recommend?

Do you use web sockets?

When it comes to socket programming, other than my initial introduction from my uni days network courses, the rest of my knowledge are all OJT, reading online articles and RFC and real implementation.

Networking is a very vast topic, so knowing just how TCP and UDP works is not enough. There are a lot of side topics like various casting approaches, application protocols, security, scalability, DNS, routing and many more that one will need to research and learn. Even knowing what products does best at what matters. It is not a matter of reading books. It is investigation, exposure and research works.

Not much of web sockets yet since most stuffs are still very versatile using the long polling. Web sockets have its own set of scalability issues and security concerns. So I have yet to implement it.

 

[twinbaby]

#! /usr/bin/env python3

This line useful, don't have to type python3 webconn.py just ./webconn.py

TypeError: can only concatenate str (not "bytes") to str

If python2 will work but python3 will show this error.

import socket
target_host = "demo.testfire.net"
#replace the IP with the machine you want to connect
target_port = 80
#create a socket object
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#connect the client
client.connect((target_host,target_port))
#send some data
output = b'GET / HTTP/1.0\nHost: demo.testfire.net\n\n'
client.send(output)
#receive some data
response = client.recv(4096)
print(response)

Alternative solution is the abovementioned, will work with python3.

I am currently using Geany editor.
Can press > button can execute the python script.
Downside is cannot catch error before I run the code.

 

[davidktw]

Learn to use the console.
The magic is in the console.

 

[twinbaby]

#! /usr/bin/env python3
#advance server that can perform the following
#1) accept multiple connections
#2) provide a list of the returned data
#3) socket for reading, writing and checking for error

import socket #for socket
import threading
import sys #for exit

bind_ip = "0.0.0.0"
bind_port = 10002

try:
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip,bind_port))
except socket.error as msg:
print ("Failed to create socket. Error code: " + str(msg[0]) + " , Error message : " + msg[1])
sys.exit()

#Start listening on socket
server.listen(10)
print("[*] listening on %s:%d" % (bind_ip,bind_port))

#this is our client-handling
def handle_client(client_socket):
#print out what the client needs
request = client_socket.recv(1024)
print("[*] Received: %s" % request)
#send back a packet
client_socket.send(b"ACK!")
client_socket.close()

while True:
#The accept function is called in a loop to keep
client,addr = server.accept()
print("[*] Accepted connection from %s:%d" % (addr[0],addr[1]))
#spin up our client thread to handle incoming
client_handler = threading.Thread(target = handle_client,args = (client,)).start()

I am in the midst of building a multi threaded python server:
print ("Failed to create socket. Error code: " + str(msg[0]) + " , Error message : " + msg[1])
TypeError: 'OSError' object is not subscriptable

2ndly, when I moved the handle_client to the bottom it was not detectable by the while True loop

Learn to use the console.
The magic is in the console.

Don't have plugin to do autoformatting indent
Must type python3 all the time to run instead of a play button >
Cannot navigate directory structure freely with a tree browser
Cannot autocomplete

 

[davidktw]

Don't have plugin to do autoformatting indent
Must type python3 all the time to run instead of a play button >
Cannot navigate directory structure freely with a tree browser
Cannot autocomplete

You know. Python is a language since 1981. You think there was any IDE back then ?

My only advice for you. "Don't be a baby".
What kind of S/W engineer you end up is all a matter of your choices.

More than half my time coding is
Terminal + VIM + Safari browser

And if it is not obvious to you why you need to type "python3" all the time is simply you don't know that for scripts to be executable is to set it so as such

chmod u+x ./test4.py

The question you need to ask yourself is "Why don't you know ?" Is it some kind of highly guarded secret ? No, it's because in the above replies, you have only decided to know what you want to know, but not investigating what are your options outside of what you know.
You need to think deep about it, instead of replying me what is not possible.
Just look at the way you type your source code in this forum versus how I did it.
If you really care, you would want to reflect on why.

This is the VIM cheatsheet I stick on a wall just right in front of my cubicle partition dividers until I no longer need to refer to them. That's how some software developers, like myself, trained to be productive. The option is for you to choose.

The magic has always been lurking, await for you to discover it.