Creating an HTTP GET polling service on a Raspberry Pi Client

Question

I have the following hardware:

1. 3 x Particle Photons. Each serve as an HTTP Server

2. 1 x Raspberry Pi 3 which will serve as an HTTP Client

Upon requesting an HTTP GET to any of the the Photons, the API returns:

{
  node: 1,
  uptime: 1234556,
  location: 'back',
  sensor: {
      Eu: {// Euler Angles from IMU
           h: 0, p: 0, r: 0
      },
      La: {// linear Acceleration values from IMU
           x: 0, y: 0, z: 0
      }
  }
}

I want to create a Polling scheme where the Raspberry Pi client performs an HTTP GET every 0.1 Second on each of the the 3 Servers.

I am not sure whether there is something like HTTP Polling and whether Asynchronous Libraries like Twisted by Python should be the one to be used.

I would like to gain some advice on how will a Multiple Server - Single Client model would function w.r.t. HTTP?

Reference

Each Particle Photon has the above mentioned JSON response to a HTTP GET Request.

The Raspberry Pi would serve as an HTTP Client, trying to get requests from each and every Particle Photons.

Answer 1

I found a good base solution for what @Chris Stratton refers to a kept alive TCP connection:

import socket

# Set up a TCP/IP socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)

# Connect as client to a selected server
# on a specified port
s.connect(("url_to_device_n",80))

# Protocol exchange - sends and receives
s.send("GET /answer_json HTTP/1.0\n\n")
while True:
    resp = s.recv(1024)
    if resp == "": break
    print resp, # here instead of print parse json

# Close the connection when completed
s.close()
print "\ndone"

You should make an eternal loop that waits 0.1s and then makes one of these steps between connect and close so that connect is called only once in start and close only when in extreme need to shut down everything.

With threads, once the previous works:

import urllib2 
from multiprocessing.dummy import Pool as ThreadPool 
import socket
import time
def sendData( urlToDevice ):
Set up a TCP/IP socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
   s.connect(("url_to_device_n",80))
   while True:
      time.sleep(0.1)
      # Protocol exchange - sends and receives
      s.send("GET /answer_json HTTP/1.0\n\n")
      while True:
         resp = s.recv(1024)
         if resp == "": break
         print resp, # here instead of print parse json
# Close the connection when completed
s.close()
print "\ndone"


#########################
This is main program:
#########################
urls = [
      'url_to_device_1', 
      'url_to_device_2',
      'url_to_device_3',
      '..',
      'url_to_device_n',
      ]
make the Pool of workers
pool = ThreadPool(n)
open the urls in their own threads
results = pool.map(sendData, urls)
close the pool and wait for the work to finish
pool.close() 
pool.join() 

Sources:

http://www.wellho.net/resources/ex.php4?item=y303/browser.py

https://stackoverflow.com/questions/2846653/how-to-use-threading-in-python

https://stackoverflow.com/questions/510348/how-can-i-make-a-time-delay-in-python

Answer 2

Maybe the following links can help you:

Basic client example: https://docs.python.org/2/library/asyncore.html#asyncore-example-basic-http-client

Basic echo server example: https://docs.python.org/2/library/asyncore.html#asyncore-example-basic-echo-server

Also, have you thought about using UDP protocol? it may be better...

And I would advice about HTTP/1.0, as far as I know, is not mandatory in its implementation, to keep connections alive, that was defined in HTTP/1.1; anyway it depends on the implementation, it can have or it can't.

---

import asyncore, socket

class HTTPClient(asyncore.dispatcher):

    def __init__(self, host, path):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.connect( (host, 80) )
        self.buffer = 'GET %s HTTP/1.0\r\n\r\n' % path

    def handle_connect(self):
        pass

    def handle_close(self):
        self.close()

    def handle_read(self):
        print self.recv(8192)

    def writable(self):
        return (len(self.buffer) > 0)

    def handle_write(self):
        sent = self.send(self.buffer)
        self.buffer = self.buffer[sent:]


client = HTTPClient('www.python.org', '/')
asyncore.loop()

---

import asyncore
import socket

class EchoHandler(asyncore.dispatcher_with_send):

    def handle_read(self):
        data = self.recv(8192)
        if data:
            self.send(data)

class EchoServer(asyncore.dispatcher):

    def __init__(self, host, port):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.set_reuse_addr()
        self.bind((host, port))
        self.listen(5)

    def handle_accept(self):
        pair = self.accept()
        if pair is not None:
            sock, addr = pair
            print 'Incoming connection from %s' % repr(addr)
            handler = EchoHandler(sock)

server = EchoServer('localhost', 8080)
asyncore.loop()