HTTP Integration

This section covers the support for HTTP-style messaging built into ansar-connect. That support requires special processing of the bytes sent and received over network connections. There are also two disparate models of execution to consider.

HTTP support exists to allow for communications with non-Ansar processes. A likely scenario is between a browser-based app and an Ansar process. There is significant commitment to HTTP-based, RPC-style network APIs in the browser ecosystem. Providing built-in support allows Ansar processes to join that ecosystem in the most seamless manner possible.

HTTP support also allows Ansar processes to connect to HTTP-based network APIs. The likely scenario is where an existing, 3rd party network API is providing valuable service to a community of browser apps. With HTTP support Ansar processes can join that community.

A final, less likely, scenario exists. This is where a project has an external requirement that HTTP will be used for communications between two points within a distributed system. This represents a forced use of HTTP between two Ansar processes, with the loss of the open, bi-directional messaging that Ansar provides by default.

Download materials supporting this guide and setup the environment with the following commands;

$ cd <folder-of-repos>
$ git clone https://github.com/mr-ansar/odds-and-ends.git
$ cd odds-and-ends
$ python3 -m venv .env
$ source .env/bin/activate
$ pip3 install ansar-connect

Listening To HTTP Clients

Adding a single parameter on the call to listen() establishes an HTTP-based service, at the specified address;

ar.listen(self, HostPort('127.0.0.1', 5080), api_server=[])

Connected clients will present requests in the standard format and these will result in HttpRequest messages being received at the expected object (i.e. self). An HttpResponse message is expected in return. There are no constraints imposed on the content of requests and the server is free to populate the request with whatever content it considers necessary.

While the receiving of requests and sending of responses at the server end occurs within an asynchronous environment, the HTTP protocol itself is synchronous, i.e. multiple concurrent requests are not supported. Until the expected response is passed back to the client, no further requests will be received at the server. HTTP sessions execute in a request-response, request-response lock step fashion. Fulfilment of a request within the server is free to leverage the local asynchronous environment.

Note

Object address information cannot be transferred across HTTP connections. This impacts the behaviour around advanced use of the return address and the use of messages that contain addresses. More information can be found here. All requests sent from client applications will be received by the object determined on the call to listen().

Receiving HTTP Forms

Inserting a list of message types into api_server enables the processing of HTTP forms (i.e. requests with a content type set to x-www-form-urlencoded);

API = [
    ar.Enquiry,
    ar.Ping,
]
ar.listen(self, HostPort('127.0.0.1', 5080), api_server=API)

There are two benefits to this arrangement. Firstly, ansar can automate further processing, breaking the complete message down into its constituent members. Secondly it can deliver distinct messages to the server by mapping the final name of the request URI to one of the listed message types. The former relieves the server of some detailed and fragile work and the latter means that requests can be automatically routed, e.g. to a function or FSM.

Note

The goal is to match the expectations of 3rd party client networking. Where these expectations are not met by the specifics of Ansar form processing, there is always the option to revert to the exchange of raw HTTP request and response messages (e.g. an empty api_server). All breakdown of messages and routing according to message contents is then the responsibility of the server.

The server now receives requests in the form of Enquiry and Ping messages. Values passed in the HTTP form are automatically loaded into the matching members of the messages. There is also a limited (and optional) conversion available. The type of the inbound values is always string, or ar.Unicode. If the matched destination member has a type other than string then basic conversions are applied;

Member	Form	Python
Boolean	true	bool
Byte, Integer[248]	2400	int
Unsigned[248]	4800	int
Float[48]	3.141	float
ClockTime	2021-02-21T04:34:55.31075	float
TimeSpan	1m7.125s	float
WorldTime	2021-02-21T04:37:03.539515	datetime.datetime
TimeDelta	00:01:46.097735	datetime.timedelta
UUID	cae03b55-346b-45d0-a321-2b6215241f48	uuid.UUID

For these conversions to be useful they must match the representations used by the client. Where this is not reliably true then the associated member should be declared with the string type and custom conversion implemented in the server. Complex types such as lists and maps - all directly supported over Ansar connections - cannot be handled in any automated fashion.

Within the full complement of request forms sent by a client there may be a few that are beyond the processing capabilities of Ansar. These forms should be omitted from the declared set of message types in api_server allowing the request to bypass form processing and instead arrive as an HttpRequest. When desired, this default behavior can also be disabled on the call to listen();

API = [
    ar.Enquiry,
    ar.Ping,
]
ar.listen(self, HostPort('127.0.0.1', 5080), api_server=API, default_to_request=False)

Where the received request URI does not match any of the listed messages, a standard error response (i.e. status code 400) is returned to the client, without any involvement from the server.

Enabling Full Ansar Messaging Over HTTP

For very specific circumstances it is possible to enable the transfer of complete request and response messages, using the application/json content type. Requests can be routed in the normal ways and all types (except addresses) are supported within the messages. This behavior is enabled on the call to listen();

API = [
    ar.Enquiry,
    ar.Ping,
]
ar.listen(self, HostPort('127.0.0.1', 5080), api_server=API, ansar_client=True)

The obvious requirement is that the client must also be “Ansar-enabled”. Refer to the introduction at the top of this page for an overview of this use case.

Connecting To HTTP Servers

Adding a single parameter on the call to connect() establishes an HTTP-based client connection;

ar.connect(self, HostPort('127.0.0.1', 5080), api_client='/')

The presence of an api_client value enables the special byte-level processing of HTTP request and response messages and also sets the leading part of the outbound request URI. The string should always start and end with /. Connected clients can send HttpRequest messages and expect HttpResponse messages in return.

Execution within the local client is asynchronous, whereas the HTTP protocol itself is synchronous. This technicality becomes signficant when the client shares the address of the connection to multiple areas of the application, and those distinct areas begin simultaneous communications with the server. This style of communications is encouraged over standard Ansar connections but is a clash of execution models where an HTTP connection is involved.

To preserve the asynchronous model within the local client, request messages to an HTTP server are sent when the associated connection is available, i.e. there is no pending request. Otherwise the request is added to an internal queue and sent when its turn eventually comes around. Responses are directed to the object that originally sent the request.

All of this extra HTTP-related processing is transparent to the client. It gives the appearance of asynchronicity in the server when that is explicitly not the case. Interaction with HTTP servers is effectively serialized.

Note

Refer to the Listening To HTTP Clients section for further details on the limitations of HTTP-based connections.

Sending HTTP Forms

Sending a message type other than HttpRequest, such as an Enquiry, will result in the sending of an HTTP form (i.e. content type x-www-form-urlencoded). The final request URI will be a catenation of api_client and the name of the message class. Members of the outbound message are scanned during the construction of the proper form representation. Empty members (i.e. a value of None) are omitted and the conversions described in Receiving HTTP Forms are applied in the appropriate way.

Ansar Messaging Over HTTP By The Client

For very specific circumstances it is possible to enable the transfer of request and response messages using the application/json content type. Complete instances of request and response messages are received at the server and client respectively. This behavior is enabled on the call to connect();

ar.connect(self, HostPort('127.0.0.1', 5080), api_client='/', ansar_server=True)

This is the client-side feature matching the functionality described in Enabling Full Ansar Messaging Over HTTP .

An Example Of HTTP Networking

An example HTTP server (i.e. http-server.py) is available in the materials downloaded at the start of this section;

import ansar.connect as ar

SERVER_API = [
    ar.Enquiry,
    api.MixedBag,
]

def http_server(self):
    ar.listen(self, ar.HostPort('127.0.0.1', 5080), api_server=SERVER_API)
    m = self.select(ar.Listening, ar.NotListening, ar.Stop)
    if isinstance(m, ar.Listening):            # Good to go.
        pass
    elif isinstance(m, ar.NotListening):    # Cant setup a server.
        return m
    else:
        return ar.Aborted()        # Control-c.

    while True:
        m = self.select(ar.Accepted,    # Added.
            ar.Closed, ar.Abandoned,    # Lost.
            ar.Stop,
            ar.Enquiry, api.MixedBag, ar.HttpRequest)    # Actual API.
        if isinstance(m, ar.Accepted):
            continue
        elif isinstance(m, (ar.Closed, ar.Abandoned)):
            continue
        elif isinstance(m, ar.Stop):        # Control-c.
            return ar.Aborted()

        if isinstance(m, ar.Enquiry):
            self.reply(ar.Ack())
        elif isinstance(m, api.MixedBag):
            self.reply(ar.HttpResponse(plain_text='Thanks for the bag'))
        elif isinstance(m, ar.HttpRequest):
            self.reply(ar.HttpResponse(plain_text=f'Received HTTP request "{m.request_uri}"'))
        else:
            self.reply(ar.HttpResponse(status_code=500, reason_phrase='Server Error', plain_text='Unreachable code'))

ar.bind(http_server)

if __name__ == '__main__':
    ar.create_object(http_server)

Running the server looks like:

$ python3 http-server.py -debug-level=DEBUG
21:31:39.203 + <00000010>SocketSelect - Created by <00000001>
21:31:39.203 < <00000010>SocketSelect - Received Start from <00000001>
21:31:39.203 > <00000010>SocketSelect - Sent SocketChannel to <00000001>
21:31:39.203 + <00000011>PubSub[INITIAL] - Created by <00000001>
21:31:39.203 < <00000011>PubSub[INITIAL] - Received Start from <00000001>
21:31:39.203 + <00000012>object_vector - Created by <00000001>
21:31:39.203 ~ <00000012>object_vector - Executable "/gh/odds-and-ends/http-server.py" as object process (2138973)
21:31:39.203 ~ <00000012>object_vector - Working folder "/gh/odds-and-ends"
21:31:39.203 ~ <00000012>object_vector - Running object "__main__.http_server"
21:31:39.203 ~ <00000012>object_vector - Class threads (9) "subscribed" (3),"api-client-session" (1), ...
21:31:39.203 + <00000013>http_server - Created by <00000012>
21:31:39.204 ~ <00000010>SocketSelect - Listening on "127.0.0.1:5080", requested "127.0.0.1:5080"
21:31:39.204 > <00000010>SocketSelect - Sent Listening to <00000013>
21:31:39.204 < <00000013>http_server - Received Listening from <00000010>

The curl command can serve as a third-party client:

$ curl -H "Content-Type: application/x-www-form-urlencoded" "http://127.0.0.1:5080/MixedBag"; echo
Thanks for the bag
$

The echo command is included to preserve a tidy console. Further examples of curl commands are shown below to illustrate the behaviour of the server given different inputs:

$ curl -H "Content-Type: application/x-www-form-urlencoded" "http://127.0.0.1:5080/Enquiry"; echo
[Ack]
$ curl -H "Content-Type: application/x-www-form-urlencoded" "http://127.0.0.1:5080/MixedBag"; echo
Thanks for the bag
$ curl -H "Content-Type: application/x-www-form-urlencoded" "http://127.0.0.1:5080/MixedBag" -d 'number=10'; echo
Thanks for the bag
$ curl -H "Content-Type: application/x-www-form-urlencoded" "http://127.0.0.1:5080/MixedBag" -d 'number=10&ratio=0.125'; echo
Thanks for the bag
$ curl -H "Content-Type: application/x-www-form-urlencoded" "http://127.0.0.1:5080/MixedBag" -d 'number=10.5&ratio=0.125'; echo
cannot recover request "/MixedBag" (invalid literal for int() with base 10: '10.5')
$ curl -H "Content-Type: application/json" "http://127.0.0.1:5080/Enquiry" -d '{"value": {}}'; echo
[Ack]
$ curl -H "Content-Type: application/json" "http://127.0.0.1:5080/MixedBag" -d '{"value": {}}'; echo
Thanks for the bag
$ curl -H "Content-Type: application/json" "http://127.0.0.1:5080/MixedBag" -d '{"value": {"number":10}}'; echo
Thanks for the bag
$ curl -H "Content-Type: application/json" "http://127.0.0.1:5080/MixedBag" -d '{"value": {"number":10,"ratio":0.125}}'; echo
Thanks for the bag
$ curl -H "Content-Type: application/json" "http://127.0.0.1:5080/MixedBag" -d '{"value": {"number":10.5,"ratio":0.125}}'; echo
cannot recover request "/MixedBag" (transformation, near "number" (no transformation for data/specification float/Integer8))

An example HTTP client (i.e. http-client.py) is also available in the download materials. This performs a sequence of HTTP requests similar to the curl commands above;

import ansar.connect as ar
import api

def http_client(self):
    ar.connect(self, ar.HostPort('127.0.0.1', 5080), api_client='/')
    m = self.select(ar.Connected, ar.NotConnected, ar.Stop)
    if isinstance(m, ar.Connected):            # Good to go.
        pass
    elif isinstance(m, ar.NotConnected):    # Cant setup a transport.
        return m
    else:
        return ar.Aborted()        # Control-c.

    self.send(ar.Enquiry(), self.return_address)
    self.send(api.MixedBag(number=10), self.return_address)
    self.send(api.MixedBag(number=10, ratio=0.125), self.return_address)
    self.send(api.MixedBag(number=10.5, ratio=0.125), self.return_address)
    self.send(ar.HttpRequest(method='GET', request_uri='/Enquiry'), self.return_address)

    value = ar.Ack()
    m = self.select(ar.Ack, ar.HttpResponse, ar.Closed, ar.Abandoned, ar.Faulted, ar.Stop)
    if isinstance(m, (ar.Closed, ar.Abandoned)):    # Lost transport.
        return m
    elif isinstance(m, ar.HttpResponse) and m.status_code == 400:
        value = m
    elif isinstance(m, ar.Stop):
        return ar.Aborted()        # Control-c.

    m = self.select(ar.Ack, ar.HttpResponse, ar.Closed, ar.Abandoned, ar.Faulted, ar.Stop)
    if isinstance(m, (ar.Closed, ar.Abandoned)):    # Lost transport.
        return m
    elif isinstance(m, ar.HttpResponse) and m.status_code == 400:
        value = m
    elif isinstance(m, ar.Stop):
        return ar.Aborted()        # Control-c.

    m = self.select(ar.Ack, ar.HttpResponse, ar.Closed, ar.Abandoned, ar.Faulted, ar.Stop)
    if isinstance(m, (ar.Closed, ar.Abandoned)):    # Lost transport.
        return m
    elif isinstance(m, ar.HttpResponse) and m.status_code == 400:
        value = m
    elif isinstance(m, ar.Stop):
        return ar.Aborted()        # Control-c.

    m = self.select(ar.Ack, ar.HttpResponse, ar.Closed, ar.Abandoned, ar.Faulted, ar.Stop)
    if isinstance(m, (ar.Closed, ar.Abandoned)):    # Lost transport.
        return m
    elif isinstance(m, ar.HttpResponse) and m.status_code == 400:
        value = m
    elif isinstance(m, ar.Stop):
        return ar.Aborted()        # Control-c.

    m = self.select(ar.Ack, ar.HttpResponse, ar.Closed, ar.Abandoned, ar.Faulted, ar.Stop)
    if isinstance(m, (ar.Closed, ar.Abandoned)):    # Lost transport.
        return m
    elif isinstance(m, ar.HttpResponse) and m.status_code == 400:
        value = m
    elif isinstance(m, ar.Stop):
        return ar.Aborted()        # Control-c.

    return value

ar.bind(http_client)

if __name__ == '__main__':
    ar.create_object(http_client)

Note that the client sends all the request messages in a burst, i.e. the last request is sent before the first response has even been detected. This is a demonstration of the special handling created around the fact that HTTP is a synchronous protocol. There is also extra processing of the response messages to detect any client error and return that as the final output of the client. The error associated with assigning 10.5 to the number member appears at the end of the logs;

$ python3 http-client.py -dl=DEBUG
23:27:53.573 + <00000010>SocketSelect - Created by <00000001>
23:27:53.574 < <00000010>SocketSelect - Received Start from <00000001>
23:27:53.574 > <00000010>SocketSelect - Sent SocketChannel to <00000001>
23:27:53.574 + <00000011>PubSub[INITIAL] - Created by <00000001>
23:27:53.574 < <00000011>PubSub[INITIAL] - Received Start from <00000001>
23:27:53.574 + <00000012>object_vector - Created by <00000001>
23:27:53.574 ~ <00000012>object_vector - Executable "/gh/odds-and-ends/http-client.py" as object process (2141630)
23:27:53.574 ~ <00000012>object_vector - Working folder "/gh/odds-and-ends"
23:27:53.574 ~ <00000012>object_vector - Running object "__main__.http_client"
23:27:53.574 ~ <00000012>object_vector - Class threads (9) "subscribed" (3),"api-client-session" (1),"keeper" (1),"socketry" (1),"connect-to-directory" (1),"published" (2),"connect-to-peer" (1),"networking-session" (5),"address-group" (1)
23:27:53.574 + <00000013>http_client - Created by <00000012>
23:27:53.574 + <00000014>SocketProxy[INITIAL] - Created by <00000010>
23:27:53.574 + <00000015>ApiClientSession[INITIAL] - Created by <00000010>
23:27:53.574 ~ <00000010>SocketSelect - Connected to "127.0.0.1:5080", at local address "127.0.0.1:49096"
23:27:53.574 > <00000010>SocketSelect - Forward Connected to <00000013> (from <00000015>)
23:27:53.575 < <00000013>http_client - Received Connected from <00000015>
23:27:53.575 > <00000013>http_client - Sent Enquiry to <00000015>
23:27:53.575 > <00000013>http_client - Sent MixedBag to <00000015>
23:27:53.575 > <00000013>http_client - Sent MixedBag to <00000015>
23:27:53.575 > <00000013>http_client - Sent MixedBag to <00000015>
23:27:53.575 > <00000013>http_client - Sent HttpRequest to <00000015>
23:27:53.575 < <00000014>SocketProxy[INITIAL] - Received Start from <00000010>
23:27:53.575 < <00000015>ApiClientSession[INITIAL] - Received Start from <00000010>
23:27:53.575 < <00000015>ApiClientSession[READY] - Received Enquiry from <00000013>
23:27:53.575 > <00000015>ApiClientSession[READY] - Sent Enquiry to <00000014>
23:27:53.575 < <00000015>ApiClientSession[READY] - Received MixedBag from <00000013>
23:27:53.575 < <00000015>ApiClientSession[READY] - Received MixedBag from <00000013>
23:27:53.575 < <00000015>ApiClientSession[READY] - Received MixedBag from <00000013>
23:27:53.575 < <00000015>ApiClientSession[READY] - Received HttpRequest from <00000013>
23:27:53.575 < <00000014>SocketProxy[NORMAL] - Received Enquiry from <00000015>
23:27:53.575 > <00000010>SocketSelect - Forward HttpResponse to <00000015> (from <00000014>)
23:27:53.575 < <00000015>ApiClientSession[READY] - Received HttpResponse from <00000014>
23:27:53.575 > <00000015>ApiClientSession[READY] - Sent HttpResponse to <00000013>
23:27:53.575 > <00000015>ApiClientSession[READY] - Sent MixedBag to <00000014>
23:27:53.575 < <00000014>SocketProxy[NORMAL] - Received MixedBag from <00000015>
23:27:53.575 < <00000013>http_client - Received HttpResponse from <00000015>
23:27:53.576 > <00000010>SocketSelect - Forward HttpResponse to <00000015> (from <00000014>)
23:27:53.576 < <00000015>ApiClientSession[READY] - Received HttpResponse from <00000014>
23:27:53.576 > <00000015>ApiClientSession[READY] - Sent HttpResponse to <00000013>
23:27:53.576 > <00000015>ApiClientSession[READY] - Sent MixedBag to <00000014>
23:27:53.576 < <00000013>http_client - Received HttpResponse from <00000015>
23:27:53.576 < <00000014>SocketProxy[NORMAL] - Received MixedBag from <00000015>
23:27:53.576 > <00000010>SocketSelect - Forward HttpResponse to <00000015> (from <00000014>)
23:27:53.576 < <00000015>ApiClientSession[READY] - Received HttpResponse from <00000014>
23:27:53.576 > <00000015>ApiClientSession[READY] - Sent HttpResponse to <00000013>
23:27:53.576 > <00000015>ApiClientSession[READY] - Sent MixedBag to <00000014>
23:27:53.577 < <00000013>http_client - Received HttpResponse from <00000015>
23:27:53.577 < <00000014>SocketProxy[NORMAL] - Received MixedBag from <00000015>
23:27:53.577 > <00000010>SocketSelect - Forward HttpResponse to <00000015> (from <00000014>)
23:27:53.577 < <00000015>ApiClientSession[READY] - Received HttpResponse from <00000014>
23:27:53.577 > <00000015>ApiClientSession[READY] - Sent HttpResponse to <00000013>
23:27:53.577 > <00000015>ApiClientSession[READY] - Sent HttpRequest to <00000014>
23:27:53.577 < <00000013>http_client - Received HttpResponse from <00000015>
23:27:53.577 < <00000014>SocketProxy[NORMAL] - Received HttpRequest from <00000015>
23:27:53.577 > <00000010>SocketSelect - Forward HttpResponse to <00000015> (from <00000014>)
23:27:53.577 < <00000015>ApiClientSession[READY] - Received HttpResponse from <00000014>
23:27:53.577 > <00000015>ApiClientSession[READY] - Sent HttpResponse to <00000013>
23:27:53.577 < <00000013>http_client - Received HttpResponse from <00000015>
23:27:53.577 X <00000013>http_client - Destroyed
23:27:53.578 < <00000012>object_vector - Received Completed from <00000013>
23:27:53.578 X <00000012>object_vector - Destroyed
23:27:53.675 < <00000011>PubSub[NORMAL] - Received Stop from <00000001>
23:27:53.675 X <00000011>PubSub[NORMAL] - Destroyed
23:27:53.675 > <00000010>SocketSelect - Sent Stop to <00000015>
23:27:53.676 < <00000015>ApiClientSession[READY] - Received Stop from <00000010>
23:27:53.676 X <00000015>ApiClientSession[READY] - Destroyed
23:27:53.676 < <00000014>SocketProxy[NORMAL] - Received Close from <00000015>
23:27:53.676 X <00000014>SocketProxy[NORMAL] - Destroyed
23:27:53.677 > <00000010>SocketSelect - Forward Closed to <00000013> (from <00000015>)
23:27:53.677 X <00000010>SocketSelect - Destroyed
HttpResponse
+   http: "HTTP/1.1"
+   status_code: 400
+   reason_phrase: "Bad Request"
+   header: (map){
+   +   @ "Date"
+   +   = "Wed, 21 Aug 2024 23:27:53 GMT"
+   +   @ "Server"
+   +   = "Ansar-API-server/1.0"
+   +   @ "Content-Length"
+   +   = "83"
+   +   @ "Content-Type"
+   +   = "plain/text"
+   }
+   body: [cannot recover request "/MixedBag" (invalid literal for int() with base 10: \'10.5\')]
$