5.5.2. cFp_HelloThemisto¶

Note: This HTML section is rendered based on the Markdown file in cFp_HelloThemisto.

A cloudFPGA project built upon the shell Themisto.

5.5.2.1. Overview¶

The integration of a user application and a cloudFPGA shell into a top-level design is what constitutes a cloudFPGA project.

This project builds on the shell Themisto which is a shell with enhanced routing support for node-to-node communications. It exemplifies the creation of a cluster consisting of one CPU and two FPGAs organized in a ring topology.

This setup is shown in the figure below and the scenario is as follows: a virtual machine (VM) sends data to a 1st FPGA (F1) via a UDP and/or a TCP network connection. When F1 is done with the processing of the incoming data, it forwards them to the next FPGA in the ring (i.e. F2), using the same network protocol as the incoming traffic. Finally, F2 which is the last FPGA of the ring, sends the data back to the VM after applying its own processing.

The shell Themisto uses the concept of MPI communicator in which each cloudFPGA role application is assigned a unique number to identify itself. This number is called the node_rank (or rank for short), its value always ranges from 0 to cluster-size-1 and that value does not change once the cluster is created.

The two FPGA applications of this project leverage both their node_rank and the cluster_size to determine the address of the next node in the following manner:

next_node_rank = (curr_node_rank + 1) % cluster_size

As a result, the rudimentary size of the proposed ring-based cluster can be extended by simply inserting more such FPGA application nodes into the ring.

5.5.2.2. Shell-Role-Architecture¶

In cloudFPGA (cF), a user application is referred to as a ROLE and is integrated along with a SHELL that abstracts the hardware components of the FPGA module. The combination of a specific ROLE and its associated SHELL into a top-level design (TOP) is referred to as a Shell-Role-Architecture (SRA).

The shell-role-architecture used by the project cFp_HelloThemisto is shown in the following figure. It consists of:

a SHELL of type ***Themisto**_. This is the current default cloudFPGA shell because it supports Partial Reconfiguration (PR) and can therefore be paired with a dynamically configured user role application. Additionally, the reconfiguration of such a role is performed by the Themisto shell itself, upon reception of a new partial bitstream for the application via the TCP/IP network of the data center. The shell Themisto includes the following building blocks:
- a 10 Gigabit Ethernet (ETH) sub-system,
- a Network Transport Stack (NTS),
- a Network Abstraction Layer (NAL),
- a FPGA Management Core (FMC),
- a DDR4 Memory sub-system (MEM),
- a Memory Mapped IO (MMIO) sub-system.
a ROLE of type ***HelloThemisto**_ which implements the set of TCP- and UDP- applications used by this project.

Toplevel block diagram of the cFp_HelloThemisto project

Info/Warning

You will be able to generate the FPGA bitstreams of this cluster example project on your computer but you won’t be able to deploy and access the cluster without connecting to the cloudFPGA infrastructure via VPN. To get such an access granted, you need to request a cloudFPGA account here.
Next, please note that the following PR build procedure is only compatible with Vivado 2019.2 to 2020.1 versions included (for the time being).

5.5.2.3. How to build the project¶

The current directory contains the sra build script which handles all the required steps to generate the partial bitstreams needed for this project. During the build, both SHELL and ROLE dependencies are analyzed to solely re-compile and re-synthesize the components that must be recreated.

$ SANDBOX=`pwd`           # (a short for your working directory)

5.5.2.3.1. Step-1: Clone and configure the project¶

$ cd ${SANDBOX}

$ git clone --recursive git@github.com:cloudFPGA/cFp_HelloThemisto.git
    (if you don't have a GitHub account and/or SSH keys set up, use the command:
     git clone --recursive https://github.com/cloudFPGA/cFp_HelloThemisto.git)

$ cd cFp_HelloThemisto/cFDK
$ git checkout main
$ cd ..

5.5.2.3.2. Step-2: Setup your environment¶

$ cd ${SANDBOX}/cFp_HelloThemisto
$ source env/setenv.sh

5.5.2.3.3. Step-3: Add your cloudFPGA credentials¶

Create a file called user.json and add your cloudFPGA credentials in it. The credentials consist of the username, the password and the projectname that were provided to you when you registered.

$ echo "{\"credentials\": {\"username\": \"YOUR_USERNAME\", \"password\": \"YOUR_PASSWORD\"}, \"project\": \"YOUR_PROJECTNAME\"}" > user.json

5.5.2.3.4. Step-4: Retrieve the latest static shell¶

Use the command ./sra update-shell to download the latest DCP shell from the cloudFPGA Resource Manager (cFRM).

$ cd ${SANDBOX}/cFp_HelloThemisto

$ ./sra update-shell
[cFBuild] Updated dcp of Shell 'Themisto' to latest version (2) successfully. DONE.
    (downloaded dcp to ${SANDBOX}/cFp_HelloThemisto/dcps/3_topFMKU60_STATIC.dcp)

5.5.2.3.5. Step-5: Activate the 1st role (optional) and generate corresponding partial bitstream¶

This project contains two roles but only one role can be active at a time. So we need to specify the current active role to build and implicitly deactivate the other one. This step is optional if you just cloned this project because the default active role is already set to msg-ring-app.

$ cd ${SANDBOX}/cFp_HelloThemisto

$ ./sra config use-role "msg-ring-app"

Now, generate a partial bitstream for the current active role (i.e msg-ring-app):

$ cd ${SANDBOX}/cFp_HelloThemisto

$ ./sra build pr

The build is successful if you get the following message after ~40 minutes, indicating that a partial bitstream was generated.

<cloudFPGA> ################################################################################
<cloudFPGA> ##  DONE WITH BITSTREAM GENERATION RUN
<cloudFPGA> ################################################################################
<cloudFPGA> End at: HH:MM:SS Day Month Day Year

You will find this newly created bitstream together with other generated files in the folder ${SANDBOX}/cFp_HelloThemisto/dcps under the name
4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.<bin|sig|bit|rtp> as shown below:

$ ls -al ${SANDBOX}/cFp_HelloThemisto/dcps/
2_topFMKU60_impl_msg-ring-app_complete_pr.dcp
3_topFMKU60_STATIC.dcp
3_topFMKU60_STATIC.json
4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.bin
4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.bin.sig
4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.bit
4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial_clear.bin
4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial_clear.bit
5_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.rpt
pr_verify.rpt

5.5.2.3.6. Step-6: Activate and generate a partial bitstream for the 2nd role¶

You can check the currently active role as well as the list of defined roles for this project with the command:

$ cd ${SANDBOX}/cFp_HelloThemisto

$ ./sra config show
[sra:INFO] The following roles are currently defined:
Role msg-ring-app in directory ${SANDBOX}/cFp_HelloThemisto/ROLE/1
Role invertcase-ring-app in directory ${SANDBOX}/cFp_HelloThemisto/ROLE/2
Current active role: msg-ring-app

Now select and activate the role invertcase-ring-app, and generate a partial bitstream for this 2nd role:

$ cd ${SANDBOX}/cFp_HelloThemisto
$ ./sra config use-role "invertcase-ring-app"

$ ./sra build pr

After another ~40 minutes, the following files will be added to the ./dcps directory:

$ ls -al ${SANDBOX}/cFp_HelloThemisto/dcps/*invertcase*
2_topFMKU60_impl_invertcase-ring-app_complete_pr.dcp
4_topFMKU60_impl_invertcase-ring-app_pblock_ROLE_partial.bin
4_topFMKU60_impl_invertcase-ring-app_pblock_ROLE_partial.bin.sig
4_topFMKU60_impl_invertcase-ring-app_pblock_ROLE_partial.bit
4_topFMKU60_impl_invertcase-ring-app_pblock_ROLE_partial_clear.bin
4_topFMKU60_impl_invertcase-ring-app_pblock_ROLE_partial_clear.bit
5_topFMKU60_impl_invertcase-ring-app_pblock_ROLE_partial.rpt

5.5.2.4. How to deploy the cluster¶

5.5.2.4.1. Step-7: Upload the generated PR bitstreams¶

Before we can deploy this cluster consisting of one CPU and two FPGAs, we first need to upload the newly generated PR bitfiles of our two roles. We refer to such PR bitfiles as application image or app_image for short. The below step-7a and step-7b cover the two possible options for uploading these two generated application images.

5.5.2.4.1.1. Step-7a: Upload the application images with the GUI-API¶

The cloudFPGA resource manager provides a web-based graphical user interface to its API. It is available as a Swagger UI at http://10.12.0.132:8080/ui/#/.

To upload the app_image corresponding to the role msg-ring-app, expand the Swagger menu Images and the operation [POST] /images/app_logic - Upload an image of type app_logic. Then, fill in the fields 1 to 6 as exemplified below:

with your username for accessing the cloudFPGA infrastructure

with your password

with specific image_details about the app_image to upload

  * INFO: You must submit this information in a dictionary format, similar to the following example:
    ```
        {
          "cl_id"      : "2",
          "fpga_board" : "FMKU60",
          "shell_type" : "Themisto",
          "comment"    : "The 1st app_image (msg-ring-app)"
        }
    ```
  * **WARNING**: The "cl_id" field stands for _composable_logic_id_ and its value **MUST BE THE SAME** as the `id` specified in the `dcps/3_topFMKU60_STATIC.json` file **!!**

4) with the **image_file.bin** corresponding to the partial bitfile of the 1st role

  * With respect to the `msg-ring-app` example --> `4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.bin`

5) with the **signature_file.bin.sig** corresponding to the current *imagefile.bin*

  * With respect to the `msg-ring-app` example --> `4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.bin.sig`

6) with the **pr_verify.rpt** file containing the output of *pr_verify* command for this build.

* With respect to the `msg-ring-app` example --> `4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.rpt`

Next, scroll down to the Response body section of the server and write down the image id for use in the next step.

Repeat step-7a for the second bitstream called invertcase_ring_app.

5.5.2.4.1.2. Step-7b: Upload image with the cFSP-API¶

The second option for uploading a PR bitstream is based on a command-line interface to the cFRM API. This method is provided by the cloudFPGA Support Package (cFSP) which must be installed beforehand as follows:

$ cd ${SANDBOX}

$ git clone https://github.com/cloudFPGA/cFSP.git
$ cd cFSP
$ which python3     # (we recommend python 3.6 or higher)
/usr/bin/python3
$ virtualenv -p /usr/bin/python3 cfenv
$ source cfenv/bin/activate
$ pip install -r requirements.txt

Once cFSP is installed, add your cloudFPGA credentials. This process creates a file called user.json which has your cloudFPGA credentials in it. The credentials consist of the username, the password and the projectname that were provided to you when you registered.

$ cd ${SANDBOX}/cFSP

$ ./cfsp user load --username=<YOUR_USERNAME> --password=<YOUR_PASSWORD> --project=<YOUR_PROJECTNAME>

$ ./cfsp user show

The output of these two commands will look like this:

0%|                                                                                                                | 0/1 [00:00<?, ?it/s]INFO: Repeat #0
Writing credentials template to ./user.json
['load']
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 3305.20it/s]

0%|                                                                                                                | 0/1 [00:00<?, ?it/s]INFO: Repeat #0
User     : YOUR_USERNAME
Password : YOUR_PASSWORD
Project  : YOUR_PROJECTNAME
['show']
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 5801.25it/s]

Finally, upload the generated bitstreams located in the dcps folder of the ${SANDBOX}/cFp_HelloThemisto project as follows:

$ cd ${SANDBOX}/cFSP

$ ./cfsp image post-app-logic --image_file=${SANDBOX}/cFp_HelloThemisto/dcps/4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.bin
$ ./cfsp image post-app-logic --image_file=${SANDBOX}/cFp_HelloThemisto/dcps/4_topFMKU60_impl_invertcase-ring-app_pblock_ROLE_partial.bin

Similarly to the GUI-API procedure, do not forget to write down the image id returned by the server (e.g. ‘dcd9bb0a-7888-4613-a9d3-319c80fcf6e3’).

0%|                                                                                                                                                                        | 0/1 [00:00<?, ?it/s]INFO: Repeat #0
INFO: This json file will be used: ${SANDBOX}/cFp_HelloThemisto/dcps/3_topFMKU60_STATIC.json
INFO: No --sig_file provided. Assuming ${SANDBOX}/cFp_HelloThemisto/dcps/4_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.bin.sig
INFO: No --pr_verify_rpt provided. Assuming ${SANDBOX}/cFp_HelloThemisto/dcps/5_topFMKU60_impl_msg-ring-app_pblock_ROLE_partial.rpt
{'breed': None,
 'comment': None,
 'fpga_board': None,
 'id': 'dcd9bb0a-7888-4613-a9d3-319c80fcf6e3',
 'shell_type': None}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.21s/it]

0%|                                                                                                                                                                        | 0/1 [00:00<?, ?it/s]INFO: Repeat #0
INFO: This json file will be used: ${SANDBOX}/cFp_HelloThemisto/dcps/3_topFMKU60_STATIC.json
INFO: No --sig_file provided. Assuming ${SANDBOX}/cFp_HelloThemisto/dcps/4_topFMKU60_impl_invertcase-ring-app_pblock_ROLE_partial.bin.sig
INFO: No --pr_verify_rpt provided. Assuming ${SANDBOX}/cFp_HelloThemisto/dcps/5_topFMKU60_impl_invertcase-ring-app_pblock_ROLE_partial.rpt
{'breed': None,
 'comment': None,
 'fpga_board': None,
 'id': 'c5e09074-78fa-4585-8621-d7dd4095d749',
 'shell_type': None}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.81s/it

5.5.2.4.2. Step-8: Request a cloudFPGA cluster and deploy it¶

Now that your FPGA images have been uploaded to the cFRM, you can request a cloudFPGA cluster to be programmed and deployed with your images. The below step-8a and step-8b will cover the two available options for creating a new cF cluster.

5.5.2.4.2.1. Step-8a: Create a cluster with the GUI-API¶

To create a new cluster via the GUI-API, point your web browser at http://10.12.0.132:8080/ui/#/ and expand the Swagger operation [POST] /clusters - Request a cluster of the menu Clusters.

Then, fill in the fields 1 to 4 as exemplified below: 1) with your username for accessing the cloudFPGA infrastructure 2) with your password 3) with the name of the OpenStack project to use (if you own multiple projects), otherwise leave the field setting to “default”. 4) with specific cluster_details about the mapping of node-IDs to images

INFO: You must submit this information in a dictionary format similar to the following example (note that the CPU is not provided with an image_id and is flagged as “NON_FPGA” instead):

[
  {
    "image_id" : "<the-image-id-retrived-at-step-8a>",  # E.g. "7891e291-8847-4302-9b07-248c4feefd69"
    "node_id"  : 1
  },{
    "image_id" : "<the-image-id-retrived-at-step-8a>",  # E.g. "f23b8025-7beb-4662-8d73-30563593ffbe"
    "node_id"  : 2
  },{
    "image_id" : "NON_FPGA",
    "node_ip"  : "<the-floating-ip-of-your-VM>",        # E.g. "10.12.2.53"
    "node_id"  : 0
  }
]

Next, scroll down to the server’s response and write down the two “role_ip” addresses for later accessing your FPGA instances.

5.5.2.4.2.2. Step-8b: Create a cluster with the cFSP-API¶

To create a similar cluster via the cFSP-API, enter the following command:

$ cd ${SANDBOX}/cFSP

$ ./cfsp cluster post --image_id=7891e291-8847-4302-9b07-248c4feefd69 --image_id=f23b8025-7beb-4662-8d73-30563593ffbe --node_ip=10.12.2.53

Next, and similarly to the GUI-API procedure, do not forget to write down the two “role_ip” returned by the server (see example below).

0%|                                                                                                                                                                        | 0/1 [00:00<?, ?it/s]INFO: Repeat #0
{'cluster_id': 265,
 'instances': [{'fpga_board': 'FMKU60',
                'image_id': '7891e291-8847-4302-9b07-248c4feefd69',
                'instance_id': 88,
                'project_name': 'cf_ALL',
                'role_ip': '10.12.200.8',
                'shell_type': 'NO_SRA',
                'slot_num': 26,
                'status': 'USED',
                'user_id': 'cfdummy'},
               {'fpga_board': 'FMKU60',
                'image_id': 'f23b8025-7beb-4662-8d73-30563593ffbe',
                'instance_id': 41,
                'project_name': 'cf_ALL',
                'role_ip': '10.12.200.236',
                'shell_type': 'NO_SRA',
                'slot_num': 21,
                'status': 'USED',
                'user_id': 'cfdummy'}],
 'nodes': [{'image_id': '7891e291-8847-4302-9b07-248c4feefd69',
            'inst_id': 88,
            'node_id': 0,
            'node_ip': '10.12.200.8',
            'slot_num': 26},
           {'image_id': 'f23b8025-7beb-4662-8d73-30563593ffbe',
            'inst_id': 41,
            'node_id': 1,
            'node_ip': '10.12.200.236',
            'slot_num': 21},
           {'image_id': 'NON_FPGA', 'node_id': 2, 'node_ip': '10.12.2.53'}],
 'user_id': 'cfdummy'}
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [02:58<00:00, 178.31s/it]

5.5.2.4.3. Step-9: How to test the deployed cluster¶

As mentioned above, the role msg-ring-app always forwards the incoming traffic to the next node in the cluster ring without modifying it, while the role invertcase-ring-app toggles the character case of the received data before forwarding them to the next node.

An easy way to exercise this traffic scenario is to open two terminals on the VM of the created cluster, and to use the netcat network utility (often abbreviated as nc) for reading from and writing to the cluster using TCP or UDP connections. With all the communications of this cluster going over the UDP/TCP port 2718, enter the following commands in the two terminals:

Terminal 1 - Set nc in TCP transmission mode

$ ping <role_ip_1>         # w/ role_ip_1 = the IP address of the 1st FPGA (retrieve at step-8)
$ nc   <role_ip_1> 2718    # Add `-u` for a UDP connection

Terminal 2 - Set nc in TCP listen mode

$ ping <role_ip_2>         # w/ role_ip_2 = the IP address of the 2nd FPGA (retreived at step-8)
$ nc -l 2718               # Add `-u` for a UDP connection

You can now type in some text in the 1st terminal, and this text should be echoed back with all its characters inverted in the 2nd terminal as the following example shows.

Info: Some firewalls may block network packets if there is not a connection to the remote machine/port. Hence, to get the HelloThemisto example to work, the following commands might be necessary to be executed (as root):

$ firewall-cmd --zone=public --add-port=2718-2750/udp --permanent
$ firewall-cmd --zone=public --add-port=2718-2750/tcp --permanent
$ firewall-cmd --reload

5.5.2.4.4. Step-10: For further experiments¶

Now that your cluster with one CPU and two FPGAs is operational, you may want to try and deploy a larger cluster at step-9 or extend the size of the current cluster with the [PUT] /clusters/{cluster_id}/extend - Add nodes to an existing cluster menu of the GUI-API.

Depending on the combination of msg-ring-app and invertcase-ring-app roles in your extended cluster, the text of the 2nd terminal will have its characters inverted or not.