Data is sent to Phirestream through its REST-based API. Any tool capable of making HTTP calls can be a client to Phirestream. Open source SDKs are available for integrating Phirestream in custom applications.

Phirestream implements a subset of the Apache Kafka’s REST API interface. Any general purpose HTTP client such as curl or client SDK that implements this interface can be used with Phirestream.

Performance Considerations

The API methods listed below both take an optional query parameter called async. This optional parameter specifies whether or not the message producing to the Kafka topic will be blocking. Set async to true to not wait for confirmation the messages were produced. This will give higher throughput but will not provide indication if a message fails to be produced to the Kafka topic. Any errors encountered while producing messages will be written to Phirestream’s log.

Phirestream API

Producing messages for redaction to an Apache Kafka topic

POST – https://phirestream:8080/topics/{topic}

Path Parameters

  • topic (string) – The Apache Kafka topic to produce the message to.

Query Parameters

  • async (boolean) – Whether or not to wait for the message to be produced to the topic after redaction. Defaults to false.
  • profile (string) – The name of the filter profile to apply. If not specified, the topic name is used as the name of the filter profile.
  • context (string) – The redaction context.
  • id (string) – The document ID.

Body

The (string) JSON-encoded messages to publish.

Responses

  • 200 – The response includes the partition and offset for each produced message.
  • 404 – Returned if the Apache Kafka topic does not exist.

Example Request

An example request using curl is shown below. In this request we are publishing a single message to Phirestream. The records property is an array so multiple messages can be included in a single request.

curl -X POST \
  https://localhost:8080/topics/default \
  -H 'Content-Type: application/vnd.kafka.json.v2+json' \
  -d '{
    "records": [
        {
            "key": "key-1",
            "value": "George Washington was president."
        },
    ]
}'

Example Response

{
  "key_schema_id": null,
  "value_schema_id": null,
  "offsets": [
    {
      "partition": 2,
      "offset": 100
    },
    {
      "partition": 1,
      "offset": 101
    }
  ]
}

If we look at the content of mytopic on the Apache Kafka broker we will see a single message with the value {{{REDACTED-entity}}} was president. Phirestream received the request, processed the person’s name as sensitive information, and published the modified data to Apache Kafka.

Produce messages for redaction to an Apache Kafka topic partition

POST – https://phirestream:8080/topics/{topic}/partitions/{partition_id}

Path Parameters

  • topic (string) – The Apache Kafka topic to produce the message(s) to.
  • partition_id (integer) – The partition to produce the message(s) to.

Query Parameters

  • async (boolean) – Whether or not to wait for the message to be produced to the topic after redaction. Defaults to false.
  • profile (string) – The name of the filter profile to apply. If not specified, the topic name is used as the name of the filter profile.
  • context (string) – The redaction context.
  • id (string) – The document ID.

Body

The (string) JSON-encoded messages to publish.

Responses

  • 200 – The response includes the partition and offset for each produced message.
  • 404 – Returned if the Apache Kafka topic does not exist.

An example request using curl is shown below. In this request we are publishing a single message to Phirestream. The records property is an array so multiple messages can be included in a single request.

If we look at the content of mytopic on the Apache Kafka broker we will see a single message with the value {{{REDACTED-entity}}} was president. Phirestream received the request, processed the person’s name as sensitive information, and published the modified data to Apache Kafka.

Example Request

curl -X POST \
  https://localhost:8080/topics/default/partition/1 \
  -H 'Content-Type: application/vnd.kafka.json.v2+json' \
  -d '{
    "records": [
        {
            "key": "key-1",
            "value": "George Washington was president."
        },
    ]
}'

Example Response

{
  "key_schema_id": null,
  "value_schema_id": null,
  "offsets": [
    {
      "partition": 1,
      "offset": 100
    },
    {
      "partition": 1,
      "offset": 101
    }
  ]
}