Azure AI Search

First time creating a connector? Read this first.

Send processed data from Unstructured to Azure AI Search. The requirements are as follows. The following video shows how to fulfill the minimum set of Azure AI Search requirements:

Here are some more details about these requirements:

The endpoint and API key for Azure AI Search. Create an endpoint and API key.

The name of the index in Azure AI Search. Create an index.

The Azure AI Search index that you use must have an index schema that is compatible with the schema of the documents that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all circumstances. This is because these schemas will vary based on your source files’ types; how you want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors. You can adapt the following index schema example for your own needs. Be sure to replace <number-of-dimensions> (in three locations in the following example) with the number of dimensions of the embedding model you are using:

{
  "name": "elements-index",
  "fields": [
    {
      "name": "id",
      "type": "Edm.String",
      "key": true
    },
    {
      "name": "record_id",
      "type": "Edm.String",
      "filterable": true
    },
    {
      "name": "element_id",
      "type": "Edm.String"
    },
    {
      "name": "text",
      "type": "Edm.String",
      "searchable": true
    },
    {
      "name": "type",
      "type": "Edm.String"
    },
    {
      "name": "metadata",
      "type": "Edm.ComplexType",
      "fields": [
        {
          "name": "orig_elements",
          "type": "Edm.String"
        },
        {
          "name": "category_depth",
          "type": "Edm.Int32"
        },
        {
          "name": "parent_id",
          "type": "Edm.String"
        },
        {
          "name": "attached_to_filename",
          "type": "Edm.String"
        },
        {
          "name": "filetype",
          "type": "Edm.String"
        },
        {
          "name": "last_modified",
          "type": "Edm.DateTimeOffset"
        },
        {
          "name": "is_continuation",
          "type": "Edm.Boolean"
        },
        {
          "name": "file_directory",
          "type": "Edm.String"
        },
        {
          "name": "filename",
          "type": "Edm.String"
        },
        {
          "name": "data_source",
          "type": "Edm.ComplexType",
          "fields": [
            {
              "name": "url",
              "type": "Edm.String"
            },
            {
              "name": "version",
              "type": "Edm.String"
            },
            {
              "name": "date_created",
              "type": "Edm.DateTimeOffset"
            },
            {
              "name": "date_modified",
              "type": "Edm.DateTimeOffset"
            },
            {
              "name": "date_processed",
              "type": "Edm.DateTimeOffset"
            },
            {
              "name": "permissions_data",
              "type": "Edm.String"
            },
            {
              "name": "record_locator",
              "type": "Edm.String"
            }
          ]
        },
        {
          "name": "coordinates",
          "type": "Edm.ComplexType",
          "fields": [
            {
              "name": "system",
              "type": "Edm.String"
            },
            {
              "name": "layout_width",
              "type": "Edm.Double"
            },
            {
              "name": "layout_height",
              "type": "Edm.Double"
            },
            {
              "name": "points",
              "type": "Edm.String"
            }
          ]
        },
        {
          "name": "languages",
          "type": "Collection(Edm.String)"
        },
        {
          "name": "page_number",
          "type": "Edm.String"
        },
        {
          "name": "links",
          "type": "Collection(Edm.String)"
        },
        {
          "name": "page_name",
          "type": "Edm.String"
        },
        {
          "name": "link_urls",
          "type": "Collection(Edm.String)"
        },
        {
          "name": "link_texts",
          "type": "Collection(Edm.String)"
        },
        {
          "name": "sent_from",
          "type": "Collection(Edm.String)"
        },
        {
          "name": "sent_to",
          "type": "Collection(Edm.String)"
        },
        {
          "name": "subject",
          "type": "Edm.String"
        },
        {
          "name": "section",
          "type": "Edm.String"
        },
        {
          "name": "header_footer_type",
          "type": "Edm.String"
        },
        {
          "name": "emphasized_text_contents",
          "type": "Collection(Edm.String)"
        },
        {
          "name": "emphasized_text_tags",
          "type": "Collection(Edm.String)"
        },
        {
          "name": "text_as_html",
          "type": "Edm.String"
        },
        {
          "name": "regex_metadata",
          "type": "Edm.String"
        },
        {
          "name": "detection_class_prob",
          "type": "Edm.Double"
        }
      ]
    },
    {
      "name": "embeddings",
      "type": "Collection(Edm.Single)",
      "dimensions": <number-of-dimensions>,
      "vectorSearchProfile": "embeddings-config-profile"
    }
  ],
  "vectorSearch": {
    "algorithms": [
      {
        "name": "hnsw-<number-of-dimensions>",
        "kind": "hnsw",
        "hnswParameters": {
          "m": 4,
          "efConstruction": 400,
          "efSearch": 500,
          "metric": "cosine"
        }
      }
    ],
    "profiles": [
      {
        "name": "embeddings-config-profile",
        "algorithm": "hnsw-<number-of-dimensions>"
      }
    ]
  },
  "semantic": {
    "configurations": [
      {
        "name": "default-semantic-config",
        "prioritizedFields": {
          "titleField": null,
          "prioritizedContentFields": [
            { "fieldName": "text" }
          ],
          "prioritizedKeywordsFields": []
        }
      }
    ]
  }
}

The record_id, element_id, and id fields are closely related, but each has a distinct purpose. For more information, see How connectors use record IDs, element IDs, and IDs.

Unstructured Pipelines

Getting started with Pipelines

Using Pipelines