./ahmedhashim

Changing field mappings in Elasticsearch isn’t straightforward because you can’t modify them in place. To update a mapping you have to create a new index, move your data into it, and atomically swap an alias over to point at the new one. The reindex API plus aliases make this possible without taking search offline.

Say you have a books index storing literary information and you want to change how the description field works with a different analyzer, or maybe a different type. Either way, you’re reindexing. The pipelines feeding it only have to pause once, to put the index behind an alias. Every migration after that can run without stopping them.

Aliasing

Elasticsearch won’t let an alias share a name with an index, so books can’t become an alias while the index of the same name still exists. The way around it is to clone the index to books_v1 and put the alias in its place. You only have to do this once.

Cloning requires a read-only source, so the pipelines have to pause for the next few steps. A write that hits the block comes back as a 403 cluster_block_exception, so any pipeline left running needs to retry those instead of dropping them.

PUT /books/_settings
{
  "index.blocks.write": true
}

POST /books/_clone/books_v1

The clone hard-links segments where the filesystem allows it, making it quick even on a large index. books_v1 also inherits the write block, which you’ll lift once the alias is in place.

The clone request can return before the new index’s shards are allocated, so wait for it to go “green” before touching anything else:

GET /_cluster/health/books_v1?wait_for_status=green&timeout=30m

Green means the replicas are up too, which matters because the next step deletes books. You don’t want the only copy of your data sitting on a single primary when that happens.

This wait is the slowest part of the pause. The clone itself is quick, but replicas copy the data over the network, so the bigger the index, the longer it takes. The default timeout is only 30 seconds, so be sure to give it plenty of room.

Nothing can write to either index now, so their counts should match exactly:

GET /books/_count

GET /books_v1/_count

Then swap the index for the alias in a single _aliases request:

POST /_aliases
{
  "actions": [
    {
      "remove_index": {
        "index": "books"
      }
    },
    {
      "add": {
        "index": "books_v1",
        "alias": "books"
      }
    }
  ]
}

The deletion and the alias creation are applied together, ensuring that books always resolves to an index. Searches carry on as before, now served by books_v1.

Lift the inherited block and resume the pipelines:

PUT /books_v1/_settings
{
  "index.blocks.write": false
}

Writes to books now go through the alias to books_v1.

New mapping

Create a new index with the updated mapping:

PUT /books_v2
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "stop"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "standard"
      },
      "author": {
        "type": "keyword"
      },
      "description": {
        "type": "text",
        "analyzer": "custom_analyzer"
      },
      "publication_year": {
        "type": "integer"
      }
    }
  }
}

Dual writes

Now the pipelines should be updated to write every document to both books_v1 and books_v2, using the index names directly. Writing through the books alias would break once it moves since both writes would land in books_v2, and books_v1 would stop getting anything.

Start the dual writes before the reindex. The reindex only copies what books_v1 holds when it starts, and the dual writes cover everything after that, so nothing gets left behind. Searches keep reading from books_v1, which still gets every write, so results stay fresh for the whole migration.

Each document now takes two writes, and one can fail while the other succeeds. Retry each side on its own, or the two indexes will drift apart.

Reindexing

The reindex works from a snapshot of books_v1 taken when it starts. Anything the pipelines write to books_v2 after that is newer than the snapshot, but by default the reindex would still overwrite it with the old version. Setting op_type to create only copies documents that don’t exist in the destination yet, and conflicts: proceed skips the ones that do instead of aborting the whole run.

This assumes the pipelines write whole documents. A partial update or delete aimed at a document the reindex hasn’t copied yet will 404 against books_v2, and a deleted document will get copied over later anyway. If your pipelines do either, make sure you log those misses and replay them once the reindex finishes.

Kick off the reindex asynchronously:

POST /_reindex?wait_for_completion=false
{
  "conflicts": "proceed",
  "source": {
    "index": "books_v1"
  },
  "dest": {
    "index": "books_v2",
    "op_type": "create"
  }
}

For very large indexes, you can throttle with requests_per_second to ease the load:

POST /_reindex?wait_for_completion=false
{
  "conflicts": "proceed",
  "source": {
    "index": "books_v1",
    "size": 5000
  },
  "dest": {
    "index": "books_v2",
    "op_type": "create"
  },
  "requests_per_second": 1000
}

You’ll get back a TASK_ID you can use to monitor progress.

Monitoring

Check the reindex status:

GET /_tasks/TASK_ID

Or watch it tick in real time (pretty satisfying on a large reindex):

# Watch progress every 5 seconds
watch -n 5 'curl -s -X GET "http://your-cluster:9200/_tasks/TASK_ID" | jq ".task.status"'

You can also list every running reindex operation:

GET /_tasks?actions=*reindex&detailed=true

Verification

Start with the summary GET /_tasks/TASK_ID returns once the reindex finishes:

{
  "completed": true,
  "response": {
    "total": 1250000,
    "created": 1198431,
    "version_conflicts": 51569,
    "failures": [],
    ...
  },
  ...
}

total is how many documents the reindex read from the snapshot. Each one was either copied (created) or skipped because a pipeline had already written it (version_conflicts). Those two should add up to total, and failures should be empty.

If they don’t, there’s no need to start over - just run the same reindex again. Documents already in books_v2 get skipped, so a second pass will only fill in what’s missing.

Since both indexes now get every write, their counts should match too, give or take whatever is indexed between the two requests:

GET /books_v1/_count

GET /books_v2/_count

A higher count in books_v2 usually means a delete the reindex undid, which is what replaying the logged misses fixes.

Cutover

The moment of truth: in a single _aliases request, move the alias from the old index to the new one:

POST /_aliases
{
  "actions": [
    {
      "remove": {
        "index": "books_v1",
        "alias": "books"
      }
    },
    {
      "add": {
        "index": "books_v2",
        "alias": "books"
      }
    }
  ]
}

Clients still hitting books won’t notice the change, since both actions are applied together.

Keep the dual writes running for now. books_v1 stays current, so if something looks off, moving the alias back is the same request with the indexes swapped, and nothing is lost.

Cleanup

Once you’re satisfied nothing was lost, stop the pipelines writing to books_v1.

One thing to be aware of is that with automatic index creation on by default, a write that arrives after the delete quietly recreates books_v1 with a dynamic mapping. The pipeline gets a success back, and nothing ever reads the new index. To flush out any writer you missed, put the write block back on first:

PUT /books_v1/_settings
{
  "index.blocks.write": true
}

A straggler now gets a 403 instead of succeeding silently. Once your logs are clean, delete the old index in its own request:

DELETE /books_v1

From here on, books is only ever an alias and every index behind it gets a version number. The next time you need to change a mapping, simply create books_v3 and skip straight past the clone and the pause.