Fix Elasticsearch 2.x to 1.7 Downgrade Issue: unexpected field in shard state index_uuid

05 December 2016 on . 2 minutes to read

The Issue

While booting up a client’s app locally, I ran into the following from Elasticsearch:

[ERROR][gateway.local.state.shards] [Kymaera] failed to read local state (started shards), exiting...
org.elasticsearch.ElasticsearchException: unexpected field in shard state [index_uuid]
{1.7.6}: Initialization Failed ...
1) ElasticsearchException
  NullPointerException2) ElasticsearchException[unexpected field in shard state [index_uuid]]
  CorruptStateException[unexpected field in shard state [index_uuid]]

Some quick searching led me to this github issue.

Unfortunately, following their approach (simply delete your ES data), failed for me. What follows is a simple, but more complete fix which took care of my problem. This is on an OS X El Capitan install; the error came from an ES 2.1 install I had, which caused problems when I did the following:

brew unlink elasticsearch
brew link elasticsearch@1.7

Approach

These are the steps I needed to solve this issue:

  • Find data folder
  • Delete data folder
  • Uninstall all installed ES versions
  • Reinstall needed ES

Find Data

brew info elasticsearch

brew info elasticsearch
elasticsearch: stable 5.0.2, HEAD
Distributed search & analytics engine
https://www.elastic.co/products/elasticsearch
Conflicts with: elasticsearch@1.7, elasticsearch@2.4
/usr/local/Cellar/elasticsearch/2.0.0_1 (51 files, 30.4M)
  Built from source on 2015-12-13 at 14:15:05
/usr/local/Cellar/elasticsearch/2.1.1 (50 files, 30.9M)
  Built from source on 2016-01-21 at 10:16:44
/usr/local/Cellar/elasticsearch/5.0.2 (98 files, 34.8M)
  Built from source on 2016-12-05 at 09:08:56
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/elasticsearch.rb
==> Requirements
Required: java >= 1.8 ✔
==> Caveats
Data:    /usr/local/var/elasticsearch/elasticsearch_nrowegt/
Logs:    /usr/local/var/log/elasticsearch/elasticsearch_nrowegt.log
Plugins: /usr/local/Cellar/elasticsearch/5.0.2/libexec/plugins/
Config:  /usr/local/etc/elasticsearch/
plugin script: /usr/local/Cellar/elasticsearch/5.0.2/libexec/bin/plugin

Delete Data

rm -rf /usr/local/var/elasticsearch/elasticsearch_nrowegt/

Uninstall ALL ES Versions

brew uninstall --force elasticsearch

At this point, which elasticsearch should return nothing, as should brew info elasticsearch.

Note: I tried this with only uninstalling specific versions, but there were some others that were left behind which gave me numerous file not found errors when booting ES. This happens because from ES’s point of view, your data store has filed, and it’s trying to restore your shards. Nuking all installs was the easy way to bypass that.

UPDATE

I ran into a similar issue today on starting up ES: marking and sending shard failed due to failed recovery in enabling replication

The issue was because launchctl had an elasticsearch process running. Killing this, unloading ES and deleting the data files solved the problem. Specific commands came from this gist: Mac uninstall elasticsearch

Install ES 1.7

brew install elasticsearch@1.7

Wrap Up

You should now be good to continue with your development. This solved the issue for me


If you enjoy having free time and the peace of mind that a professional is on your side, then you’d love to have me work on your project.

Contact or view a list of available services to see how I’ll make your life better, easier and bring satisfaction back into you running your business.