delete index after inserting documents fails

[andreasbaumann]

2016-06-07 08:36:10; cppcms_http, info: POST /strus/index/delete/rrb (http_api.cpp:251)
strusWebService: db/version_set.cc:789: leveldb::VersionSet::~VersionSet(): Assertion `dummy_versions_.next_ == &dummy_versions_' failed.

Restarting the webservice solves the problem, after restart the index can be deleted
without any problems.

[andreasbaumann]

This also happens when running the tests on a fast machine, so presumably a race condition again.

[andreasbaumann]

This was another unclosed iterator. Fixed. Happened only with assertions enabled.

[andreasbaumann]

Happens again on production system. Which operation is it this time, I wonder?

[andreasbaumann]

Possible race: thread using iterator and SIGHUP is simply closing the index.

[andreasbaumann]

Happens in:

leveldb, db/version_set.cc

VersionSet::~VersionSet() {
  current_->Unref();
  assert(dummy_versions_.next_ == &dummy_versions_);  // List must be empty
  delete descriptor_log_;
  delete descriptor_file_;
}

[andreasbaumann]

A race is quite unlikely as queries are reasonably fast and the SIGHUP is received a
second after the last query has been excuted:

2016-12-14 04:38:34; strusWebService, info: query(0.033757s) (query.cpp:334)
2016-12-14 04:38:48; cppcms_http, info: POST /strus/query/xxx (http_api.cpp:251)
2016-12-14 04:38:48; strusWebService, info: query(0.003088s) (query.cpp:334)
2016-12-14 04:38:51; cppcms_http, info: POST /strus/query/xxx (http_api.cpp:251)
2016-12-14 04:38:51; strusWebService, info: query(0.008907s) (query.cpp:334)
2016-12-14 04:38:58; cppcms_http, info: POST /strus/query/xxx (http_api.cpp:251)
2016-12-14 04:38:58; strusWebService, info: query(0.001636s) (query.cpp:334)
2016-12-14 04:40:01; strusWebService, info: Reloading configuration on SIGHUP.. (main.cpp:116)

[andreasbaumann]

Another theory: somewhere an iterator is not closed in a query (for instance an early abort of a
calculation). The question here is: is it in the webservice code or in the strus library code?

[andreasbaumann]

3rd possibility: a bug in leveldb itself, like:
google/leveldb#3

Fixed in 1.18?

[andreasbaumann]

index_stats and query in logfile.
index_stats merely reads a block in leveldb with number of documents, no iterators involved.
query?

[andreasbaumann]

Resists proper analysis, postponed into blue sky.

[patrickfrey]

blue sky. closed