Database corruption issues - fix bdb or ditch it

Huh - sounds like the berkley db got its knickers in a twist.

I'd try running db_recover on the database as suggested (in this case it'll be $rbothome/plugin_registry.db) and see if that sorts it out. If not, perhaps a db_dump -> db_load?

Was rbot interrupted somehow during use? Perhaps killed or crashed?

Note to self - improve bdb pieces. Try to autorecover. Probably move from one big plugin db, to one for each plugin.

I'm modifying this ticket. It may be time to ditch bdb as this is becoming a hassle. Other users have had db problems too. Looking into other alternatives eg sqlite.

I'm not entirely sure this is a related issue but it seems likely enough that I won't create a seperate issue for it:

/usr/local/lib/site_ruby/1.8/rbot/dbhash.rb:74:in `initialize': PANIC: Invalid argument -- DB_RUNRECOVERY: Fatal error, run database recovery (BDB::Fatal)
        from /usr/local/lib/site_ruby/1.8/rbot/dbhash.rb:74:in `initialize'
        from /usr/local/lib/site_ruby/1.8/rbot/keywords.rb:96:in `initialize'
        from /usr/local/lib/site_ruby/1.8/rbot/ircbot.rb:153:in `initialize'
        from /usr/bin/rbot:78

Do you want help porting to SQLite? I'd be interested in contributing.

I'm giving bdb one more shot. I think the current svn has a reasonably robust db setup using a single environment and auto recovery. I'd like people to try it a while and let me know if they have problems. If they do then I will indeed ditch the darn thing :)

I'm also a victim of the PANIC: Invalid argument -- DB_RUNRECOVERY bug, using rbot svn-HEAD. Is this perhaps an issue with my ruby-bdb version? (Gentoo, dev-ruby/ruby-bdb 0.5.3) For now i just removed the recovery feature.

Another issue seems to be the environment. When I start rbot with "rbot ./Botdata" for example, BDB @env would be set to ./Botdata but then code like

@db = DBTree.create_db(@bot.botclass + "/#{key}.db")

would try to create the actual db in ./Botdata/Botdata/key.db.

I worked around this with:

@@env = BDB::Env.open("#{@bot.botclass}/..", BDB::CREATE | BDB::INIT_MPOOL)

i vote for pulling bdb for these reasons:

• install requires non-gem dependancies which are a PITA and require additional docs, etc

• BDB versioning can be problematic

• the Subversion team started with BDB and then added FS as the default store for similar reasons

• rbot is very close to being fully installable with gem - this should be the goal to maximize use of the system. ie:

  gem install rbot
  rbot

(no intermediate steps to google's for documentation, resolve deps, find libs, etc)

The above issues are pretty much resolved in trunk now.

I'd need a good alternative suggestion to pull bdb. These databases are too big to keep in memory and need fast key-based lookups. I'm not aware of an obvious alternative :/

I think we can keep bdb. I've just commited changeset [192] which should give a strong push towards preventing further database corruption.

However, if you have a corrupted database you will now have problems starting the bot ... as it should be! If with the new changes the bot will not start, follow this procedure:

1. move all your databases (and log.00000.... files) in a different location
2. start rbot (it should now start and create all the databases anew). If it doesn't start, you probably forgot something.
3. shut the bot down and repeat the following procedure until the second point fails:
   1. overwrite one database file with the old copy
   2. start bot
   3. quit bot
4. the last db you copied is (one of) the corrupt database(s). Rename it to something like broken.db and run db_dump broken.db | db_load originalname.db where originalname was its original name
5. try starting the bot again.
6. repeat until all old db files have been copied over and recovered if broken

This should help you set up a functional environment again :)

Please report here if this really works for you (after upgrading to the latest svn). If it does, we can close this ticket.

And oh yes, there ought to be a bdb gem. Does anybody know where we can put pressure for this to happen?

I think I fixed the issues with config directories in [195]. Rbot didn't treat relative paths correctly when the config directory was specified on the command line.

I think we can close this bug, although we should put the info on how to recover a corrupted bdb environment somewhere on the wiki.